org.wandora.piccolo.utils.crawler

Class AbstractCrawler

java.lang.Object

org.wandora.piccolo.utils.crawler.AbstractCrawler

All Implemented Interfaces:

Crawler, CrawlerAccess

Direct Known Subclasses:

WebCrawler

public abstract class AbstractCrawler
extends java.lang.Object
implements CrawlerAccess, Crawler

Author:

akivela

Field Summary

Fields

Modifier and Type	Field and Description
private CrawlerAccess	callback
protected boolean	forceExit
protected int	handleCount
private java.util.HashMap<java.lang.String,java.util.ArrayList<Handler>>	handlers
private java.util.HashMap	interruptHandlers
private URLMask	mask
private int	maxLeft
private java.util.HashMap	properties
private boolean	verbose

Constructor Summary

Constructors

Constructor and Description
AbstractCrawler() Creates a new instance of AbstractCrawler

Method Summary

Modifier and Type	Method and Description
void	addHandler(Handler h) Adds a ContentHandler to the used content handlers.
void	addInterruptHandler(InterruptHandler h) Adds a InterruptHandler to the used content handlers.
static java.lang.Object	createObject(org.w3c.dom.Element e) Deprecated.
void	forceExit()
CrawlerAccess	getCallBack()
int	getCrawlCounter()
int	getHandledDocumentCount()
java.util.Collection<Handler>	getHandler(java.lang.String contentType)
InterruptHandler	getInterruptHandler(int interruptType)
URLMask	getMask()
java.lang.Object	getProperty(java.lang.String key)
boolean	isVerbose()
void	loadSettings(org.w3c.dom.Element rootElement)
void	loadSettings(java.io.InputStream in)
void	loadSettings(java.lang.String file)
void	modifyCrawlCounter(int delta)
void	setCallBack(CrawlerAccess cb) Sets the callback object.
void	setCrawlCounter(int n) Sets the maximum number of pages the crawler will process.
void	setMask(URLMask m) Sets the used URLMask
void	setProperty(java.lang.String key, java.lang.Object value)
void	setVerbose(boolean v)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.wandora.piccolo.utils.crawler.CrawlerAccess

add, addObject

Methods inherited from interface org.wandora.piccolo.utils.crawler.Crawler

crawl

Field Detail

properties

private java.util.HashMap properties

handlers

private java.util.HashMap<java.lang.String,java.util.ArrayList<Handler>> handlers

interruptHandlers

private java.util.HashMap interruptHandlers

mask

private URLMask mask

verbose

private boolean verbose

callback

private CrawlerAccess callback

maxLeft

private int maxLeft

forceExit

protected boolean forceExit

handleCount

protected int handleCount

Constructor Detail

AbstractCrawler

public AbstractCrawler()

Creates a new instance of AbstractCrawler

Method Detail

forceExit

public void forceExit()

Specified by:

forceExit in interface CrawlerAccess

setProperty

public void setProperty(java.lang.String key,
                        java.lang.Object value)

Specified by:

setProperty in interface CrawlerAccess

getProperty

public java.lang.Object getProperty(java.lang.String key)

loadSettings

public void loadSettings(java.lang.String file)
                  throws java.lang.Exception

Throws:

java.lang.Exception

loadSettings

public void loadSettings(java.io.InputStream in)
                  throws java.lang.Exception

Throws:

java.lang.Exception

loadSettings

public void loadSettings(org.w3c.dom.Element rootElement)
                  throws java.lang.Exception

Throws:

java.lang.Exception

setMask

public void setMask(URLMask m)

Sets the used URLMask

getMask

public URLMask getMask()

addHandler

public void addHandler(Handler h)

Adds a ContentHandler to the used content handlers.

getHandler

public java.util.Collection<Handler> getHandler(java.lang.String contentType)

addInterruptHandler

public void addInterruptHandler(InterruptHandler h)

Adds a InterruptHandler to the used content handlers.

getInterruptHandler

public InterruptHandler getInterruptHandler(int interruptType)

setCallBack

public void setCallBack(CrawlerAccess cb)

Sets the callback object. When content handlers recognize an url in the content of a document (or otherwise figure out an url the crawler should crawl), they give this url to the call back object. They can also give any other objects they build from the content to the call back. The default call back is the crawler itself. If you want the crawler to crawl deeper when url are recognized, you should, at some point, use the call back functions of the crawler. That is, if you set your own call back, in its call back functions, you should forward the method calls to the crawler if you want the page to be added to the crawler's queue.

getCallBack

public CrawlerAccess getCallBack()

setCrawlCounter

public void setCrawlCounter(int n)

Sets the maximum number of pages the crawler will process.

getCrawlCounter

public int getCrawlCounter()

modifyCrawlCounter

public void modifyCrawlCounter(int delta)

getHandledDocumentCount

public int getHandledDocumentCount()

isVerbose

public boolean isVerbose()

setVerbose

public void setVerbose(boolean v)

createObject

public static java.lang.Object createObject(org.w3c.dom.Element e)
                                     throws java.lang.Exception

Deprecated.

Creates an object from the given XML Element. If the element has the 'class' attribute, makes a new instance of the class given in that attribute. The element is created either by using the constructor with no parameters, if the element does not have 'param' child elements. If the element has one or more 'param' child elements an Object is created from each of those params. The parameters are put in an Object array and the object array is given as a parameter to the constructor. If the element does not have the 'class' attribute, the method returns the element itself. This means that, if the 'param' child element does not have the 'class' attribute, the xml markup is given as a parameter to the constructed object. For example: If this method is called with the 'o' element in the example, an instance of com.foo.MyObject is created using a constructor that takes an Object array as a parameter. The array will have two objects. The object with index 0 is an instance of com.foo.MyParam and the object with index 1 is an instance of org.w2c.dom.Element, the xml element <param number="3"/>. The com.foo.MyParam is instantiated with one object in the Objects array, the xml 'param' element containing two 'a' elements. Calling with the 'o2' element will also produce an instance of com.foo.MyObject, but this object is instantiated with the constructor taking no parameters.

Throws:

java.lang.Exception