org.wandora.application.tools.extractors.datum

Class CrawlerDataSource

java.lang.Object

org.wandora.application.tools.extractors.datum.CrawlerDataSource

All Implemented Interfaces:

ConfigurableDataSource, DataSource

public class CrawlerDataSource
extends java.lang.Object
implements ConfigurableDataSource

DataSource that can be used to process a predetermined list of URLs or a list of urls that can grow as it is processed. To support such crawling, you must provide the crawler thread that is responsible for adding urls to the list. Call setCrawler to set the crawler thread and start the thread before you start using this DataSource. The next method will block while the list is empty but crawler thread is still alive. When the crawler dies (or if it is null to begin with) and list becomes empty, next will return null. You may want to override makeDataStructure method.

Author:

olli

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.Thread	crawlerThread
protected ExtractTool	extractTool
protected int	processed
protected java.util.LinkedList	queue

Constructor Summary

Constructors

Constructor and Description
CrawlerDataSource() Creates a new instance of URLDataSource
CrawlerDataSource(java.util.Collection urls)

Method Summary

Modifier and Type	Method and Description
void	addToQueue(java.lang.Object o)
javax.swing.JPanel	getConfigurationPanel()
double	getProgress()
java.lang.String	getSourceName()
void	guiCancel()
void	guiStart()
protected DataStructure	makeDataStructure(java.lang.Object o)
DataStructure	next(Logger logger)
void	setCrawler(java.lang.Thread crawlerThread)
void	setExtractTool(ExtractTool extractTool)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

processed

protected int processed

queue

protected java.util.LinkedList queue

crawlerThread

protected java.lang.Thread crawlerThread

extractTool

protected ExtractTool extractTool

Constructor Detail

CrawlerDataSource

public CrawlerDataSource()

Creates a new instance of URLDataSource

CrawlerDataSource

public CrawlerDataSource(java.util.Collection urls)

Method Detail

getProgress

public double getProgress()

Specified by:

getProgress in interface DataSource

addToQueue

public void addToQueue(java.lang.Object o)

setCrawler

public void setCrawler(java.lang.Thread crawlerThread)

makeDataStructure

protected DataStructure makeDataStructure(java.lang.Object o)

next

public DataStructure next(Logger logger)
                   throws ExtractionException

Specified by:

next in interface DataSource

Throws:

ExtractionException

getConfigurationPanel

public javax.swing.JPanel getConfigurationPanel()

Specified by:

getConfigurationPanel in interface ConfigurableDataSource

guiCancel

public void guiCancel()

guiStart

public void guiStart()

setExtractTool

public void setExtractTool(ExtractTool extractTool)

Specified by:

setExtractTool in interface ConfigurableDataSource

getSourceName

public java.lang.String getSourceName()

Specified by:

getSourceName in interface ConfigurableDataSource