spider.crawl
Class FetcherPool

java.lang.Object
  |
  +--spider.crawl.FetcherPool

public class FetcherPool

extends java.lang.Object

A pool of multi-threaded fetchers that can be used to fetch many pages at the same time

Author:

Gautam Pant

Field Summary

java.lang.String[]

tags locate and remove these tags from the content

Constructor Summary

FetcherPool(Cache c, RobotExclusion robotsX, Statistics stat)

FetcherPool(int max_fetchers, Cache c, RobotExclusion robotsX, Statistics stat)

Method Summary

void	fetchPages(java.lang.String[] urls) called to fetch one or more pages

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tags

public java.lang.String[] tags

locate and remove these tags from the content

Constructor Detail

FetcherPool

public FetcherPool(int max_fetchers,
                   Cache c,
                   RobotExclusion robotsX,
                   Statistics stat)

FetcherPool

public FetcherPool(Cache c,
                   RobotExclusion robotsX,
                   Statistics stat)

Method Detail

fetchPages

public void fetchPages(java.lang.String[] urls)

called to fetch one or more pages