Web IR, topical crawlers, and InfoSpiders

Try MySpiders!

We study Web crawling algorithms. We are particularly interested in adaptive, intelligent agent techniques inspired by ecological and artificial life systems.

InfoSpiders is a class of adaptive multiagent algorithms for autonomous, scalable topical Web crawling. Adaptive Information Retrieval (ch. 7 of Finding Out About by Richard K. Belew) discusses InfoSpiders in the context of adaptive systems for Web IR. InfoSpiders/MySpiders are also featured Web-Searching Agents in AAAI's AI Topics and in UMBC's AgentWeb.

Demo movies

Movies of three InfoSpiders runs on the Encyclopaedia Britannica (the EB5 corpus is described in my dissertation and available on the CD-ROM included with the FOA book)


PI: Filippo Menczer

Former students

Other collaborators

More info

This material is based upon work supported by the National Science Foundation under CAREER grant No. IIS-0133124/0348940. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.