Web IR, topical crawlers, and InfoSpiders

Try MySpiders!

We study Web crawling algorithms. We are particularly interested in adaptive, intelligent agent techniques inspired by ecological and artificial life systems.

InfoSpiders is a class of adaptive multiagent algorithms for autonomous, scalable topical Web crawling. Adaptive Information Retrieval (ch. 7 of Finding Out About by Richard K. Belew) discusses InfoSpiders in the context of adaptive systems for Web IR. InfoSpiders/MySpiders are also featured Web-Searching Agents in AAAI's AI Topics and in UMBC's AgentWeb.

Demo movies

Movies of three InfoSpiders runs on the Encyclopaedia Britannica (the EB5 corpus is described in my dissertation and available on the CD-ROM included with the FOA book)

An easy query: QuickTime (9.6 MB) or gzipped (1.3 MB)
An impossible query: QuickTime (4.4 MB) or gzipped (272 KB)
An ambiguous query: QuickTime (11.1 MB) or gzipped (2.2 MB)
Single gzipped archive (3.7 MB) of the 3 movies

People

PI: Filippo Menczer

Former students

Other collaborators

More info

Papers on InfoSpiders and related projects
Support script and files associated with paper on topical crawler evaluation framework
Java library for topical crawlers associated with paper on crawling infrastructure
First demo of InfoSpiders presented at Agents'97, the First International Conference on Autonomous Agents
1996: InfoSpiders/0.1 announcement
First paper: CIKM Workshop on Intelligent Information Agents (1994)
Original project overview (1994)

This material is based upon work supported by the National Science Foundation under CAREER grant No. IIS-0133124/0348940. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.