The Whole Ed Catalog

Intelligent Agents

Agents do something for someone else. Intelligent agents -- including 'bots, crawlers, spiders, worms, and others -- are computer programs that traverse the world-wide web, applying their own artificial intelligence to extract and index a variety of information.

Search engines are the most familiar example of intelligent agents. Knowing how these programs work will help you to zero in on the information you're looking for, or to create 'bot-friendly webpages that draw more visitors to your site. Search Engine Watch is an excellent all-around reference on the subject.

The Spider's Apprentice is an excellent review of the top search engines from the user's point of view, including search-refining tips, a search engine FAQ, a web-search wizard, and in-depth analysis and rankings of popular search engines. There is also some material on keyword searching and HTML <META> tags.

Northern Webs' Search Engine Tutorial describes methods for optimizing search engine placement for web designers, through descriptive URLs, title words, keyword selection, summary paragraphs, and more.

We're saving the best for last here, because BotSpot -- an annotated directory of all currently operating 'bots, with links to further information -- may literally be more than you want to know!