Construct · 1 fields · 1 citations

spider: a standalone construct

One name. One construct. A landmark on the map.

● anchor · a standalone construct Nearest construct by meaning·not indexed
spider1 CONSTRUCTWeb crawlerHEYDON & NAJORK · 1999

A single standalone construct; nearby constructs are located against it in the Collider. Tap a field for its definition, unit, and measures.

Web search engineering

A program that crawls and indexes the web

In web search engineering, a spider (or crawler) is a program that systematically fetches web pages, follows their links, and gathers content to build a search index, traversing the web's link graph much as a spider moves across its web. Heydon and Najork's Mercator described a scalable, extensible crawler and the engineering (queueing, politeness, and deduplication) needed to crawl at scale. The referent is an automated web-traversal program, named for the arachnid only by the metaphor of moving across a web.[1]

Unit
The program and the rate and coverage of its crawl across pages
Recovery means
Not applicable
Measured by
Characterised by crawl rate, coverage and freshness of the pages fetched, and politeness toward servers

Next

Run it through the Collider, or take the citations with you.

See how spider sits against neighbouring constructs in the Collider, or export every reference on this page in one click.

Open in ColliderExport BibTeXReport this entry

Nearby constructs

agent3 fields · 5 citationsjingle · one word, three constructsbody2 fields · 3 citationsjingle · one word, two constructsindex5 fields · 11 citationsjingle · one word, five constructs

References

  1. Heydon, A., & Najork, M. (1999). Mercator: A scalable, extensible Web crawler. World Wide Web, 2(4), 219–229. https://doi.org/10.1023/A:1019213109274

Cited on this page?

Confirm your construct, or correct us.

If your work is cited behind one of these entries, you know better than we do whether we captured the meaning right. Confirm it, suggest an edit, or dispute it, and your name goes on the entry as an author who has checked it, linked to your ORCID.

Claim your construct