To measure the exact size of the world wide web (i.e. a census). The measure used is the number of publicly accessible web servers on port 80.
Every IP address on the internet is queried for the presence of a web server.
The census found 18,560,257 web servers.
Any web servers hidden behind a firewall, or that did not respond within a reasonable amount of time (20 seconds) were not counted by the census.
Whenever a server is found, we download and store a copy of its homepage. The resulting database of homepages is a historical snapshot of the web which will be mined for information in the future.
Past web surveys performed by various research groups were only estimates of the size of the web. This is the first time its size has been exactly measured.
