User Agent Strings

Boitho's Web Crawler, a distributed crawler that downloads web pages to build the database used by to search in. To allow volunteers to donate their superfluous bandwidth and idle CPU time, they have developed a distributed crawler, like [email protected] and Grub. That way people can install a program on their computers and help them with the crawling.

Click on any string to get more details 0.85 0.83 0.82 0.81 0.79