Crawler scans websites to find new content and also for evaluating the quality of Web Pages.

An automated piece of software that scans websites. The name reflects how the software “crawls” through the code, which is why they are sometimes also referred to as “spiders”. Crawlers are used by Google to find new content and to evaluate the quality of webpages for their index.

« Back to Glossary Index

Pin It on Pinterest

Share This