Parse and Test Robots Exclusion Protocol Files and Rules

Documentation for package ‘spiderbar’ version 0.2.2

spiderbar-package	Parse and Test Robots Exclusion Protocol Files and Rules
can_fetch	Test URL paths against a 'robxp' 'robots.txt' object
crawl_delays	Retrive all agent crawl delay values in a 'robxp' 'robots.txt' object
robxp	Parse a 'robots.txt' file & create a 'robxp' object
sitemaps	Retrieve a character vector of sitemaps from a parsed robots.txt object
spiderbar	Parse and Test Robots Exclusion Protocol Files and Rules