SEO

See if a path is allowed.

Check a path against robots.txt.