# Disallow all crawlers access to certain pages. User-agent: * Disallow: /images/ Disallow: /_private/ Disallow: /_vti_*/ Disallow: /personalwebsite/ Disallow: /aspnet_client/ User-agent: aipbot Disallow: / User-agent: BecomeBot Disallow: / User-agent: psbot Disallow: /