# https://www.robotstxt.org/robotstxt.html User-agent: Twitterbot Disallow: User-agent: ia_archiver Disallow: /