{"description": "Remove any <tt>robots.txt</tt> files that may exist with any web content.\nOther methods must be employed if there is information on the web site that\nneeds protection from search engines and public view. Inspect all instances of\n<tt>DocumentRoot</tt> and <tt>Alias</tt> and remove any <tt>robots.txt</tt> file.\n<pre>$ sudo rm -f path/to/robots.txt</pre>", "rationale": "Search engines are constantly at work on the Internet. Search engines are\naugmented by agents, often referred to as spiders or bots, which endeavor to\ncapture and catalog web-site content. In turn, these search engines make the\ncontent they obtain and catalog available to any public web user.\n<br /><br />\nTo request\nthat a well behaved search engine not crawl and catalog a site, the web site may\ncontain a file called robots.txt. This file contains directories and files that\nthe web server SA desires not be crawled or cataloged, but this file can also be\nused, by an attacker or poorly coded search engine, as a directory and file\nindex to a site. This information may be used to reduce an attacker's time\nsearching and traversing the web site to find files that might be relevant. If\ninformation on the web site needs to be protected from search engines and public\nview, other methods must be used.", "severity": "medium", "references": {}, "control_references": {}, "components": [], "identifiers": {}, "ocil_clause": "it is not", "ocil": "Inspect all instances of <tt>DocumentRoot</tt> and <tt>Alias</tt>. No\n<tt>robots.txt</tt> file should exist.", "oval_external_content": null, "fixtext": "", "checktext": "", "vuldiscussion": "", "srg_requirement": "", "warnings": [], "conflicts": [], "requires": [], "policy_specific_content": {}, "platform": null, "platforms": [], "sce_metadata": {}, "inherited_platforms": [], "cpe_platform_names": [], "inherited_cpe_platform_names": [], "bash_conditional": null, "fixes": {}, "title": "The robots.txt Files Must Not Exist", "definition_location": "/aptdata/openscap/scap-security-guide/linux_os/guide/services/http/securing_httpd/httpd_secure_content/httpd_remove_robots_file/rule.yml", "template": null}