Internet Archive will ignore robots.txt files to keep historical record accurate

The Internet Archive will no longer conform to instructions laid out in robots.txt files, as their calibration toward search engines is causing certain sites to be rendered inaccessible.
The post Internet Archive will ignore robots.txt files to keep historical record accurate appeared first on Digital Trends.