Experimental browser for the Atmosphere
Once a quiet piece of internet plumbing, robots.txt is now in the spotlight, write Audrey Hingle and Mallory Knodel. AI has turned this humble file, long used to guide web crawlers for search, research, and more, into ground zero for debates about consent, control, and digital exploitation.
Apr 3, 2025, 4:18 PM
{ "uri": "at://did:plc:k3sjdodbk2cofbfzxucculeb/app.bsky.feed.post/3llwckf3epu2k", "cid": "bafyreifgzqgwt4hoxqhvjmnsyusp47wa34tg3ppkhulkldlgjlgp7g6xvi", "value": { "text": "Once a quiet piece of internet plumbing, robots.txt is now in the spotlight, write Audrey Hingle and Mallory Knodel. AI has turned this humble file, long used to guide web crawlers for search, research, and more, into ground zero for debates about consent, control, and digital exploitation.", "$type": "app.bsky.feed.post", "embed": { "$type": "app.bsky.embed.external", "external": { "uri": "https://buff.ly/mESIm8J", "thumb": { "$type": "blob", "ref": { "$link": "bafkreigxwocrjon7llahe3cnnvelz44mwklom7ufjxfuy43othmwkh2jfy" }, "mimeType": "image/png", "size": 844941 }, "title": "Robots.txt Is Having a Moment: Here's Why We Should Care | TechPolicy.Press", "description": "Once a quiet piece of internet plumbing, robots.txt is now in the spotlight, write Audrey Hingle and Mallory Knodel." } }, "createdAt": "2025-04-03T16:18:59.702Z" } }