Davriellelouna@lemmy.world to Technology@lemmy.worldEnglish · edit-26 days agoThe AI company Perplexity is complaining their bots can't bypass Cloudflare's firewallwww.searchenginejournal.comexternal-linkmessage-square90linkfedilinkarrow-up114arrow-down10
arrow-up114arrow-down1external-linkThe AI company Perplexity is complaining their bots can't bypass Cloudflare's firewallwww.searchenginejournal.comDavriellelouna@lemmy.world to Technology@lemmy.worldEnglish · edit-26 days agomessage-square90linkfedilink
minus-squareubergeek@lemmy.todaylinkfedilinkEnglisharrow-up0·5 days agoAnd I’m assuming if the robots.txt state their UserAgent isn’t allowed to crawl, it obeys it, right? :P
minus-squareKissaki@feddit.orglinkfedilinkEnglisharrow-up1·5 days agoNo, as per the article, their argumentation is that they are not web crawlers generating an index, they are user-action-triggered agents working live for the user.
And I’m assuming if the robots.txt state their UserAgent isn’t allowed to crawl, it obeys it, right? :P
No, as per the article, their argumentation is that they are not web crawlers generating an index, they are user-action-triggered agents working live for the user.