Je hebt een klacht over de onderstaande posting:
Het lijkt alsof de zoekmachines moeten betalen: $ w3m -dump www.facebook.com/robots.txt # Notice: if you would like to crawl Facebook you can # contact us here: http://www.facebook.com/apps/site_scraping_tos.php # to apply for white listing. Our general terms are available # at http://www.facebook.com/apps/site_scraping_tos_terms.php User-agent: baiduspider Disallow: /ac.php Disallow: /ae.php Disallow: /album.php Disallow: /ap.php Disallow: /autologin.php Disallow: /checkpoint/ Disallow: /feeds/ Disallow: /hacked/ Disallow: /l.php Disallow: /o.php Disallow: /p.php Disallow: /photo.php Disallow: /photo_comments.php Disallow: /photo_search.php Disallow: /photos.php User-agent: Googlebot Disallow: /ac.php Disallow: /ae.php Disallow: /album.php Disallow: /ap.php Disallow: /autologin.php Disallow: /checkpoint/ Disallow: /feeds/ Disallow: /hacked/ Disallow: /l.php Disallow: /o.php Disallow: /p.php Disallow: /photo.php Disallow: /photo_comments.php Disallow: ...
Beschrijf je klacht (Optioneel):