View Menu

Technical Support Forums

Free, outstanding support from WebAssist and your colleagues

stopping robots from taking data

Thread began 8/13/2009 9:51 pm by aaron322044 | Last modified 8/18/2009 10:52 am by aaron322044 | 1364 views | 4 replies

Danilo Celic

If what is hitting you is really a search engine robot, then perhaps you can use robots.txt to limit or prevent parsing of certain areas of your site.

Assuming that the pages are open for anyone to see (as in not password protected), then the quick answer is that you can't stop someone from doing that. All you can really do is make it more cumbersome to accomplish what they are trying to do. I've not needed to do this myself, so not sure what is really effective, but I'd suggest reading up on "throttling". As you supplied PHP links, here's a start:
#hl=en&q=php+throttling

One thing I immediately thought of was to track requests by IP address and if more than XXX requests in YY seconds, then make the requests by that IP address take longer, perhaps using sleep() With that, you'd need some way to track the number of requests over a period of time, perhaps with a database. Settings a session value will likely not be any good as the robot is probably making individual requests and not saving any session state on it's end.

Maybe someone else has some good suggestions.

Build websites with a little help from your friends

Your friends over here at WebAssist! These Dreamweaver extensions will assist you in building unlimited, custom websites.

Build websites from already-built web applications

These out-of-the-box solutions provide you proven, tested applications that can be up and running now.  Build a store, a gallery, or a web-based email solution.

Want your website pre-built and hosted?

Close Windowclose

Rate your experience or provide feedback on this page

Account or customer service questions?
Please user our contact form.

Need technical support?
Please visit support to ask a question

Content

rating

Layout

rating

Ease of use

rating

security code refresh image

We do not respond to comments submitted from this page directly, but we do read and analyze any feedback and will use it to help make your experience better in the future.

Close Windowclose

We were unable to retrieve the attached file

Close Windowclose

Attach and remove files

add attachmentAdd attachment
Close Windowclose

Enter the URL you would like to link to in your post

Close Windowclose

This is how you use right click RTF editing

Enable right click RTF editing option allows you to add html markup into your tutorial such as images, bulleted lists, files and more...

-- click to close --

Uploading file...