I am trying to figure out how I can exclude certain directories from being crawled. I see I have the option to set parameters but none of those seem to work. I am just trying to exclude a folder called /visit/ which contains my affiliate links.
Surely this is an option right? You guys don't mention it anywhere in the documentation so I thought I would ask here. Thanks