Whats a yahoo slurp spider?
And how do you stop them?
check for yahoo.com's webmaster related info on robots.txt file to tell spiders to go away
while checkin board activities, i noticed this spider is on the board.
what does it do?
*edit, google is my friend.
slurp spider indexes pages to be used in searches.
so, does it send out a spider when someone does a search.
or, does it search gathering info, to speed up a future search from someone?
from what i read last night it is the latter.
i also read about some boards that were having issues with too much access allowed for the spiders actually slowin the board down.
It's what happens when you hook up a 7-11 Slurpee machine to the internet.