1 lano1106 May 29, 2007 00:45
3 lano1106 May 29, 2007 06:37
Hi whoo,
No, I cannot find the link neither. That is why I asked the question in this forum! After reading your reply, I havec checked the main page output. I have found that there was few links that were similar to the one found by Google spider such as the links for register/login. That would be weird but would it be possible that this spider tries to interpret URLs it is seeing to extrapolate other possibly valid URLs such as when it sees 'redirect_to=' maybe it just take the parameter value and try to construct another URL from it.
Anyway, I have edited my robots.txt file. Thank you for the advice.
I cant find the link on your front page, (I use Firefox and it has this magical thing under tools called "page info" that lets you look at all outgoing links). Try it (Firefox) sometime, it's awesome.
That said, you dont want google spidering that directory anyway, so rather than worrying about it being there (or not), use a robots.txt to prevent it from being spidered in the first place.