I use this

Disallow: /htsrv/
Disallow: /xmlsrv/
Disallow: /?disp=msgform

Thanks for the advice, sam2kb. :)

Cool, I was searching for some tip like that :)

Thinking about this a little bit more than I probably should. ;)

Is there any danger in adding more directories to the exclude list? For example, if I added the skins directory, would the 'bots not be able to see my blog at all?

Thanks.

I think there's no reason to 'hide' the skins directory.

For example, if I added the skins directory, would the 'bots not be able to see my blog at all?

It's hard to say, since there are no links into skins directory in default b2evo install (except images and scripts which don't get indexed anyway).

sam2kb wrote:

I think there's no reason to 'hide' the skins directory.

My thinking was just that there were so many files uploaded when I installed b2evolution that it would just make it easier on the 'bots if I excluded more directories. And we all want to keep the 'bots happy. :)

My robots.txt is a smidge fuller

User-agent: *
Disallow: /babbles/guestbook/
Disallow: /guestbook/
Disallow: /conf/
Disallow: /getdetail/
Disallow: /getpic/
Disallow: /getthumb/
Disallow: /htsrv/
Disallow: /image/
Disallow: /inc/
Disallow: /install
Disallow: /locales/
Disallow: /media/
Disallow: /pastel-palace/
Disallow: /plugins/
Disallow: /rsc/
Disallow: /xmlsrv/
Disallow: /feeds/
Disallow: /z/
Disallow: /skins/
Disallow: *am_nestCmnt_reply_to
Disallow: *iv_gallery

¥

¥åßßå wrote:

My robots.txt is a smidge fuller

I thought about it some more and went ahead and added all the directories that b2evolution added, since there're so many files that the 'bots don't need to index. My biggest worry was that by disallowing a directory, such as skins, that 'bots wouldn't see the content of my blog.

The blog has only been around a couple of weeks or so, so I don't really have enough data to let me know if the robots.txt file is having any effect.

disallowing skins will not stop bots indexing your blog ;)

¥