I need some help with robots.txt.
Under Privacy Settings, I selected “I would like my blog to be visible to everyone, including search engines…”
As I understand it, WordPress automatically creates a robots.txt file. But, I need to exclude one page from the automatically generated robots.txt file. Let’s say the page is called, “myPage”. How do I keep myPage from being indexed?
If I make my own static robots.txt will it be overwritten by WordPress’ dynamic one?
Also, does WordPress’ auto-generated robots.txt exclude files like wp-admin, that shouldn’t be indexed?
Thank you!
-Laxmidi
you can just make a file in your root named robots.txt if you don’t already have one
User-agent: *
Disallow: /myPage
is the format for that one page
there is no allow, only disallow. disllowing pages you follow the above format
WP may not have generated a robots.txt, I believe it only would have if you wanted your site not indexed
I’m not sure if WP has inbuilt mechanism for not indexing your admin stuff….but mine’s never been indexed. If you are worried, to block entire directories you add
Disallow: /wp-admin/
it’s one entry per line/disallow