Where is the robots.txt stored for a WordPress Multisite install?

1 Views

Google Webmaster Tools can see it, but I can’t find it. Any idea on how it’s generated or where it is the file structure? I don’t see the file in my root. My website is at http://mikewills.me and the robots.txt file URL is http://mikewills.me/robots.txt.

In the end, Google isn’t indexing my site and I am trying to figure out how to edit the robots.txt so that indexing is allowed. I have changed the privacy option to allow indexing, but that hasn’t updated the robots.txt.

Post Views: 1

Related posts

Leave a Reply

2 comments

  1. It’s dynamically generated by the function do_robots(), which has both an action (do_robotstxt) and a filter (robots_txt). If you create a robots.txt file in your WordPress root, it will (probably) be served up when /robots.txt is requested, otherwise processing will fall back on WordPress.

    Your current file looks like this:

    User-agent: *
Disallow:

Sitemap: http://mikewills.me/sitemaps/mikewills-me.xml.gz

    The first two settings are appropriate for an indexable blog. So, you may just be waiting for Google at this point. Maybe you need to request reconsideration of your site.