How to set up robots.txt file for WordPress

[UPDATE 2013]

I can’t find an authoritative page with a format for robots.txt file for WordPress. I promise to maintain one on my site but I want one here on stack overflow.

Read More

If you know what your doing please check current draft here:

http://mast3rpee.tk/?p=127

Everyone else comment on this:

robots.txt

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Sitemap: http://domain.com/sitemap.xml
Crawl-delay: 4

Related posts

Leave a Reply

4 comments

  1. User-agent: *
    Allow: /
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content
    Disallow: /e/
    Disallow: /show-error-*
    Disallow: /xmlrpc.php
    Disallow: /trackback/
    Disallow: /comment-page-
    Allow: /wp-content/uploads/
    
    User-agent: Mediapartners-Google
    Allow: /
    
    User-agent: Adsbot-Google
    Allow: /
    
    User-agent: Googlebot-Image
    Allow: /
    
    User-agent: Googlebot-Mobile
    Allow: /
    
    Sitemap: http://yoursite.com/sitemap.xml
    

    I think this code is very authentic for robots.txt file, Just go to Public_HTML and create file with robots.txt and paste above code.
    You can make in your Notepad, just copy above code and paste into notpad but remember file name should robots.txt and upload to your public_HTML.

  2. Create in notepad robots.txt and upload it to public_html in CPANEL .

    *remember rename your file notepad to robots before you upload it to public_html