Disallow /feed and /trackback on wordpress robots but Google still indexing

1 Views

I got a problem with my site and the robots.txt file. I got Disallow for /feed and /trackback url on my robots, but for some reason im still receiving on my Google Webmaster url like http://www.mydomain.net/year/month/post.html/feed and http://www.mydomain.net/year/month/post.html/trackback and I not have idea how to fixe properly. After that happend my site go down on visits.

Can anyone help me, please?

Read More

Regards

This is my robots.txt 

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Allow: /wp-content/uploads/
Disallow: /trackback
Disallow: /?ref=
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$
Disallow: /?s= 
Disallow: /search

User-agent: Googlebot-Image
Allow: /

Sitemap: http://www.mydomain.net/sitemap.xml
Post Views: 1

Related posts

Leave a Reply

2 comments

  1. your robtos file seems to be ok, I think you have misunderstood how to use it.
    You can’t prevent indexing of a certain folder by adding a regexp into a robot file, this is because the robots.txt controls the crawler activity only. So in order to prevent the indexing of certain pages you have to use the robots meta tag.

  2. I think your problem is the use the wildcard character * This is not required just use 

    Disallow: /feed/

    Same applies to all other uses of the wild character.

    Also wild characters are not accepted by all search engines.

    You could try one of the robots.txt syntax checkers for more assistance.