A client want that we deliver content via RSS feed, they use cURL to get the feed contents, but they say that they get an 404 error instead. I have tried this command in the terminal: $ curl -g --compressed http://mediosymedia.com/wp-content/plugins/nextgen-gallery/xml/media-rss.php > temp.xml
and as the client says I get the 404 page instead of the feed. When I type the URI in the browser it shows the feed without problem.
I cannot change anything in the client app, so, how can I ensure that they get the feed instead of the 404 error?
Thanks!
Indeed, the
curl
returns a 404 status page:Many webservers will be suspicious of requests without a browser
User-Agent
because they expectcurl
to be used for scraping. This is probably not the smartest technique because a simple UserAgent spoofing will fix that problem:So, in practice, make sure you set up a User-Agent for your requests that is not Curl’s.
My initial though was that this may be related to cookies (see this question), but this may be a localized issue. This is working fine from my machine:
CORRECTION:
Thanks to Julien for pointing out that the contents of the downloaded file was the custom 404 page contents. As he mentions, you need to add a useragent flag (
-A
) to yourcurl
requests:I would just delete my answer, but it’s worth leaving up as a warning to others who might be experiencing this issue – make sure you validate the response!