Should I cache RSS feeds?

[Dani]

Our RSS feeds tend to get a decent amount of traffic. I would love to cache them, but that would break PubSubHubbub. In other words, PubSub would provide real-time notifications to subscribers of the RSS feed, but the RSS feed would not be updated in real-time when new content was published.

I was thinking of invalidating the cache every time new content was posted, but my understanding is that's really bad form, especially for HTTP Cache-control headers that are sending signals to web browsers with expiry times that end up consistently being not true.

[jkon]

I don't know how you generate those RSS Feeds or even if you are already doing what I will propose. If you generate them dynamically each time, you could easily implement a middle-ground solution. You could generate them only when new content is posted, cache those files in your server, and serve them either directly or by using your server-side language to serve the file (streaming it in chunks) at minimal burden. The latter choice also allows you to log who is consuming what, and even to add access controls if needed. This is a common way to save server resources and even make it faster.

[Dani]

My question is not about how to technically implement caching. Essentially I’m asking, more from a usability standpoint, what HTTP Cache-control headers should look like for RSS feeds that both get a lot of traffic and somewhat frequent (though not relatively speaking) updates? Additionally, what the caching directives at the edge should look like?

[chesterloke]

Invalidating the cache on every new post defeats the purpose of caching and can cause inconsistent behavior with Cache-Control headers. A better approach is to use conditional requests or short cache lifetimes, then trigger a background refresh when new content is published, so PubSubHubbub updates stay accurate without breaking caching efficiency.

[jkon]

@Dani, maybe there's something here that I don't understand, or I'm missing. Who are the consumers of your RSS feeds? From my experience, those are mainly robots, but perhaps in this case, it's different. If the consumers are indeed robots, why are you thinking about "HTTP Cache-control headers"? Have you seen many robots respecting those headers? Even for the few well-behaved systems (few major aggregators), there is no way to set a cache signal like max-age on RSS feeds that could be updated without seriously risking stale states.

The middle-ground solution i shared, generating the feed once on publish and caching it, will save server resources. However, you could extend this using your CDN. The most reliable path in my view is to abandon time-based freshness and use instant invalidation: you could set a very long maximum caching TTL for URLs that match a pattern (e.g., /feeds/*) on the CDN, and then invalidate or refresh it instantly when you publish new content to those RSS feeds through the CDN's API. I haven't used the CDN service that I think you use, but from a quick search I did, I think that this is doable. This could be even cheaper in resources because requests will only hit your origin servers when you actually update the feed. Just a proposal...

[Dani]

Invalidating the cache on every new post defeats the purpose of caching and can cause inconsistent behavior with Cache-Control headers.

Yes, that was exactly my confusion.

A better approach is to use conditional requests or short cache lifetimes, then trigger a background refresh when new content is published, so PubSubHubbub updates stay accurate without breaking caching efficiency.

@chesterloke , can you please explain this a bit better? What do you mean by a background refresh, and how does that differ from invalidating the cache?

Who are the consumers of your RSS feeds? From my experience, those are mainly robots, but perhaps in this case, it's different.

A little over half are search engine bots on the hunt for new content, and the remainder are individual feed reader subscriptions.

If the consumers are indeed robots, why are you thinking about "HTTP Cache-control headers"?

Because the bots are currently visiting and re-visiting the same URLs every 1 or 5 seconds polling for new content, the bandwidth being wasted is somewhere in the stratosphere. Cache-control headers would tell them when the cache expires and when it's beneficial to revisit next.

Have you seen many robots respecting those headers?

Of course. Always. You don't??!!

Even for the few well-behaved systems (few major aggregators), there is no way to set a cache signal like max-age on RSS feeds that could be updated without seriously risking stale states.

DaniWeb is not a current events news site or anything that really demands RSS feeds be instantaneously updated. If I could cache each of the feeds for 30 minutes, it would cut the amount of wasted bandwidth by sooooooooo much, and I'm pretty confident that not a soul would notice one of the DaniWeb feeds they subscribe to being ever so slightly stale to the point of where their feed reader notifies them a whopping 10 minutes after a new post is made. We don't currently get nearly as much new content as we used to, so it's not like we're getting new posts every minute, or even new posts every hour right now. We certainly don't get enough new content to make feed readers think they need to be polling each of our feeds every minute. (We have more direct subscribers than those using a PubSub hub. That being said, I don't want to do anything that would actually break PubSub, which is where my dilemma is.)

[Dani]

I suppose it's not a lot, relatively speaking, but RSS feeds account for about 90,000 hits a day, with each request being dynamically generated by querying the database. Meanwhile, to put that in perspective, there was only 1 new post yesterday.

[Dani]

Update: I've decided to remove myself from PubSub and cache each of our RSS feeds for up to an hour.