StackOverflow stole all our search traffic!

As many of you know, DaniWeb was hit by a Google algorithm update back in November 2012 and we lost about 50% of our search traffic. In investigating the issue, I discovered that DaniWeb, in addition to most other programming forums out there, all lost their google traffic to StackOverflow.

For those who don't know, StackOverflow is a Q&A-based wiki-forum hybrid that was born a few years ago and subsequently received a lot of venture capital. Hands down, it deserves really high listings in Google results: it is fast-loading, lots of people link to it, it has almost exclusively high quality content, and their unique Q&A platform ensures that the best answers are always shown immediately below the question, optimized for those Google searches where you just need to find a quick answer without having to wade through discussions. All in all, no complaints. It deserves to rank high in many cases.

However, back on November 20th 2012, DaniWeb, and many other programming forums, all lost 50% or more of our traffic literally overnight. We were happily receiving about 300,000 pageviews per day for quite some time until we received 330,000 pageviews on Monday, November 19th. We then instantly took a hit down to 230,000 the next day. By Friday the 23rd, we were at 182,000 pageviews. It's been slowly going downhill ever since, and today we're down to averaging a mere 130,000 pageviews daily. I have heard very similar stories from many other programming forums. For example, the programming forum DreamInCode was hit at the exact same time as us by a similar percentage of traffic, as told by their Quantcast report.

Here's a graph of our organic traffic that shows the sudden decrease and subsequent decline:

What makes this story interesting is that at the exact same moment as every other programming community on the web lost their traffic, StackOverflow increased its own traffic proportionally. Within the month, at least according to Quantcast, they went from averaging 2 million uniques per day in October 2012 to 2.7 million in January 2013 and 3.2 million in March 2013.

It gets even more interesting, however, when you compare the traffic decrease that we received from Google organic SERPS to the traffic increase that we received from referrals from StackOverflow! The following graph shows that at the exact same time as traffic from Google to DaniWeb decreased, traffic from StackOverflow to DaniWeb increased by proportional percentages. In other words, we lost half of our traffic from Google, but we doubled the amount of traffic we were receiving from StackOverflow at the exact same time, practically down to the day.

Upon close investigation, it seems that Google is replacing StackOverflow with DaniWeb for most of what we used to rank for. However, it's actually DaniWeb that has the better answers in certain circumstances, and so StackOverflow is actually linking to DaniWeb. What this means is that instead of just linking to the answer on DaniWeb, Google is now linking to StackOverflow who is linking to the answer on DaniWeb. How is this good for Google end-users exactly??

Here are some specific examples:

• Example 1: A blatant link to DaniWeb that directly answers the specific question posed on StackOverflow is the most voted-up answer of all the answers provided. A Google search for the question has StackOverflow in three of the top results, and yet none of the three results even have a direct answer to the question. In fact, the second result was a closed and down-voted question with no answers at all. In Google's defense, DaniWeb did make it to the SERPS here as well, but it's clearly in StackOverflow's favor.
• Example 2: Just like in the previous example, the most upvoted answer is a link to where the question was answered on DaniWeb. A Google search gives the StackOverflow page that points to DaniWeb for the answer #1 in the SERPS while that exact DaniWeb page containing the answer directly is #9 in the SERPS.
• Example 3: This example couldn't be any more cut and dry. DaniWeb is linked to in the most upvoted answer provided, and a Google search gives StackOverflow the top three SERPS results (with the #1 result being a link to find the answer on DaniWeb!) with DaniWeb itself no where to be found in the SERPS.
• Example 4 is just a blatant link to DaniWeb for the answer with DaniWeb no where to be found in StackOverflow-ruled Google SERPS
• Example 5 is just a blatant link to DaniWeb for the answer with DaniWeb no where to be found in StackOverflow-ruled Google SERPS

And that's just the very tip of the iceberg.

C'mon Google, really? If I were into conspiracy theories, I might just think that because so many Google employees use StackOverflow themselves, they've tipped the algorithm to work in StackOverflow's favor in nearly all circumstances.

Want proof? DaniWeb used to host one of the most active C++ communities on the web, bar none. Today, most of our top members are seasoned C++ professionals and even published authors. If I do a simple google search just for the broad keyword "C++" and filter the results to only include Discussions, StackOverflow owns nine out of the ten results on the page! They own the top five results for the broad keyword php and the same for java.

Don't get me wrong, StackOverflow is a great site and deserves to rank well. But they don't deserve to be the only site that deserves to rank well. Not everyone in this world is a StackOverflow junkie, and a single domain serving nine out of ten of the SERPS results is more than a tad biased IMHO.

On a slightly unrelated note, it was brought to my attention at some point in history that people felt like StackOverflow has a higher caliber audience than DaniWeb does, which is the reason it ranks so much better. Judge for yourself: DaniWeb's demographics vs StackOverflow's demographics

Google has created a monopoly here, and on top of StackOverflow receiving millions in venture capital, every other programming community is being slowly squeezed out of existence. StackOverflow deserves to rank, but so do so many other programming communities on the web. There's solid proof here that Google is pushing quality forum pages out of the SERPs in favor of StackOverflow pages that do little more than link to the pages they've replaced.

Discuss ...

The first example your websie shows in #2 while Stack shows in #1. If you cannot keep up with the huge costs (I guess you can't) I think you might need to invest in a little advertising and not just relie on SEO.

[Dani]

The first example your websie shows in #2 while Stack shows in #1.

While I've made sure to ensure that I had personal results turned off when writing this article, Google results do fluctuate and also differ based on geolocation and other such things.

If you cannot keep up with the huge costs (I guess you can't) I think you might need to invest in a little advertising and not just relie on SEO.

We use a lot of different means of driving traffic, not just SEO. As you can read in this article, a significant amount of our traffic comes from other sites (such as StackOverflow!) that link to us. However, as with all Q&A discussion communities on the web (including StackOverflow, of course), SEO is the #1 traffic generator hands down. A huge hit from Google is crippling no matter which way you slice it.

However, the point of this article is not to complain that we were hit. It was to point out that all programming communities were hit at the same time and had all of their SERPs traffic replaced by StackOverflow, to the point of where StackOverflow owns nine out of ten of the google results for the broad keyword C++. Additionally, not only have we lost our traffic, but we lost our traffic to StackOverflow pages that do nothing more than link to the pages on our sites that they replaced.

[NardCake]

The first example your websie shows in #2 while Stack shows in #1. If you cannot keep up with the huge costs (I guess you can't) I think you might need to invest in a little advertising and not just relie on SEO.

I believe Daniweb alreaduy invests in advertising.

I have noticed this, when I first began there were tons of results linking to Daniweb (which were often extremely useful in the end being the reason I registered) but now all I see is StackOverflow and a ton of useless pages. StackOverflow is a very good site as is Daniweb, but sites like Daniweb also need the traffic. The main thing StackOverflow lacks in my opinion is the discussion aspect and intimacy and familiarity between users.

[AussieWebmaster]

Google seems to not care about true value they want homogenized content - in this case they are taking the curated over the original - bad form G

What did the Panda update actually do, did it remove un-quality/uique content? If that is the case how come websites like w3schools still appear in top results..? They do have old content and quite short.

[michel.cote]

This is rather odd indeed... Looks to me as StackOverflow spent a lot of money to get favorable placement. I may be mistaken because search engine placement is not really my strong suit, it's not one of my suits at all actually (lol)...

Keep up the good work though.

[Dani]

From a strictly SEO perspective, StackOverflow has overall higher quality content than the forums they have replaced in the SERPs because they have fewer mispellings, grammar mistakes, and leet speak, than most other forums.

However, this is a double edged sword. StackOverflow is able to achieve this because they have ridiculously stringent rules and moderators manually correct all spelling/grammar mistakes and rewrite question titles, etc. Questions that aren't perfectly written and concise are immediately closed or rewritten. Many feel as if StackOverflow is over-moderated and therefore completely lacks a friendly, community feel to it.

From an SEO perspective, sure, their content is written in perfect sentences with perfect spelling and grammar. Googlebot eats that right up!! But many humans get offended having a site over-moderate what they contribute, and Google needs to understand that when it comes to forums and community discussions (especially global communities where English isn't always necessarily the poster's native language), grammar shouldn't be the end-all be-all major factor.

[Dani]

Looks to me as StackOverflow spent a lot of money to get favorable placement.

That's just the thing. You can't buy Google placements. Or at least, it's never been heard of before.

[mike_2000_17]

I'm starting to hate the whole new "convergence" and "personalized" strategies for google searches and others.

When I search for a coding issue, it is usually some obscure and very specific issue. It is true that SO often offers one or two reasonably related threads that can be useful to me, but very often there are gems to be found in more obscure sources, like less-prevalent forums, old bulletin boards, comment sections, personal pages from enthusiasts, mailing-list archives, or scholarly articles. I have noticed recently (roughly in the time frame you are referring to), that at least the top 5 results on google are almost always SO threads, and typically (for the kind of specific issues I look up), only one SO result is remotely related to the issue I looked up and the rest are unrelated or don't have any useful discussion in it. This is really annoying. I know that SO ranks high (deservingly) in general, but it should not mean that it should rank high for every specific search.

My impression is that the way it used to be, roughly-speaking, is that the top results are those that contain the best match for the search terms, with some conservative filtering out (or pushing down) of less reputable sources. Now, it seems to be reversed quite significantly, i.e., pull up the most reputable sources (most popular sites) and consider the search terms matching almost as a second thought. It is true that the way it was before (and even worse back in the old days (late 90s early 2000s)) there would always be a lot of junk popping up, like link-vessels and other content-duplicating sites, but they were quick and easy to ignore. Now, it's much less junk, but also much less variety.

On a related note, I feel that all this business of personalized searches is also annoying in the same way: I'm not searching the web to find things I already know about. I'd wish there was a button to reverse it completely (as in, "give the results that are as far removed as possible from what I've previously visited").

[Banderson]

Wow Dani, I am sorry to hear this. I have been a member here for a few days. I do not post a lot, but I do frequent the forums. Personally I enjoy this forum more than Stackoverflow.

Looking at your examples, I believe it is the way the answers are linking to Daniweb.

Example 1 : Link Text is "Here"
Example 2 : Doesn't add up at all to me. DaniWeb should clearly be on top in this case.
Example 3: Is the entire link instead of link text.
Example 4: Another entire link instead of link text.
Example 5 : This one doesn't add up either.

I would contact Google and find out why these results are being displayed like this. Maybe they can give you some insight. As a last resort you could always block any referer you wish.
Eventually the linking would stop. I think this could possibly hurt even more short term, but increase traffic like it was in the future.

[|-|x]

I feel that all this business of personalized searches is also annoying in the same way: I'm not searching the web to find things I already know about. I'd wish there was a button to reverse it completely (as in, "give the results that are as far removed as possible from what I've previously visited").

I agree with that sentiment completely. I remember reading something a while back where a Google engineer was talking about the "personalised" search features and mentioned that there are over 50 different metrics used to track user activity for this purpose, including IP address, geolocation and other data that is somewhat more difficult to strip from your browser headers.

I almost always use incognito mode for searches in order to remove as much personal data and history from skewing the results as possible, but it is still ridiculously localised. IMHO the only time this is useful is in searching for a local business such as "pizza" and in most other cases amounts to a form of censorship.

Dani, I found this site years ago via a Google search, but honestly can't say the last time I have seen DW in a result set - I do notice it has become a rarety. In fact my search results are serviced almost exclusively by a small set of websites including the stackexchange family, wikipedia and youtube - anything beyond that I have to really dig around for.

It annoys me to no end that certain aggregator sites containing no original content and often chain-linking to each other (until finally bouncing you back to SO pages you already read earlier in the search) still manage to place highly in the result set. I feel like ever since Google won the search engine war there has been a significant lack of competition (and no I don't think Bing really cuts it, sorry).

I don't really see the point in all their quality algorithms TBH. Most crap sites get weeded out in the long run by the internet community anyway, and short of actually hiring their own editors to curate the entire net they really can't guarantee accuracy or correctness of data at all. Wikipedia is case in point for this, yet somehow they are still viewed as some omniscient authority.

Anyway, sorry if this became a bit of a rant. But it does kind of make one long for the good ole days of simple keyword based search results.

[Dani]

My biggest gripe is that while they definitely deserve to rank well, they don't deserve to be the ONLY site that ranks well. Owning nine out of ten results for all discussion-based results with the broad keyword "c++" is a bit too extreme IMHO.

They seem to be ranking well NO MATTER WHAT, even in cases where they are simply doing little more than linking out to other sites that have the complete answer. The fact that Google is sending traffic to StackOverflow even in cases where the StackOverflow landing page doesn't contain the answer directly is proof that the algorithm is too biased towards the stackoverflow.com domain as a whole.

Google should be spreading the wealth around a bit more. Not everyone appreciates the Stack Overflow format, and Google is creating a monopoly here. IMHO no single broad keyword should ever be nearly completely owned by a single domain unless the keyword is a brand.

[Dani]

I dunno, in my mind, the concept of the search engine is that, at its core, it's supposed to be designed to give the end-user options of where to visit, given the user's specific criteria. Imagine looking up attorneys in the yellow pages and the first complete page of listings are just all different offices for the same law practice. Sure, they might win 90% of their cases, but that doesn't mean that I still don't want to be presented with options. Not everything is suited well for everyone.

Well I agree with Dani, the results should be spread out, but since StackOverFlow is huge there are same / similar questions and I think this is the reason they get ranked more. If you type "c# xml writer" the #2 and #3 are StackOverFlow. When you type C++ stackoverflow does not show. Just remember one thing Dani, the results will be different depending on which country you're from. If you type something simple like HTML, or HTML tables then W3schools is likley to get top results. So I don't think Google has pointed everything there..If they did I'd expect w3schools to disappear as well..

[enam]

Google is killing millions of small websites and favoring a few big websites. This is unfair. This will create monopoly in every field.

Google is killing millions of small websites and favoring a few big websites. This is unfair. This will create monopoly in every field.

Is there any solid proof?

[Dani]

Yes, there has been proof lately that Google's algorithm tends to significantly favor big, established brands. However, as with DaniWeb, it is possible for a small one-or-two person company to grow into a huge online brand without millions of dollars. Remember, having a huge online brand identity and huge online presence doesn't necessarily have to match up with what you would consider a big brand in the real world.

[Msanches]

The word "Stole" is pretty serious accusation. It implies that they have deliberately did something unethical.

I think the explanation is rather simple - Daniweb insults their users with popups in their face and flashing banners while SO just gives people what they want without all the annoying ads.

I don't beleive that this has anything to do with favoring big brand, I think it's that Google has a way to find what people preffer. When Google noticed that most people preffer answers from SO over almost all other programming forums then it started placing the SO threads even higher in search results.

Simple as that. If you want to compete with SO then stop whining and throwing the words "Stole" around and start concentrating on providing the best answers with the least amount of annoyances.

PS. I am going to inform SO about you accusing them of theft and hoping that they will remove all the links to Daniweb from their answers. I think it will be very fair answer to your unfounded and groundless accusations.

[Msanches]

Dani, you asked me a question in the commend, but I cannot reply to a comment. So I have to reply here:
You asked me "if you don't like our flashing banners..."
My questions is - how many people do you know that do like flashing banners and popups?
As for disabling banners in profile, I don't see the reason for that at all. if you have an option to disable banners in profile and you know that most users don't like banners, then why not just disable banners for every registered member and then have an option to enable banners - so those who love banners can enjoy them.

[deceptikon]

so those who love banners can enjoy them.

Spoken like someone who doesn't depend on ad revenue to stay in business. Hop in chat sometime and ask Dani to explain where Daniweb's funding comes from. She'll talk your ear off, but it's an eye opener.

[Msanches]

I just did not like the title of this post. It's so angry and bitter and targeted at Stackoverflow like they are some villains. I'm sure they did nothing to steal anybody's traffic, all they did is created a site the programmers like to go to get answers quickly and usually they get high quality answers.

She said SO "stole all our traffic", then goes on to say that daniweb now receives 130,000 views a day.

[deceptikon]

I just did not like the title of this post.

It caught your attention though, didn't it? :D

[Msanches]

So you saying a deliberately deceptive headline that catches an attention is a good thing?
It's not OK to accuse someone of stealing just to create a catchy headline.
It's just another way to piss off some people who happen to like SO.

[deceptikon]

So you saying a deliberately deceptive headline that catches an attention is a good thing?

I'm only saying that it's effective.

It's just another way to piss off some people who happen to like SO.

How about "Google stole our traffic and gave it to StackOverflow!"?

[Dani]

I meant for the headline to be tongue-in-cheek and catchy. From the article, you can tell that I have no ill feelings towards StackOverflow nor do I think they are doing anything deliberately wrong.

then why not just disable banners for every registered member and then have an option to enable banners - so those who love banners can enjoy them.

Because it's been my honest experience (I promise I'm not making this stuff up) that, when presented with the option to disable advertising via a profile setting, most members choose to leave advertising enabled because they found the website helpful and feel like it is their way of being able to give back instead of via a monetary donation.

Since we are an advertising-supported site and rely on our visitors seeing ads in order to afford our hosting bills, most members feel like it's the least they can do in lieu of a donation.

[|-|x]

@Msanches; Firstly, let me address your comments on advertising by saying that many long time users are happy to have ads displayed in order to support the site. Many online communities achieve this by disabling ads for paying members, so I find it extremely generous and convenient that Dani allows free (but verified) members to disable ads.

Also, Stack overflow does have ad banners as well - the content of which appears to be mostly their own, which demonstrates only that they have reached the point where advertising revenue is no longer necessary to support their continued operation, but is no less annoying than any other ads.

Secondly, while I agree that "stole" is a strong (and perhaps inaccurate) word to use, if you had read past the title you would see that it is not actually used in the article at any point, and that she is in fact presenting evidence of possible collusion between Google and SO - whatever view you take of this theory, I think most of us recognised that the terminology used in the title was somewhat metaphorical.

Furthermore, "preference" has no baring on Googles search results. They track usage history and clickthroughs - but most people click through all links that appear to be vaguely interesting on the first page of a result set, and Google has no way of knowing whether that site was actually useful or not. The sites shown on that first page are algorithmically determined to be relevant based on content. (yes, for community sites membership and activity comes into ranking as well) And it stands that larger sites have more - arguably better quality - content, due in no small part to their capacity to moderate said content. Therefore the bias is in their favour.

In my opinion it is not useful to see 8 or 9 results from the same site on the first page. I would much rather see links to small independant sites/blogs/etc where an expert has written about the specifics of my search criteria, rather than a large forum where 100 noobs have asked the same question and no-one has anything interesting to say.

[Netminder]

Dani,

Welcome to our world. I have to quibble with your headline, though. StackOverflow didn't steal your traffic; Google handed it to them on a silver platter. Since both Mr Spolsky and Mr Atwood don't like EE (despite never having used it), EE was their marketing:

http://www.codinghorror.com/blog/2008/04/introducing-stackoverflow-com.html
http://zurb.com/article/987/stack-overflow-co-founder-jeff-atwood-don ("to take the 'evil' out of Expert's Exchange")
http://zurb.com/soapbox/events/30/Jeff-Atwood-ZURBsoapbox ("It’s awesome to have an enemy. Not in the sense that you’d go out and fight this person, because that’s ridiculous, but it’s nice to have this clear good and evil continuum.")
http://www.joelonsoftware.com/items/2008/04/16.html ("click on the link, and discover that it's a pay site, and the answer is cloaked or hidden or behind a pay-wall, and you have to buy a membership" -- inaccurate, but that didn't stop either of them)
http://www.reddit.com/comments/6g2u2/stackoverflowcom_jeff_atwood_joel_spolsky

But all was not rosy between Stack and Google until this:
http://www.codinghorror.com/blog/2011/01/trouble-in-the-house-of-google.html

Less than four weeks later...
http://www.mattcutts.com/blog/algorithm-change-launched/
http://www.warriorforum.com/adsense-ppc-seo-discussion-forum/323105-matt-cutts-google-algorithm-change-launched.html
https://news.ycombinator.com/item?id=2152286
http://smackdown.blogsblogsblogs.com/2011/01/31/how-matt-cutts-leveraged-the-stack-overflow-and-hacker-news-communities-in-redefining-the-phrase-content-farms/
http://blog.sitesell.com/sitesell/2011/03/itas-dont-be-evil.html

The short answer is that if you got screwed by Panda and its subsequent updates, there is apparently nothing you can do to get Google to change its ranking of you. We have a paywall and you don't, but we both saw the same results (more or less). The implication is that (for all intents and purposes) what Matt Cutts says about you goes. He does not respond to questions on Twitter if they're going to question his judgment. Google will not re-evaluate your site, even if you've fixed everything they said was wrong about you in the first place. Even if you do things to improve the quality of your content (like de-indexing the less-than-stellar content every Q&A site gets), you are forever relegated to the lowest division on Google's scorecard. And if everything (except the subscription vs advertising business models) is equal, then the solution must be that you have to blog a lot, appear at a lot of conferences, and make sure you have Mr Cutts on speed-dial.

But apparently, Mr Atwood is still threatened by EE (or was last October), because he still either doesn't check his facts or flat out lies: http://www.uxmatters.com/mt/archives/2012/10/an-interview-with-jeff-atwood.php
We even get a link in their Wikipedia profile: http://en.wikipedia.org/wiki/Stack_Overflow / http://en.wikipedia.org/wiki/Stack_Exchange_Network

The fallout...
http://www.seo-theory.com/2013/04/10/how-the-google-panda-algorithm-works/
http://www.badgerthoughts.com/random-thought/google-failed-evil-test/ (disclosure: Jason and I are colleagues at EE)
http://www.seroundtable.com/leading-blocked-sites-experts-exchange-14240.html

I don't believe in coincidence. Experts Exchange was (is) unquestionably Stack's ongoing target (because they remember our name), but it's curious that the Panda algorithm took out pretty much every other competitor Stack had.

Eric Peterson / Netminder
Senior Administrator
Experts Exchange

[Dani]

Wow, that's a lot of links. I really don't have the time to read them all now, but here's how I remember the story going (off the top of my head).

• StackOverflow launches
• StackOverflow rolls out an API and a lot of other sites start syndicating its content through its API and public RSS feeds
• StackOverflow gets killed by Google because all of the sites that are syndicating it are outranking it, and it is suffering from duplicate content penalties
• StackOverflow reaches out to Google and lobbies for an algorithm change, stressing that other sites syndicating their content should reflect positively on the quality of their content and is a good thing and they shouldn't be penalized for it
• Soon after, Matt Cutts announces a massive change to Google's algorithm targeting duplicate content, and specifically cites Stack Overflow as being the instigator
• Panda is released soon after that

Whether or not StackOverflow also had an inside track to Panda (or any other Google algorithms) is a thing of conspiracy theorists, but clearly Google has made them out to be the one and only winner in their niche ... and let's face it, online programming communities is not a small niche.

[Netminder]

stressing that other sites syndicating their content should reflect positively on the quality of their content and is a good thing and they shouldn't be penalized for it

But anyone else who has their content stolen should be penalized.

Google has made them out to be the one and only winner in their niche

If it walks like a duck and swims like a duck and quacks like a duck... smile