New search functionality

[Dani]

I've changed our search functionality to be controlled in-house again and no longer use the Google search API. Let me know your thoughts.

[Reverend Jim]

Oodles better than using Google. Super job.

[Gribouillis]

Much better!

[cereal]

Thank you Dani, it works really fine.

[rproffitt]

Just tried it. Thank you for doing that and the front page fixes.

[diafol]

Much better. I actually get some returns from my queries, at last!

[Dani]

I don't think I realized that Google was really THAT bad. They certainly weren't when we first started using them! Goes to show why we rank so poorly in search results. Apparently they have no clue what our content really is.

[diafol]

Why is that?

[Dani]

Because google just doesn't place a lot of value in forums anymore. That being said, they require that we "noindex" the majority of our content. Therefore, a lot of our content just isn't accessible to them anymore. As for why they have such difficulty understanding the context of the remainder of the content, I have no idea.

[cereal]

they require that we "noindex" the majority of our content.

what?! o_o

[diafol]

So SO does that too?

[Reverend Jim]

What type of content are you allowed to index? How do you separate the indexable content from the non-indexable?

You mentioned that tutorials and code snippets are indexable. I would be happy to put together vb.net tutorials/snippets if I had some idea of what would be generally useful.

[Dani]

What type of content are you allowed to index? How do you separate the indexable content from the non-indexable?

You should only index content that Google would consider a superior experience for their searchers. So put yourself into the shoes of a searcher looking for an answer to their specific question. Google tells them a page on DaniWeb has the answer they're looking for. So the searcher clicks through to DaniWeb. If the question has not been answered, it's not valuable to the searcher. If the question doesn't have a clear, concise answer, it's a frustrating experience for the searcher. This is where StackOverflow is such a win and why Google loves them. StackOverflow is designed for Google first: They create such an optimal experience for Google searchers because any page a searcher lands on has a clearly defined question followed immediately by a clearly defined answer. It's all about the optimal experience for the searcher: giving them the quickest answer possible. To that end, we noindex all sorts of content ... questions that haven't been answered, questions that haven't been marked solved, questions whose answers haven't been upvoted, questions not replied to by any of our top members, etc. The list goes on.

[Reverend Jim]

any page a searcher lands on has a clearly defined question

So why don't we delete poorly asked questions? Are you afraid of alienating people who ask poorly defined questions? Look at some of these

• help with multiplying three numbers in VB6

• how do I start making a simple search engine

• example. pointer. int p ; double k ; (p)++ ; (k)++; cout<<"..........................." ;

To that end, we noindex all sorts of content ...

So someone has to manually go through all the questions and answers, post by painstaking post, and decide what is, and is not worthy of indexing? That sounds mind-numbingly brutal. Obviously the mods aren't able to do this so that leaves the admins and there aren't many of those.

[Dani]

So why don't we delete poorly asked questions? Are you afraid of alienating people who ask poorly defined questions?

I'm basically telling you what StackOverflow does. Not that it's something that I think we should do. StackOverflow puts Google first. They alienate newbies so that they can have content perfectly curated for Google. For that, Google absolutely loves them. Newbies, not so much. Because of that, about 100% of their traffic comes from Google. If Google decides to change their algorithm one day, and no longer favor SO, they will be further up a creek without a paddle than DaniWeb ever was, because they have actively alienated all new visitors outside of Google. I want to do whatever we can by way of SEO to encourage Google to like us, but not put Google first, and alienate visitors in the process, the way SO does. Here's a metric for you: DaniWeb has a million registered members. Stack Overflow has five million registered members. That means they have 5X the number of members and about 5,000X the number of visitors from Google. What does that tell you about how much "community" they really are? Or what they would be left with without Google?

So someone has to manually go through all the questions and answers, post by painstaking post, and decide what is, and is not worthy of indexing?

It's all already done via an algorithm.

[Reverend Jim]

DaniWeb has a million registered members

Do you have any stats on how many of those are still active (like within the last 12 months)?

It's all already done via an algorithm

Of course it is, because you are awesome ^_^

[Dani]

Do you have any stats on how many of those are still active

I do, but the numbers are not good. Not good at all.