I do realize that backlinks to URLs disallowed via robots.txt do still have whatever Google's modern version of pagerank is. Depending upon how many incoming backlinks there are, even if Google can't crawl a URL, it might use those external signals to still rank the URL in the search results (albeit with no page title or description).

My question, however, is whether the domain overall is helped? I suspect URLs that are unable to be crawled can't spread pagerank to other pages. By any chance does the domain root get a little lift up? Or does that pagerank completely evaporate at the URL level?

Dani I have never seen in in that way , just two clarification questions if you would like. As I understood you are talking about URLs that are disallowed in robots.txt to be crawled by Google , but what meta name="robots" do the have in their html head ? Also how do you know that it is using those signals to rank those URLs in page results without title and description ? Do you see them in certain queries ? And if so how do they appear , only a URL ?

but what meta name="robots" do the have in their html head

It doesn't matter. If a URL is disallowed via robots.txt, then Googlebot never crawls the page to discover what is or is not in any meta tags in the page's HTML. Regardless, the <meta name="robots" content="noindex"> tag tells Google to not index the URL, but it can see the URL, crawl the URL, find other links pointing out from the URL, etc. Therefore, it's not relevant to my question :)

Also how do you know that it is using those signals to rank those URLs in page results without title and description ? Do you see them in certain queries ? And if so how do they appear , only a URL ?

Yes, they appear just as URLs, as explained here.

I do come across these in the search results from time to time, but it is pretty rare because it can only happen when there are enough external signals pointing to a URL that Google can't access, to make the weight of the external signals enough to overshadow that Google has no clue what is on the page.

The title and description can be pulled by Google without one being there - so same is done when blocked from crawling - if there is a link on the site to the page that does not have a no index no follow wrapped around the link will be how Google gets the page and ranks it.
Google does not follow robots.txt instructions

Google's PageRank system has changed alot over the years, and while not publicly disclosed, some general principles remain relevant. Here's what is understood about how PageRank and link signals work, particularly regarding URLs disallowed by robots.txt:

Impact of Backlinks to Disallowed URLs:

URLs disallowed by robots.txt can still accumulate PageRank from incoming backlinks. While Google cannot crawl these pages, it can recognize the backlinks pointing to them. These backlinks can contribute to the overall authority and ranking signals of the domain.

Distribution of PageRank:

Typically, PageRank flows through links from one page to another. If a URL cannot be crawled, it cannot serve as a conduit to pass PageRank to other pages. Therefore, while the disallowed URL might accumulate PageRank, it doesn't effectively distribute that PageRank to other pages on the domain.

Effect on the Domain:

The authority from backlinks pointing to a disallowed URL doesn't completely evaporate. Instead, it may still contribute to the overall domain authority, providing a slight lift. This means that while the specific disallowed page doesn't pass PageRank, the existence of strong backlinks to any part of the domain can positively impact the domain's reputation and authority in Google's eyes.

Google's Use of External Signals:

Even without crawling a URL, Google can use external signals such as backlinks to infer the importance and relevance of the page. This can indirectly benefit the domain, especially if there are numerous high-quality backlinks pointing to the disallowed URLs.

While URLs disallowed via robots.txt can't directly pass PageRank to other pages through internal linking, they can still accumulate authority through external backlinks. This authority can contribute positively to the overall domain's strength. However, the inability to crawl these URLs limits their direct influence on the distribution of PageRank across the domain.

No, backlinks to disallowed URLs do not benefit your domain. Search engines ignore these links, so focus on building quality backlinks to allowed pages for better overall domain authority and ranking.

commented: Wow - so i can disavow toxic links simply disallowing them? genius. +0
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.