A small but unusually instrumented indexing experiment has produced a striking pattern: every tested page that Googlebot fetched entered Google’s index, while every page Googlebot failed to fetch remained outside it.
The September 11 test published by AskedAbout tracked 14 previously unindexed news URLs after the site introduced a redesigned internal-linking hub. Fourteen days later, eight pages were reported as indexed in Google Search Console. Server logs showed that all eight had received at least one verified Googlebot request after the hub launched. The other six had received none, and all six were still unindexed.
The result does not prove that crawling guarantees indexing, nor does it prove that the new hub caused Googlebot to visit the pages. It does, however, illustrate a distinction that can get lost when SEOs focus only on Search Console’s indexing labels: on this site, the immediate bottleneck was not what happened after Googlebot processed a page. It was getting Googlebot to request the URL in the first place.
The experiment started with 14 unindexed news posts
AskedAbout changed its /news architecture on August 28 after observing a crawl stall. Thirteen posts published between August 20 and August 27 had received zero Googlebot requests during the preceding seven days. A sweep of all 110 news URLs showed 96 indexed and 14 not indexed, so the company fixed those 14 URLs as the test denominator before evaluating the result.
The intervention added a month-sectioned news index listing every post, while individual articles gained rel=next and rel=prev links to neighboring posts. AskedAbout also preregistered a simple success threshold: if at least seven of the 14 URLs were indexed at the September 11 read, the test would score the hub as working; zero would suggest internal linking was not the constraint; an intermediate result would be treated as inconclusive.
That preregistration is useful because it prevents the success criterion from being rewritten after seeing the outcome. But the label “hub works” needs more caution than the raw scoring rule implies, because the design cannot isolate the hub as the cause of the subsequent Googlebot visits.
Eight fetched pages became indexed; six unfetched pages did not
At the final September 11 read, eight of the 14 URLs were listed as “Submitted and indexed” by the Search Console URL Inspection API. Each of those eight had at least one Googlebot request recorded in AskedAbout’s server-side crawler ledger after the August 28 deployment.
The relationship was exact within this sample. Eight fetched pages were indexed. Six pages with zero Googlebot requests were not indexed. AskedAbout says this same one-for-one pattern held at each intermediate read after the experiment began.
On September 6, two URLs had been fetched and both were indexed, while the twelve unfetched URLs were still out. By September 10, the count had reached seven indexed pages. One additional page received its first Googlebot request on September 11 and was indexed by the final read that afternoon.
The server-log evidence makes the observation more useful than a Search Console status screenshot alone. A page marked “Discovered, currently not indexed” can lead site owners to speculate about content quality, duplication or indexing thresholds. In AskedAbout’s sample, the discovered or unknown state coincided with an empty lastCrawlTime and zero verified Googlebot requests for every affected URL.
Indexing followed the first observed Googlebot fetch quickly
The timing data also suggests that processing after the first fetch was relatively fast for the cases that could be bracketed by consecutive measurements. One page was fetched at 14:48 UTC on September 10 and appeared indexed in a 16:39 read, a gap of less than two hours. Another was fetched at 13:03 on September 11 and appeared indexed by 16:38, less than four hours later. An earlier measured case was detected as indexed within fourteen hours of its fetch.
AskedAbout also reports that Search Console’s lastCrawlTime for all eight indexed pages matched the first post-hub Googlebot request to the second, including pages that its server logs showed Googlebot fetched again later.
That does not mean all eight pages were definitively indexed between two and fourteen hours after crawling. Only several observations had reads close enough to bracket the transition. The safer conclusion is that where the delay could be measured, indexing was observed within hours rather than days.
For this particular site, once Googlebot arrived, the measured indexing problem largely disappeared.
The experiment cannot prove the hub caused those fetches
The strongest limitation is the absence of a control group. Every one of the 14 previously unindexed pages received the new hub treatment. No comparable group of stalled URLs was deliberately left without the additional links, so there is no untreated baseline showing what Googlebot would have done over the same two weeks.
That means the eight new fetches could reflect the hub, Googlebot’s normal recrawl timing, another discovery path or a combination of factors. AskedAbout’s server ledger does not record a crawler referrer that would identify which link led Googlebot to each URL.
The timing is suggestive but not causal. The thirteen stalled posts had recorded zero Googlebot requests in the seven days before the hub shipped, then eight first requests appeared among the test URLs during the following fourteen days. That sequence is consistent with the intervention helping, but it cannot establish that the intervention produced the change.
The site’s own follow-up evidence makes that uncertainty even clearer.
Googlebot had stopped visiting the hub itself
The redesigned /news hub received three Googlebot requests after launch: once on September 2 and twice on September 3. But during the seven days immediately preceding the September 11 measurement, Googlebot did not request the hub at all.
Other crawlers did. AskedAbout counted 63 requests to the page from non-Google crawlers during that seven-day period, including Meta’s external agent, OAI-SearchBot, ChatGPT-User, PerplexityBot, Bingbot and Claude-User.
If Googlebot did not fetch the hub during the week when most of the tested URLs received their first requests, the experiment cannot simply say Googlebot visited the hub and followed its links. It may have discovered the URLs through an earlier hub crawl, another internal page, a sitemap or another route entirely.
This is why the most interesting result is not “internal linking solved indexing.” The logs do not demonstrate that. The stronger observation is that Googlebot visitation separated the indexed and unindexed groups perfectly in this sample.
Eight newer linked posts remained completely unfetched
A second set of pages provides an important counterweight to the apparent hub success. AskedAbout published eight new posts between September 7 and September 10. Each was added to the news hub within its first hour, included in the sitemap and pinged through IndexNow in the same publishing workflow.
At the September 11 read, none of the eight had been indexed. More importantly, none had received a Googlebot request. Six were shown as discovered and two were unknown to Google, but the server logs contained zero Googlebot fetches across the entire group.
That result shows why simply adding a link to a hub cannot guarantee discovery on a useful schedule. An internal link can only become a crawl path when Googlebot visits a page containing that link or learns about the target through another channel. If the hub itself is not being recrawled, freshly added links may sit there without immediately changing crawler behavior.
The sitemap did not produce an immediate fetch either. Sitemaps help search engines discover and prioritize URLs, but submission is not a command to crawl on demand.
“Discovered, currently not indexed” may sometimes mean exactly what it says
The test offers a useful diagnostic lesson for Search Console. SEO discussions often treat “Discovered, currently not indexed” as a broad quality verdict, but the label literally allows for a simpler possibility: Google knows the URL exists and has not crawled it yet.
On AskedAbout’s test pages, that was observable directly. URLs in discovered or unknown states had no Googlebot request in the site’s verified server ledger and no lastCrawlTime in the URL Inspection API. Once a tested URL received its first post-deployment Googlebot fetch, it appeared indexed at the next measured read.
This pattern should not be generalized to every website. Google can crawl pages and decide not to index them because of duplication, canonicalization, quality, robots directives, rendering issues or other signals. Plenty of site owners can find examples where Googlebot fetched a URL repeatedly without indexing it.
But before diagnosing an indexing problem as a content problem, checking whether Googlebot has actually fetched the page can eliminate an entire category of speculation.
Server logs can answer a question Search Console alone cannot
Search Console remains the primary first-party tool for checking Google indexing, but server logs add a different kind of evidence. They show whether a crawler actually requested a path and when that request occurred.
Combining the two allowed AskedAbout to distinguish discovery from crawling and crawling from indexing. Search Console supplied the index state and lastCrawlTime; the site’s server ledger supplied verified Googlebot requests by URL and timestamp.
For technical SEO teams investigating persistent indexing gaps, that combination can be more informative than repeatedly clicking “Request indexing.” If an important page has never received a Googlebot request, the first problem to investigate is discovery and crawl access. If Googlebot has fetched it repeatedly and it remains excluded, the investigation shifts toward canonicalization, duplication, quality and other indexing decisions.
Internal links only help crawling when crawled pages contain them
The experiment also illustrates a basic architectural principle that becomes easy to overlook: a link sitting on an unvisited page cannot lead a crawler anywhere.
Internal-link audits often measure link counts, depth and anchor text as static properties of a website. Those metrics matter, but crawl behavior adds a temporal dimension. A category page may contain perfect links to every new article, yet those links cannot influence a crawler until the category page is fetched again or the targets are discovered elsewhere.
That is especially relevant for sites with large archives, weak authority or low crawl demand. Improving architecture may make the site easier to traverse without necessarily forcing Googlebot to traverse it immediately.
The AskedAbout data therefore points toward a more useful question than “Does this page have internal links?”: “Are the pages carrying those links actually being visited by Googlebot often enough to discover new URLs?”
The next experiment is designed to test that distinction more directly
AskedAbout has preregistered a follow-up test on the six remaining unfetched URLs. Beginning September 12, three posts that Googlebot already revisits are being given dated body links to the six remaining pages, without changing their titles, slugs or introductions.
The next read is scheduled for September 19. If at least three of the six URLs receive a Googlebot request, AskedAbout will interpret that as evidence that links from actively fetched pages can move the crawler. If none is fetched, the site plans to treat broader crawl demand as the constraint rather than continuing to experiment with link shape.
That design still will not replicate a laboratory-controlled experiment, but it targets the weakness exposed by the first test: a hub cannot act as an effective discovery surface during periods when Googlebot does not visit it.
The real bottleneck on this site was getting Googlebot through the door
The headline result is unusually clean: among 14 stalled URLs, all eight that received a Googlebot fetch were indexed, and all six that received no fetch remained out. In the cases where the transition could be closely measured, indexing followed within hours.
But the experiment does not justify a universal rule that Google indexes every page it crawls. Nor does it prove that the new internal-linking hub caused the eight successful fetches. The missing control group, absence of crawler referrer data and lack of recent Googlebot visits to the hub prevent that causal conclusion.
What the test does show is a practical diagnostic sequence. Before rewriting an unindexed article, adding more schema or debating whether Google dislikes the content, establish whether Googlebot has actually requested it. On AskedAbout’s measured URLs, that one distinction explained the entire split between pages inside and outside the index.
For technical SEO, that is a useful reminder: indexing is downstream of crawling. Sometimes the most important optimization is not persuading Google to index a page it has evaluated. It is getting Googlebot to show up and evaluate the page at all.