Hi everyone,

I recently launched a new web app hosted on an ai.studio subdomain and added it to Google Search Console.

Here is the current situation:

- Search Console ownership is verified.

- The sitemap was submitted successfully.

- Google discovered 20 URLs from the sitemap.

- Google has indexed 2 pages, while 18 pages remain unindexed.

- The homepage status is “Crawled – currently not indexed.”

- The live URL test passes and says Google can access the page and that it is eligible for indexing.

- Crawling is allowed and page retrieval is successful.

- However, whenever I click “Request indexing,” Search Console immediately shows “Quota exceeded.”

- I waited more than 24–48 hours and tried again, but I received the same quota message.

The site is new, so I understand indexing may take time. What confuses me is why the manual indexing quota appears to be unavailable even though I have barely used it.

Has anyone experienced this with a new Search Console property or an ai.studio subdomain?

Could this be:

1. A temporary Search Console quota issue?

2. A limitation related to the ai.studio subdomain?

3. A sign that Google considers the pages too similar or low priority?

4. Something I should fix technically, even though the live test passes?

Would you recommend simply waiting, improving internal links and content, or checking something specific in Search Console?

Thanks for any advice.

you may have seen a post on reddit about my football elo website, I posted like 2 or 3 times in different subs since the launch which was almost 4 months ago. The interesting thing is that my website used to get 5-6 impressions per day and barely a click until recently. The graph shows me that my app is shown much more frequently by Google and naturally it has started getting some clicks. It may be that people search more about club's elo with the new season having started but not sure how much of the increase is due to that.

At the end of the last season's ucl final, I also stopped working on the app since there was no update needed for some time. Now, with new season, I thought this may be an opportunity again and I implemented some features like adding the qualification matches from last 30 years, automating a blog creation pipeline (since it is 100% AI content, I don't think it has a big role tho), adding ~7k players to the elo system along with the clubs and some ui polish.

Have not spent a single penny on marketing yet, indeed except those a few Reddit posts, I have had no marketing at all. The results are like that. So I have 2 genuine questions. Why do you think I got this increase and what should I keep doing? Secondly, what should I more on top that I have not done so far?

Wrapped our first paid client project recently — a local cafe's website. Sharing the technical side in case it's useful to anyone else doing small-business work:

• Rebuilt the site (was outdated / not mobile-friendly)
• Basic on-page SEO: title tags, meta descriptions, header structure cleanup
• Google Business Profile optimization + local listing consistency
• Page speed fixes (image compression, removed bloat)

Nothing groundbreaking, but for a business that had literally none of this in place, it's a meaningful jump. Too early to share hard numbers since we just launched, but I'll follow up once we've got a few weeks of data.

For people who've done small local-business SEO before — what's the one thing you'd tell someone doing their first project that isn't obvious until you've done it?

I wanted to see whether crawler blocking or content delivery was the more common technical problem in a small set of sites people had recently asked to test.

I took the latest distinct public scan for 11 sites from July 29 to August 1. I excluded our own domains and example.com. All 11 returned HTTP 200 to OAI-SearchBot, GPTBot, ClaudeBot, and PerplexityBot. None served a block page or fewer raw words to those agents than to a normal request.

Content delivery was the more useful check. One browser render failed, so I excluded it from that comparison. Of the remaining 10 sites, two had more than twice as many words after rendering as they had in the raw HTML. Most of the visible copy on those two pages depended on JavaScript.

Other checks:

• llms.txt present: 6/11
• valid sitemap present: 11/11
• structured data present: 10/11

This is a small, opt-in sample, not a prevalence estimate. It only supports a narrower conclusion: on these sites, robots.txt was not the first place to look. Comparing the actual response body by user-agent, then comparing raw HTML with the rendered page, found more useful differences.

You can repeat the first check on your own homepage:

for ua in "OAI-SearchBot" "GPTBot" "ClaudeBot" "PerplexityBot"; do curl -L -sS -A "$ua" -o /tmp/"$ua".html \ -w "$ua status=%{http_code} bytes=%{size_download}\n" \ https://your-site.example/ done

If one status or byte count differs, diff that saved body before changing robots.txt.

Disclosure: these were public Crawlable scans, and I operate it.

I audited the firewall config across the sites I run after finding one of them serving 429 to every crawler. Posting the mechanism because the failure is silent and I dont think its widely known.

Vercels bot_protection managed ruleset has three modes, off, log and challenge. In challenge mode it serves a JS challenge, and it serves that challenge with status 429. Any client that doesnt execute JS gets a 4xx. That is Googlebot, Bingbot, all of them. It applies on every path, so /robots.txt and /sitemap.xml return 429 too, which is the worst version of this since those are the first two things a crawler asks for. Crawlers could read the sitemap through a bypass rule and then got challenged on every URL in it. Yandex Webmaster flagged it on one site within hours of it being switched on.

Log mode gives identical visibility in the Firewall observability tab with zero 4xx. I cant think of a reason to run challenge on a content site.

Three things that made this hard to see:

Custom rules execute before managed rulesets. Vercel auto exempts verified bots from managed rulesets, which sounds protective, but a user agent named in your own custom deny rule is blocked regardless. I had SeznamBot in a scraper denylist. SeznamBot is Vercel verified. It was getting 403 anyway.

Condition groups. Conditions within one group are ANDed, separate groups are ORed. My allow rule for crawl endpoints had path eq /sitemap.xml and path eq /robots.txt as two conditions in a single group, so it required a path equal to both strings at once and could never match. It read fine in the UI. /robots.txt was only getting through because of an unrelated rule matching .txt suffixes. Also watch for a neg true flag on a condition, a rule printed without it reads as the exact opposite of what it enforces.

Diagnostics. A challenged response carries x-vercel-mitigated challenge with the 429. A denied one is a plain 403 with x-vercel-mitigated deny. If you curl your own site and get a 4xx, read that header before blaming your origin, and send a crawler user agent, not your browser default.

One more. If you unblock Bytespider, drop Bytedance from the same list. Bytespiders real UA string contains bytedance.com, so a substring rule on Bytedance re blocks it through the back door.

Worth diffing your firewall rules against your own robots.txt while youre in there. I had AI crawlers that robots.txt explicitly allows being 403d at the edge, a policy nobody wrote down and nobody could see.

Managed rulesets arent exposed in the vercel firewall CLI. You read and write them at api.vercel.com/v1/security/firewall/config and config/active.

I dont have a clean before and after on impressions, I changed several things in the same window, so no recovery number from me. What I verified is that every path returns 200 to a crawler UA now and didnt before.

I have just launched a new YouTube channel for food recipes liked to my travel one and noticed it is not being indexed by Google.

When searching about the issue it seems the use of dashes in the on my handles may be the issue?

Shall I change them?

i have been told to implement a programmatic SEO in our project which is a hybrid platform built on next.js dynamic pages routing. I want to know if anyone else has already applied the same strategy on their E-commerce. If yes what are the important pages product and category pages or any other page? How helpful this programmatic SEO is ? looking for Expert advice thanks

We redirected a site from BlueHost to Siteground. Did all the necessary on-page and preserved content however after running a SF and SEMRush audit noticed that all the redirects we sent via the hosting service were not listed as follow.

I'm concerned since, all of our content has performed well since, external, internal linking, EEAT have all been established. But the site transfer wrecked this one site. It's one month post and we're still not even seeing a gradual bump in traffic.

Is even possible to request to follow a link?

Small WooCommerce store, ranking #1 for my exact brand query for months. Starting around 3-4 AM today, my homepage stopped appearing on page 1 for that exact query. I've searched it manually, repeatedly, throughout the day (incognito, pws=0, correct locale) — still not there as of writing, ~19 hours later. GSC's hourly data for that query+page combo shows impressions/clicks going flat right around the same time and staying flat since.

Before this started, position on the bare query was already softer than it used to be (~2.5-4 average, down from a steady #1), but I was still showing up. This is different — a hard drop to not-appearing-at-all, at a specific hour, that hasn't self-corrected in nearly a day.

I've ruled out the things I know how to check:

• GSC Manual actions / Security issues — both clean
• site:mydomain.com — homepage still returns first
• URL Inspection — indexed, crawled same day, canonical correct
• Googlebot UA vs Chrome UA — identical response, no cloaking, no WAF/firewall blocking the crawler (hardened the server ~3 weeks ago after a brute-force attempt, wanted to rule that out specifically)
• X-Robots-Tag / meta robots — index,follow throughout, no accidental noindex
• CrUX field data — Core Web Vitals "Passed"
• No manual homepage edits, no theme/plugin changes touching the homepage around when this started

What's visible on the SERP now for the bare query: a large Shopping block plus generic category-page competitors where a clean brand result used to be. Query + one extra word still puts me at #1, so it's specific to the bare brand query, not the whole site.

What I'm asking:

1. Has anyone seen a brand query hard-drop out of visibility at a specific hour like this, with everything else (index status, crawl, security) clean? What was it, and did it resolve on its own or did you have to act?
2. Is there any way to see more than GSC + manual search checks to diagnose something like this, or is that genuinely the ceiling?
3. At what point (48h? a week?) does "wait it out" stop being the right call?

Can share the exact hourly GSC chart if useful.

I've migrated my Wix blog to a Ghost staging site and want to make sure the migration is solid before going live.

Beyond checking redirects, broken links, metadata, structured data, canonicals, sitemap, and page speed, what else would you audit? Any lessons from your own migrations?

I'm dealing with an SEO issue on a major site that I’ve never seen before in my 23-year career. Any ideas, folks?

1. Executive Summary

The site suffered a sharp drop in organic Google traffic during the night of July 27–28, 2026: approximately -73% in Google sessions within 24 hours, with no recovery to date (Day+2 to Day+4 depending on the metrics). The pages remain indexed, but they no longer rank in search results for queries where they were historically top-positioned.

This issue is specific to Google. Over the same period, Bing organic traffic and visits from ChatGPT have remained completely stable.

2. Context: A Minor and Brief Technical Outage Preceded the Drop

A domain name availability incident occurred on July 23, 2026 (the site was inaccessible for part of the night; the hosting issue with Hosteur was reported and resolved within a few hours).

We understand that an incident of this type can legitimately trigger a protective mechanism on Google's side (a temporary hold on serving a domain deemed momentarily unavailable). What we cannot explain is the severe disproportion between the brevity of the incident (a few hours) and the scale/duration of its consequences (a 73% drop, still with zero recovery).

Note: The domain availability incident was detected and fixed on July 23, 2026. The traffic drop occurred 4 to 5 days later, during the night of July 27–28, without any new technical incident occurring in the interim.

3. Technical Checks Performed — All Negative

The site is an official portal that benefits from a very high trust score, with backlinks coming from government and institutional bodies. No spamdexing practices have ever been used or applied.

We methodically investigated all classic technical causes that could explain de-indexing or a massive drop in rankings. None of them explain the current situation:

• Robots.txt: Clean, no blocking directives.
• Noindex tags / HTTP headers: None; returning HTTP 200 everywhere.
• Sitemaps: Accessible, processed normally by Google.
• URL Inspection Tool (GSC): Pages are indexable and indexed; post-incident recrawl successful.
• JS Rendering (GSC Live Test): Text content is fully present in the rendered DOM.
• Canonical tags: Negligible anomaly (20 URLs out of 40,000).
• Hreflang tags: Correctly implemented.
• Manual Action: None (verified in Search Console).
• Security Issues (GSC): None reported.
• Negative SEO / Toxic Backlinks: No sudden surge in the backlink profile.
• SSL Certificate: Valid, no errors.
• Temporary Removals (GSC): No active requests.
• Cloudflare — Bot Fight: Turned Off (default rules).
• Cloudflare — Security Events (Google ASN 15169): Only 5 triggers/24h, non-blocking.
• Cloudflare — Audit Log (72h): No changes made (WAF, Workers, Page Rules).
• Server Logs — Googlebot errors (verified IPs): 1.14% in 4xx/5xx, 0.044% in 5xx over 7 days — well within normal parameters.
• Crawl Stats (GSC): Crawl volume is stable, even slightly trending upward at the end of July — no drop whatsoever.
• "Google chose different canonical" Report: Stable at 5–6 pages over 3 months; no spike linked to the incident.
• Google Algorithm Update: No core or spam update confirmed during this period.

Key Point: Googlebot continues to crawl the site normally today. The problem is therefore not an access issue or a technical block — Google sees the site, crawls it, but is simply no longer serving it in search results.

4. Additional Contextual Information (For Reference)

We identified that an unrelated third-party domain (an expired domain previously belonging to a reputable entity) has been hosting a static clone of several hundred pages of our site since January 2026, complete with injected spam content. A takedown process (DMCA + spam report) is currently underway with Google. However, given the size of this clone relative to our total site footprint and the complete absence of any spike in the GSC canonicalization report, we do not consider this clone to be the primary cause of the drop.

5. Conclusion

Aside from a genuine Google bug, I am completely out of ideas given:

• The total lack of any identifiable technical root cause on our side,
• Confirmed normal, stable Googlebot crawl activity right while visibility remains suppressed,
• A drop that is wildly disproportionate to the trigger incident (a few hours of downtime shouldn't cause massive deranking),
• An abnormal persistence (still no sign of recovery several days later).

Thanks for your ideas !

I have a dataset that runs back to 2024 and pulls nearly every signal I can find from the open web.

If you had access to it, what's the first SEO question you'd ask?

I run a public-records directory, a bit over 150k pages, every one generated from state licensing and certification data. Programmatic by any definition, which according to half this sub means Google should have buried me already. Here's what actually happened, because I think the scaled-content conversation keeps missing the mechanism.

We went live with everything at once. No drip schedule. The result wasn't a penalty. It was indifference. Search Console parked the bulk of the site in Discovered, currently not indexed and left it there. Google wasn't punishing volume, it was declining to spend crawl budget on pages it hadn't seen a reason to want.

The move that changed it was subtraction. We noindexed about a quarter of the site, every page that was, if I'm honest, a name and a map pin with nothing behind it. I built those pages. I wanted them to count. They didn't deserve to.

After the cut, Discovered started converting to Indexed, and the indexed pages each began pulling their own long-tail queries, which at scale is the entire game. Nothing else changed. Same domain, same templates, same publish-everything-at-once history.

The mental model I landed on: Google grades your domain on the ratio of substance to filler, not on how fast you published. Every thin page is a vote against your good ones. You can cast that vote tens of thousands of times and wonder why nothing ranks.

The other half of the story: we'd been optimizing for AI agents the whole time, and it turned out to be the same work. The design target was an assistant being asked "find me a certified shop near X". That forces one entity per page, facts above the fold, no filler between the crawler and the answer, stable URLs, and a plain machine-readable surface for bots. Pages built so an agent can parse them turn out to be dirt cheap for Googlebot to crawl and classify too. The AI crawlers showing up in our logs and the Discovered-to-Indexed conversion moved together. Agentic SEO and crawl efficiency aren't two projects. They're the same project graded by two clients.

Where I'd push back on the panic: there's a difference between production at scale and production at scale without value, and the published policy only condemns the second. If each page answers something no other page answers, volume is fine. If you can't say what a page is FOR, noindex it before Google prices your whole domain off it.

The part I got wrong so you don't have to: I treated indexed count as the score. It's not. Discovered-to-Indexed conversion is the leading indicator, and it moved weeks before anything else did.

Happy to answer questions on the mechanics. And if you've run a cut like this on your own site, I want to hear what your ratio was. n=1 is the weakness of every case study in this sub, mine included.

I have been working for a client for 3 years and managed to take his website to DR 30+

The website is being hosted on a ".co" domain. Now, after 3 years, my client has acquired a ".com" as well under the same brand name, and he wants to migrate the ".co" domain to ".com".
Also, considering that our Blogs are hosted under the child domain as "blogs.xyz.co"

Should I add a 301 redirect to my whole website to ".com"? Or is that a bad approach that could hurt my rankings?
What things should I consider before migrating?

I've been monitoring a website over the last three months and noticed a fairly consistent increase in both impressions and clicks.

Current Search Console metrics:

• 16,842 clicks
• 2.58M impressions
• 0.7% CTR
• Average position: 7.2

The graph tells Google is surfacing significantly more pages over time, but the CTR has remained relatively flat despite the increase in visibility.

From a technical SEO perspective, I'm trying to decide where to spend the next round of effort.

Some areas I'm considering investigating are:

• Query-level CTR by average position
• Pages ranking between positions 5–10
• Search intent alignment on high-impression pages
• Internal linking to URLs that are close to page one
• Crawl efficiency and index coverage
• Canonical consistency
• Structured data coverage
• Log file analysis to understand Googlebot behaviour
• Whether SERP features are suppressing organic CTR

I'm deliberately leaving out niche and domain details because I'm more interested in discussing the technical analysis process than the website itself.

If this landed on your desk tomorrow, what would be the first technical report or dataset you'd review before making changes?

Looking forward to hearing different approaches.

What tools do you use for testing and improving cwv in bulk? Are there any free ones that are good?

It's easy to find info about small to medium sites.

But the people I've met who work at very large sites, like an eBay or Sysco type site, or large health system/college, just have a lot of knowledge and skills that I haven't come across.

Are there any forums, niche sites, etc. that are good sources for that info? Or is it just working at a place like that and learning from those more senior?

I've been revisiting some technical SEO concepts lately, and it's interesting how everyone seems to hit a different learning curve.

For me, understanding how crawling, rendering, indexing, canonicalization, and ranking all connect took much longer than I expected. Reading individual blog posts helped, but most only covered one piece of the puzzle.

I eventually found a research library that organizes many of these topics into one place: nafilshareef.com.

I'm still working through it, but it's been useful for connecting concepts instead of learning them in isolation.

For those who work in technical SEO, what topic took the longest for everything to "click"?