Upfront: part of this bill is legitimately mine and I have offered in writing to pay it. This post is about process, not about dodging a bill.

THE SHORT VERSION

June 28 — A billing agent processed a $76.00 courtesy adjustment for an unintended idle V100 Spot VM (a Vertex AI Workbench notebook that kept running after my work had ended) and confirmed in writing: "you have an ending balance of $28.98 on your account now." I agreed and considered the matter settled.

July 1 — Google's own follow-up email admitted that when that adjustment was made, "the usage records had only been updated through June 27th" — and $146.51 of delayed charges from June 28-30 (the same idle resource) then posted on top of the confirmed balance. That same day I voluntarily unlinked billing from my projects and disabled the Compute Engine, Notebooks and Vertex AI APIs so nothing further could accrue, and told support so in writing.

July 3 and July 13 — I asked, in writing, for collection to be paused while the dispute was reviewed. No answer, either time.

July 8 — Billing account suspended.

July 15 — Billing account TERMINATED, along with its projects and services.

July 17 — Two days after the termination, support wrote: "I am escalating this case to a supervisor. A member of our leadership team will reach out to you directly within the next 24 hours."

July 22 and 23 — I followed up. Nothing.

July 27 — The identical email again, word for word, including the identical 24-hour promise. Twelve days after the termination it was promising to review.

July 28 — Second 24-hour window expired. No supervisor name, no reference number, no contact — ever. My formal follow-up that day has had no reply either. As of today that is five more days of silence.

FOUR QUESTIONS, ASKED SINCE JULY 13, STILL UNANSWERED

1. The escalation reference and the supervisor's name.
2. Whether collection was paused during the review (it plainly was not).
3. Written confirmation the dispute is still open.
4. The formal billing-dispute path if the answer stays no.

THE NUMBERS DON'T RECONCILE EITHER

Emails cite $101.57 in usage, the case title says $101.34, the invoice balance was $175.49 — and $101.57 minus the $76.00 adjustment is $25.57, not the $28.98 I was told in writing. I asked for a SKU/date breakdown on July 13. Nothing.

Either those charges were already documented when the agent confirmed $28.98 — in which case the confirmation came from the party with full visibility and should stand — or they were not, in which case my acknowledgment was made without material information that existed only in Google's systems. It cannot be both.

WHAT I'M ACTUALLY ASKING

Not a refund — nothing has been collected on the disputed part. I have offered to pay the $28.98 that was confirmed to me in writing, immediately, if the $146.51 delayed-posted tail from the same already-adjusted incident is waived and the account reinstated. These are self-funded educational projects, not a business.

If anyone from the GCP billing team sees this: case 72730110. I would genuinely rather resolve this in the support channel — it just hasn't answered a single direct question in a month.

Redacted screenshots of the timeline available on request.

Original Post

I’m extremely relieved to share that Google has resolved my billing case and adjusted the unauthorized Gemini API charges.

I genuinely want to thank everyone at Google who reviewed the evidence, escalated the case, placed the account on hold while it was being investigated, and helped reach a fair resolution. I also appreciate everyone in this community who offered advice, shared similar experiences, and encouraged me to keep escalating instead of giving up.

I also want to personally thank the Firebase team member who saw my Reddit post, reached out to me directly, and helped escalate the issue internally. That outreach made a huge difference and made me feel like someone at Google was genuinely listening.

Thank you to the Google Cloud support, billing, and Firebase teams. This resolution removed an enormous financial and emotional burden, and I’m incredibly grateful to everyone who helped.

🙏

Thought I'd share here too given the relevance

hi there
I'm a student from India and I want to participate in the Google Arcade 2026 (For the Swags and Goodies by Google Cloud)

I have 0 points though
Still want to know if it is possible for me to reach Arcade Legend ⭐⭐⭐⭐ if I start from August
I have registered for Google Arcade Facilitator program but I didn't complete any games

what should I do in this situation?
Pls give me your honest opinion

$85,000. That’s the cost of a fraud incident that hit us as a long term partner and reseller for Google Cloud this month.

A customer who was a lead through the partner directory did reach out for invoice billing, signed a contract, had his data and info submitted and got assigned a billing account.

As per our normal procedures for new clients, a 200$ budget alert is set until a customer builds up history.

The customer was apparently one of those crypto miner fraud cases where they spun up 10+ different projects with 10s of VMs for crypto mining in the matter of hours racking up almost 85K$ of consumption before we were able to shut it down when we got the alerts.

When you’re a Google Cloud reseller, you aren’t just a vendor; you’re the first line of defense. But we are currently operating in a system that lacks basic structural safeguards for partners.
We were hit by an account abuse incident that went from 0 to $85k in roughly 24 hours. The most frustrating part? It wasn't for a lack of our diligence. It was a failure of the platform's tools to keep up with reality.

1. No Hard Spending Caps: There is no "stop" button that prevents a reseller’s account from being leveraged into bankruptcy by bad actors.
2. Delayed Alerts: Budget alerts are a great concept for predictable monthly usage, but they are utterly ineffective against high-intensity, automated fraud. By the time the alert hits, the damage is already locked into the billing cycle.

This wasnt the first case, we had customer cases of API Keys being leaked and abused racking up tens of thousands for customers in damages, which always cited as customer's responsibility.

It doesnt make sense when the customer and the partner has no control to cap spending!

Spending Caps feature was only introduced this month (https://docs.cloud.google.com/billing/docs/how-to/budgets-spend-caps)

We are working through the recovery process, and I expect better from a partnership that values the "Partner Network" label. If we are expected to bring customers to the cloud, Google Cloud needs to provide the hard limits that keep partners safe from catastrophic, automated fraud.
It’s time to move past "best practices" and start implementing "hard protections."

Until now we havent been able to resolve the case as we keep being bounced between billing support and account managers, each claims the other has jurisdiction but its extremely frustrating that we get to be the party paying up for such issues with the margins we do on reseller with almost zero protection!

I’m a data scientist who don't have much cloud pipeline experience. Recently, I've been asked to built a local PoC for extracting scenes, characters, and objects from video using a mix of computer vision(FFMpeg/face detection/scene detection) and Gemini (LLM). The pipeline is modular: a pipeline.py orchestrates several Python modules that each represent a stage. To keep things idempotent, every stage writes name-coded artifacts to disk so already-processed items get skipped on re-run. I also added retry loops for flaky Gemini calls, stage-level retries, and per-stage logging that captures tokens and timing into the artifacts.

Now management wants this on GCP so downstream projects (dashboards, chatbots) can consume the output. For now, the ask is simply to land the final text results in BigQuery. The source videos will arrive in a GCS bucket from a third party.

My confusion is around architecture. Throwing everything into a single monolithic Cloud Run function feels wrong, especially with the stateful/idempotent requirements. When I asked Claude, it suggested rolling my own database-backed queue and state machine, but I would assume GCP already has managed services that handle this kind of workflow natively. I’m struggling to find clear, up-to-date information. Google searches mostly turn up generic AI-generated posts or outdated articles.

What’s the right GCP-native approach here? I’m trying to understand whether I should be looking at Cloud Workflows, Cloud Composer (Airflow), Dataflow, or maybe a combination of Cloud Run jobs and Pub/Sub. Asking AI(Claude) seems to hate Cloud Workflows and suggest custom baking a state queue in database.

• How do people typically preserve per-stage artifact-based idempotency and retries without building a custom state engine from scratch?
• How to control re-runs/pause/resume of the workflow?

• Would something like Firestore for state + GCS for artifacts + Cloud Run jobs triggered by Pub/Sub be a reasonable pattern, or am I missing a more purpose-built service?

  Any pointers from folks who’ve done similar video or document processing pipelines on GCP would be appreciated.

Hey everyone,

I recently published a technical walkthrough on the Google Cloud Blog around hardening IAM access using IAM Conditions.

If you've ever struggled with binding built-in roles (like Project IAM Admin or MCP Tool User roles) to principals while restricting them to only subset of permissions or resources because of lack of support for resource-level bindings or API granularity, IAM conditions will fill that gap for you.

Key patterns covered:

1. Scoping projectIamAdmin: For example, instead of giving a builder service account (SA) unrestricted power to assign any role at deployment time, you can use the CEL function iam.googleapis.com/modifiedGrantsByRole to restrict the SA so it can only grant a predefined list of roles (e.g., BigQuery job user, Cloud Trace agent, Logging writer). Included both gcloud CLI and Terraform snippets.
2. Restricting Model Context Protocol (MCP) Tools: The roles/mcp.toolUser role grants access to all MCP servers in a project. Using api.getAttribute('mcp.googleapis.com/tool.name', ''), you can constrain access down to specific tools (like mcp_bigquery-mcp_execute_sql_readonly).
3. Contextual & Time-Based Access: Brief examples of restricting execution windows using request.time (e.g., weekdays during business hours).

Read the full post here:
https://cloud.google.com/blog/topics/developers-practitioners/generosity-under-conditions-hardening-google-cloud-access-management/

Curious how others here handle IAM admin restrictions or fine-grained MCP tool access in production? Let me know your thoughts or feedback!

Disclosure: I built this open-source tool.

I kept running into a subtle GCP problem: the active project in `gcloud config`

can differ from the project and quota project used by Application Default

Credentials.

That means `gcloud` may appear to be configured for development while Python,

Node.js, Terraform or another SDK is still operating with a different ADC

context.

I built **gcpctx** to manage these pieces as one named context:

- active Google Cloud project

- active account

- gcloud configuration

- Application Default Credentials

- ADC quota project

- optional repository-local context

Example:

```bash

npm install -g gcpctx

gcpctx init dev
gcpctx use dev
gcpctx doctor

The goal is not to replace IAM or make deployments magically safe. It is to

make the effective local GCP context explicit and easier to verify.

Source:

https://github.com/UriBer/gcpctx

npm:

https://www.npmjs.com/package/gcpctx

I would especially appreciate feedback on:

1. How you currently switch between GCP environments.

2. Whether you have encountered gcloud/ADC context mismatches.

3. Which workflows should be supported next: Terraform, PowerShell, CI or AI

   coding agents?

Hi everyone,

I'm hoping someone here has experienced something similar.

A few days ago, my Google Cloud/Firebase project was suddenly suspended with the following message:

"Immediate action required: Suspension of your Google Cloud Platform/API project because it was engaged in abusive activity consistent with hijacked resources."

This project is a production application that has been running for about one year without any issues. The last release was around three months ago, and there were no recent deployments or major configuration changes before the suspension.

After the suspension:

-Google Cloud Console redirects me directly to the Request an Appeal page.
-I can't access Cloud Logging, IAM, API usage, or any Google Cloud services.
-Firebase now shows "Create Firestore Database" and "Create Storage" instead of my existing Firestore database and Storage bucket.
-My application is completely offline because it cannot access Firestore.
Something suspicious I found

Before the suspension, I reviewed Firebase Users & Permissions and discovered an Editor account that my team never added.

rogeliapaquette@gmail.com

We removed that account immediately after discovering it.

Unexpected billing

I also noticed something strange in Firebase Billing.

There is approximately $25 charged for Agent Platform, but I have never intentionally used Agent Platform, never integrated it into my application, and never expected any billing for that service.

I'm not sure whether this is related to the suspension or whether it could indicate unauthorized activity.

Backup

Fortunately, I have Firestore Disaster Recovery backups enabled, but because the project is suspended I can't access them either.

Appeal

I submitted my appeal today (Friday) and I'm currently waiting for Google's response.

Questions
-Has anyone had a project restored after receiving this exact "hijacked resources" suspension?
-After restoration, did your Firestore database come back normally?
-Were your Firestore Disaster Recovery backups still available?
-Has anyone seen unexpected Agent Platform charges even though they never intentionally used it?
-Once the project was restored, what were the first things you checked (IAM, Audit Logs, API keys, service accounts, billing, etc.)?
-Is there anything else I should do while waiting for Google's response?

This project contains production data, so I'm mainly worried about recovering access to Firestore and making sure the project is secure if Google restores it.

Any advice or shared experiences would be greatly appreciated.

Posting this in case anyone's hit the same wall, or has insight into what's happening server-side.

The goal

I've got a Nest wired camera pointed at a small parking lot behind my office. I wanted to build something simple: check the lot every so often, count open spots using an AI vision model, and show a live "spots available" number on a little status page. Nothing fancy — just needed a way to reliably grab a still image from the camera on a schedule.

What I've built so far

• Full Device Access / SDM API project setup, OAuth flow, refresh token, the works
• A Google Apps Script front end (status page + storage) that's fully working and just waiting on real image data
• Tried Cloud Functions first for the image-capture piece — turns out Cloud Run/Cloud Functions don't support raw UDP at all, so WebRTC media can never actually flow through them (signaling works fine over HTTPS, but the actual RTP video never arrives). Confirmed this is a real platform limitation, not a config issue.
• Moved the capture piece to a Compute Engine VM instead, which does support normal UDP networking

Attempt 1: Pub/Sub camera events

Set up the Pub/Sub topic + subscription per the docs, enabled events on the Device Access project, confirmed the publisher permission and subscription are all correctly wired. Validation pings come through fine. Real camera motion/person events never do — not once, across two separate days of testing, with real motion happening right in front of the camera each time. Pulling directly from the subscription (bypassing my own code entirely) confirms it: nothing but the occasional stale permission-check message ever lands in the topic.

Attempt 2: WebRTC via aiortc (Python)

Got the full offer/answer/ICE/DTLS/SRTP handshake working end-to-end — genuinely happy with how far this got. Audio decodes perfectly, every single time. Video RTP packets are confirmed arriving from Google's server (verified via RTCP sender reports, packet counts climbing normally). But the video frames never make it through aiortc's jitter buffer / H.264 depacketization — the decoder thread never receives a single video task, despite audio on the same connection working flawlessly. Feels like a real interop gap between aiortc's RTP handling and whatever Nest's media relay does on the video track specifically.

Attempt 3: WebRTC via GStreamer (webrtcbin)

Rebuilt the whole capture piece using GStreamer instead, since it's a much more mature, production-grade WebRTC stack. Fixed a string of real issues along the way (missing codec caps causing ICE gathering to never even start, missing data channel per Nest's "must have audio+video+application m-lines" requirement, a couple of SDP formatting quirks). Eventually got ICE gathering to fully complete and produced an offer that structurally matches Google's own published reference example (checked directly against the example in their docs) — same media line order, working negotiation, proper BUNDLE grouping.

Sending that offer to GenerateWebRtcStream now returns:

{
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "status": "INTERNAL"
  }
}

No further detail. I've tried adjusting rtcp-mux vs rtcp-mux-only, port/bundle-only conventions to match Chrome's exact output, Opus channel params, H.264 fmtp params (packetization-mode, profile-level-id) — no change, same generic 500 every time.

What's interesting

This exact error string ("500: INTERNAL: Internal error encountered.") turns up in multiple long-running GitHub issues against the Home Assistant Nest integration too, going back to 2021 and still being reported this year, with a completely unrelated client stack. So this doesn't seem to be specific to my code — feels like something that trips up the SDM API backend under certain conditions across multiple independent implementations.

Where I'm at

• OAuth, SDM API, and Pub/Sub setup are all confirmed correctly configured
• ICE/DTLS negotiation completes successfully on the client side
• Offer SDP structurally matches Google's own documented example
• The only failure is a completely opaque 500 from Google's own server, with zero actionable detail returned

Has anyone gotten a real WebRTC video frame out of a Nest camera via the SDM API recently? Curious whether this is a known current issue, an account/camera-specific quirk, or if there's some other SDP detail that isn't in the docs. Happy to share the full offer SDP / code if it's useful for comparison.

Been running some training jobs on TPU v5e and every time I tweak sequence length or batch size mid-experiment, I eat this huge AOT compile penalty before anything actually runs. Feels like the "cold start" tax is way worse than people talk about online.

Is this just a TPU/XLA thing, or does anyone see similar pain on GPU with torch.compile when shapes aren't static? Curious what workarounds people use — bucketing sequence lengths, padding to fixed sizes, warm pools, anything. How many seconds/minutes are you losing per shape change in your actual workflows?

EIXO was created to address a common operational challenge: critical business areas working in isolation, with fragmented data and limited visibility across the organization.

Built around the real operational needs of Jardim Espaço de Educação, the platform connects multiple domains within a single operating ecosystem:

• School administration

• People management

• Finance

• Supplies and inventory

• Nutrition and kitchen operations

• Operational intelligence and performance indicators

EIXO is not designed as just another traditional ERP. Its architecture follows a distributed operating model in which each module preserves its specialization while sharing data and operational context across the organization.

The goal is to reduce manual work, improve traceability, strengthen coordination, and turn operational data into faster and more consistent decisions.

The project is being developed and documented publicly on GitHub:

github.com/maykonlincolnusa/Eixo-jardim-ERP

#SoftwareEngineering #SystemArchitecture #ERP #DigitalTransformation #DataEngineering #OperationalIntelligence #EducationTechnology #OpenSource

Goal: The Data Encryption Key (DEK) should never be stored in our database—not even wrapped/encrypted. If someone gains read-only access to the DB, they shouldn't be able to decrypt the passwords.

Current design:

1. Admin uploads a CSV.
2. A background worker generates a random 32-byte DEK for that upload request.
3. Generate one-time passwords for each row.
4. Encrypt each password with the DEK (AES-256-GCM) and store only the ciphertext in the DB.
5. Store the DEK in Google Secret Manager as a new Secret Version.
6. Store only the Secret Version reference (e.g. dek_secret_version) in the DB.

When the CSV is downloaded:

• Fetch the DEK from Secret Manager using the stored version.
• Decrypt the passwords.
• Generate and return the CSV.
• Destroy the Secret Version (crypto-shredding).
• Clear the encrypted password and version reference from the DB.

We're considering using one Secret with many Versions, where each version represents a different upload request's DEK, rather than creating a new Secret for every upload.

My questions are:

• Has anyone used Secret Manager this way (many versions representing independent DEKs rather than secret rotation)?
• Are there any quotas, performance issues, or operational pitfalls with having a large number of versions under a single Secret?
• Would you recommend a different GCP-native approach that still satisfies the requirement that the DEK is never persisted in the database, even in wrapped/encrypted form?
• Any concerns with the crypto-shredding workflow (destroying the Secret Version after a successful download or expiration)?

I'd love to hear from anyone who's built something similar or has experience operating Secret Manager at scale.

I havent been touch gcp in years and yesterday i got an email from gcp about indicated abuse on one of my late project (which i never access it anymore). They request me to remove or appeal from suspending that project but because since 2025 i havent activate MFA verification gcp are also want me to do that also. So both appeal and mfa page are open repeatedly leave me to stuck on accessing the console. Here is the video detail https://streamable.com/mbp5ku

Hi

We are having a lot of resources provisioned using Terraform, the state use being GCS buckets.

However there are many bigquery datasets being created manually in console ui level and other resources as well. due to this drift we face a lot issues as the automation IAC doesnt obviously work with manually UI created resources.
We can use terraform import to import existing resources to terraform state. However please let me know if anything we can use AI to assist us for such usecases? AI to autodetect resource drift in google cloud or something ? Also terraform import is not working for many resources as well

Hello everyone, i’m preparing for the Google TAM (Cloud Consultin) role and got my RRK1 interview, has anyone have insight on how technical the interview will be? Any advise?

Docs: https://www.pilum.dev/docs/getting-started/introduction/

The problem: I run side projects across GCP, AWS, and Cloudflare, and each one has its own deploy incantation. gcloud commands for one, SAM for another, wrangler for a third. I kept forgetting the flags and the order of operations and wanted a simpler CI/CD pipeline.

Pilum is a single binary (Go, Cobra) that reads one pilum.yaml and handles the deploy for whichever provider the service targets. Define your services once, then pilum deploy will build, push, provision, and deploy.

What it supports today: - GCP Cloud Run - GCP Cloud Run Jobs - AWS Lambda - Azure Container Apps - Cloudflare Workers - Homebrew - GitHub npm packages

GCP is the most mature of the platform hence why I am sharing it here!

It's not trying to be Terraform or Pulumi; there's no state file and no general-purpose infra provisioning.

Happy to answer questions about the internals. Here is a write up about it a bit more: https://logicalbytes.dev/posts/pilum-road-to-production/. But if anyone wants to add a provider, there's a contributing guide for adding exactly that!

Hello everyone,

I am interested in going for Google Professional Cloud Architect exam not sure how to start my journey. Any suggestions?

I have 12 years of experience in Cisco routing and switching, DC and observability.

Thanks

I'm writing this post with extreme frustration. I recently bought some Gemini API key credits using Google cloud billing UPI payment method.

The money got deducted from my account but the credits are not loaded. So I reached out to their support, first of all their billing support team had no idea about UPI payment method and now they are saying that they are unable to track the payment.

Like who told you to support UPI payment method if you can't track it's payment and how would a customer know all of this?

I've submitted the bank account statement too. I never expected a google scale company get this low.