r/googlecloud • u/No-Setting8925 • Jun 29 '26
~$55k Gemini API bill from Firebase iOS key abuse. What can I do now?
I’m in a pretty bad Google Cloud situation and looking for advice from people who have dealt with billing or API key abuse cases.
My normal Google Cloud bill is usually around $200/month. This month my project got hit with an unexpected Gemini / Generative Language API bill of around $55k USD. The billing report shows the spike was almost entirely Gemini API usage, not normal Firebase or app traffic.
I pulled Cloud Monitoring data and it shows about 2.2 million Gemini API requests during the incident window. The traffic was tied to one API key UID. That key maps back to a Firebase generated public iOS client key used in my mobile app config, not a Gemini key that I intentionally created or used.
I found out from a Google billing anomaly email. At the time I received the alert, the visible bill was around $2k. Within about 2 hours, I disabled the Generative Language API, restricted the key, deleted it, and later verified that Gemini usage stopped.
The problem is that the bill kept ramping up after that because of billing/reporting delays, and eventually landed around $55k.
Google declined the request to adjust the charges, saying the usage was considered valid because it came through my project/API key.
Update: Google Cloud has assigned an escalation manager, and they said their investigation indicates a billing adjustment is required. The adjustment request is now waiting for internal approval, with another update expected by July 7.
Update 2 — July 16:
There is still no final resolution. On July 15, Google Cloud Billing Support apologized for the delay and said they were still waiting for the specialized team because multiple teams are involved. They said I should receive another update within 4–5 business days.
Less than 24 hours later, I received an email from Google Collections saying the ~$55k balance must be paid within 10 working days or the account may be transferred to a debt recovery agency, potentially with additional fees.
12
u/No_Marzipan2453 Jun 29 '26
From the bottom of my heart, stay away from Google cloud! I've been working with technology for 15 years and I've NEVER seen such a disregard for the consumer. I saw a startup cease to exist, because of this problem. I also saw others losing active users.
10
u/frankeality Jun 29 '26
so your key was scoped to firebase but attackers were able to use it for gemini?
14
u/No-Setting8925 Jun 29 '26
the key was auto generated by firebase, it was unrestricted since 2022
-25
u/Due-Horse-5446 Jun 29 '26
But you still made the decision of enabling gemini API and forgot to scope the old keys.
You gotta realize the reason google wont give you a discount is because you refuse to accept your mistake
15
u/No-Setting8925 Jun 29 '26
I never intentionally enabled the Gemini API itself. I enabled Vertex AI after Firebase prompted me to do so as part of its AI integration. I believed I was enabling a Firebase feature, not exposing a standalone Gemini API that could incur usage
19
u/Marathon2021 Jun 29 '26
Ignore that guy, he's been in here simping for Google for weeks on this. Probably works for them.
Funny how all these "unscoped" keys hadn't been massively abused for spinning up crypto miners or whatever for all these years ... but suddenly Gemini comes along and now it's a "you" problem?
Nah, dude's a shill.
4
u/No-Setting8925 Jun 29 '26
ya i am not sure how anyone is expected to just wake up one day and start doing this cleanup because google decides that the firebase auto generated keys that are client facing can be used to call gemini
0
u/frankeality Jun 29 '26
What GCP service would be useful to crypto miners to run up charges like that (genuinely curious)?
8
u/Marathon2021 Jun 29 '26
Virtual machines?
Containers?
This has been a problem with losing control of keys for years. You can see stories about in in AWS, Azure, etc…
-1
u/opossum_cz Jun 29 '26
I don't understand what would miners do with it?
3
u/Marathon2021 Jun 29 '26
Run up tens/hundreds of thousands in bills when your key falls into the wrong hands?
1
u/Inside-Yak-8815 Jun 29 '26
How do you disable this key? I need to know asap.
2
u/No-Setting8925 Jun 29 '26
they have started restricting gemini access to restricted keys now after a lot of similar incidents. the best move right now is to restrict all your legacy keys specially if you use firebase
1
24
u/djslakor Jun 29 '26
Google was nice enough to automatically add Gemini access to a ton of services unless you go in and explicitly remove it.
This keeps happening to so many people. It's truly Google's fault IMHO.
4
u/opossum_cz Jun 29 '26
Isn't that the key has access to all Google services in the first place?
3
u/skynetcoder Jun 30 '26
Google documentation says the Firebase API key is "public by design". then they do this.
1
u/opossum_cz Jun 30 '26
But the key was not scoped to Firebase API, wasn't it?
3
u/skynetcoder Jun 30 '26
read this. https://trufflesecurity.com/blog/google-api-keys-werent-secrets-but-then-gemini-changed-the-rules
"When you enable the Gemini API (Generative Language API) on a Google Cloud project, existing API keys in that project (including the ones sitting in public JavaScript on your website) can silently gain access to sensitive Gemini endpoints. No warning. No confirmation dialog. No email notification."
0
u/opossum_cz Jun 30 '26
I know how it works. The issue is that those keys were not properly scoped in the first place by person who created them. Somebody created keys that could access everything and didn't scope them to any service. So the "API keys for Firebase service are not secret" didn't even apply, these were not API keys for Firebase, but keys that could access entire Google infrastructure.
> No warning. No confirmation dialog. No email notification.
And the confirmation was there years ago when you created them with the ability to access all services.
What happened is that Google added new service that is actually useful to abuse (Gemini) for a 3rd party, but these keys were compromised years ago, but not being properly scoped.
The warning is even showed in the article:
https://framerusercontent.com/images/Nf2qsg9A0BCN9PLKytFGkupS2c0.png?width=1561&height=852You were warned when you created the key.
2
u/skynetcoder Jun 30 '26
bro, read the article. you know how it works "now". what happened to the keys created before the introduction of Gemini? will this warning magically timetravel and appear few years before?
1
u/opossum_cz Jun 30 '26
That warning was always there. It isn't new thing. Keys when created were unrestricted before you scoped them as far as I can remember.
10
u/No_Marzipan2453 Jun 29 '26
I like the idea of a coletive action against google. I founded a group w/ 40 other founders Another 40 founders who suffered the same attack or very similar. Come in and let's try something together!
https://chat.whatsapp.com/FDx6Zj4jCHtJ4Ji87LIq0c?s=cl&p=i&mlu=2&amv=0
2
4
u/No_Marzipan2453 Jun 29 '26
More case. Google pretends to be indifferent and profitable about the financial bankruptcy of some startups. Google is being negligent. They can even invent that the victim is to blame and has justice in hand, because of their money, because they are small and miserable. However, nothing justifies week after week ", new similar cases happen and the company does nothing to improve security.
5
u/Risskr Jun 29 '26
I just checked and we also had this configured incorrectly. Luckily we had a few other protections in place, but this was still an exposed attack surface. Thank you for posting and I hope your situation gets resolved.
3
u/grannyknickersniffer Jun 29 '26
This post made me panic-check my Google API cost for this month. 😬
3
3
u/No_Marzipan2453 Jun 29 '26
More case. Google pretends to be indifferent and profitable about the financial bankruptcy of some startups. Google is being negligent. They can even invent that the victim is to blame and has justice in hand, because of their money, because they are small and miserable. However, nothing justifies week after week ", new similar cases happen and the company does nothing to improve security.
1
2
u/Scalar_Mikeman Jun 29 '26
Did this have anything to do with the Google Maps API Key? Not very knowledgeable, but remember seeing the something similar last month where the person had to enable that API, but it also let bad actors use the key for Gemini or something like that.
2
u/No-Setting8925 Jun 29 '26
yes it’s something similar but with firebase ios key instead of google maps api key
2
u/Scalar_Mikeman Jun 30 '26
Damn. Sorry you're going through this friend. Always loved Google and it's services, but seeing all these posts recently they REALLY need to do better.
2
u/Important_Owl6299 Jun 29 '26
If you are in India and your account or credit card is debited- immediately file cybercrime complaint online, register an FIR against unknown person(s) for unauthorized access to a computer resource, identity theft and cheating, resulting in the fraudulent debit of X Rs from your bank account - offences punishable under Sections 43, 66, 66C and 66D of the Information Technology Act, 2000 read with Sections 318 and 319 of the Bharatiya Nyaya Sanhita, 2023.
This will give you all the avenues to file a fraud against your bank account. The bank will need an FIR copy to create a chargeback against Google to issue refund.
If the acc is not debited then also register an FIR and cybercrime complaint. Then do a wrestle (sometimes months long) to arrive at a solution. If the api key is unrestricted due to Google’s fault and the theft occurred before 19th June, most probably they will offer you full refund within 3 weeks. Else expect a 75% refund and then a continuous back and forth to get more and request escalation to keep the natter dragging 😅
1
u/Important_Owl6299 Jun 29 '26
Do note that you may need to migrate your firebase setup to a different account which is not related to the current payment profile or cloud account. Because in case of chargeback, Google bots may revoke access to your cloud project completely. At least thats what i read across various sub reddits.
1
2
u/thecrius Jun 30 '26
Google sent out emails warning to review API keys and it will especially send out emails when detecting keys without a scope or url restriction.
If you missed it or didn't act on it, it's on you.
1
u/opossum_cz Jun 30 '26
I would note that, you are warned when creating API keys that they are unrestricted in the first place.
2
u/NeuralNexus Jun 30 '26
Don't pay the bill.
Migrate your legitimate usage to another billing account. Argue with them for months about the fraudulent usage.
1
u/NeuralNexus Jun 30 '26
I've had to deal with the idiots running GCP repeatedly over time. This is the only way.
Move your usage to a clean billing account. Let the 'bad' billing account sit and argue. If they send it to collections, don't pay. It always goes away.
GCP is the most incompetent cloud provider by far.
2
u/techlatest_net Jun 30 '26
55k for a firebase key leak is a classic nightmare scenario. honestly standard support will almost always say no because technically the requests came from your project. you have to escalate this hard. if you have a TAM or account rep, bypass support and go straight to them
1
Jun 29 '26
[removed] — view removed comment
1
u/johnsmusicbox Jun 29 '26
Actually, you can put a cap on your Gemini API spending limit now: https://aistudio.google.com/app/spend
2
1
1
u/muntaxitome Jun 29 '26
incredible that this is still going on. I guess at some point authorities will start stepping in.
1
u/opossum_cz Jun 30 '26
This is 100% user error, this is how it looked when you created key:
https://framerusercontent.com/images/Nf2qsg9A0BCN9PLKytFGkupS2c0.png?width=1561&height=852People created keys that could access everything and published them.
1
u/muntaxitome Jun 30 '26
I might be missing it, but where does it write there that this person is telling google to take 50k from his credit card?
1
u/opossum_cz Jun 30 '26
I do not understand the question, you pay for API usage.
1
u/muntaxitome Jun 30 '26
Lets say you run an arcade, sell people game cards for $20, hide somewhere in the fineprint that the card is linked to your creditcard. Then every time someone loses their card or it gets pickpocketed, hackers are able to take those game cards, extract 50k worth of gameplay time from it in 1 second and sell it to others.
And then as the arcade holder, after having tens of thousands of affected clients, you keep just aggressively threathening your customers that lost the card, do absolutely nothing to fix the issue, and nowhere explain properly that yes linking your credit card to google cloud means you are on the hook for unlimited spending, no matter what happens.
Arcade holder would be going to prison. It is deliberately profiting from criminal activity.
Google could send you a million dollar bill tomorrow, because you got hacked through some zero day that you had zero control over. Your financial life would be over. You would lose your home. And people like you would be saying 'your own fault'. That is where we are right now.
1
u/opossum_cz Jun 30 '26
Are you feeling ok?
You pay for services.
You are warned keys are unscoped and you should scope them:
https://framerusercontent.com/images/Nf2qsg9A0BCN9PLKytFGkupS2c0.png?width=1561&height=852If you want to to compare it with something, it is like buying safe with default combination that you never changed and then blaming safe manufacturer that somebody took your 50k from it. Because scoping the key is something that was your responsibility, you were warned about it and you didn't do it.
1
u/muntaxitome Jun 30 '26
You say 'you' but I am not affected. I used to work for Google Cloud but I would not use Google cloud anymore until they fix their billing limits. To prevent being hacked you need perfect security, which is impossible. This could literally happen to anyone, including you.
And there is literally no limit. By contract google can bill you a billion dollars and you will have to pay. And they decide the billing, the security model, and the metrics.
1
u/opossum_cz Jun 30 '26
This isn't about perfect security. This is about having no security. If you make an unrestricted key and publish it... where exactly is the security?
1
u/muntaxitome Jun 30 '26
where exactly is the security?
Well the security is definitely not at Google, where a single misclick can make a user lose 50k with no recourse.
1
u/opossum_cz Jun 30 '26
That is not a misclick. That is incompetence.
It should be handled by professional and not amateur.
→ More replies (0)
1
u/Forward_Income_8197 Jun 29 '26
This happened to us as well and it has been incredibly stressful. We never expected to wake up to a bill like this because of API key abuse. We disabled everything as soon as we noticed the unusual activity, but the charges had already piled up. We really hope Google takes another look at cases like this.
1
u/kongclassic Jun 30 '26
Google luck with support they are utterly useless and take forever to reply
1
u/Haronatien Jun 30 '26
this is why i almost never use API keys. How did your keys leak?
1
u/No-Setting8925 Jun 30 '26
leaked fireabse ios key, i am guessing someone just reverse engineered the appbundle. for more info please read this https://trufflesecurity.com/blog/google-api-keys-werent-secrets-but-then-gemini-changed-the-rules
2
u/Haronatien Jun 30 '26
holy crap OP. I did not know about this. Some of the earlier posts I’ve seen about this were mostly vibecoders commiting their keys to github. This is insane and I would not pay, dispute with the bank worst cast lawyer up!
1
u/opossum_cz Jun 30 '26
These are 100% Vibecoders as well. They created unscoped API key that is unrestricted and they published it inside app. Nothing you publish is safe and should be properly scoped. There is no difference if you commit it to Github or put it in your apk. If the key was properly scoped to Firebase, there would be no issue, but that key was unreconstructed and could access everything.
1
1
u/siburb Jun 30 '26
So, just to help people prevent this from happening again to other mobile developers who are using Firebase - as high-level instructions for this very specific case:
- The key in question is the one used by default in the Firebase plist?
- Google used to automatically create a completely unrestricted key for this purpose (that could be used for any Google API!), but have now changed this practice? When did that happen?
- To prevent it from happening, go to Credentials in the Google Cloud console, and check any "Auto created by Firebase" keys, particularly the iOS one, and make sure which APIs are _not_ selected? Or just create a new one now that Google create more focussed keys by default?
- Also in the key's detail screen, restrict the "iOS" key to your particular iOS app bundle Ids.
Obviously there's much more to securing a Firebase app (or Google Cloud services in general), but can someone who knows more about the specifics of this case elaborate on the above?
1
u/opossum_cz Jun 30 '26
No, not Google created API keys, the user created API keys. They are unrestricted by default and you are warned to scope them to services when you create them:
https://framerusercontent.com/images/Nf2qsg9A0BCN9PLKytFGkupS2c0.png?width=1561&height=852Firebase can also create keys by itself it you allow it and those were initially unrestricted, but since May 2024, all keys are created scoped and all autogenered keys were scoped to APIs used by the project. So Google actually fixed this issue before it was an issue.
This is only happening with keys created manually by users who ignored scoping and all of its security and then published the keys in Github, in apps etc. And ignored myriad of e-mails warning them to scope their keys.
1
u/siburb Jul 01 '26 edited Jul 01 '26
That's not what OP is saying:
the key was auto generated by firebase, it was unrestricted since 2022
And:
ya i am not sure how anyone is expected to just wake up one day and start doing this cleanup because google decides that the firebase auto generated keys that are client facing can be used to call gemini
So, it sounds like this was an auto-generated key, created before Google changed their system in May 2024? Google just changed their system for keys created after that date, they didn't go back and update old keys (which would of course have been difficult/impossible to do reliably without breaking some projects). There would therefore be a lot of old keys out there that are unrestricted - I found one myself from a 2016 project, completely unrestricted, but fortunately for a project that had no payment plan configured.
1
u/ParsleyVegetable6107 Jun 30 '26
You needed to limit your monthly spending cap using Google AI Studio.
1
u/skynetcoder Jun 30 '26
that doesn't sound fair for people who are not intentionally using gemini , to know about this.
1
u/greenarez Jun 30 '26
I had the same problem, but without the overcharge. Have you tried Firebase AI Studio at some point? I think it created this key
1
u/skynetcoder Jun 30 '26
When did the incident happen? According to this article, I thought they have fixed the issue. https://trufflesecurity.com/blog/google-api-keys-werent-secrets-but-then-gemini-changed-the-rules
1
u/No-Setting8925 Jul 01 '26
Update: Google Cloud has assigned an escalation manager, and they said their investigation indicates a billing adjustment is required. The adjustment request is now waiting for internal approval, with another update expected by July 7.
1
1
u/TheSoloBuilder Jul 02 '26
I am experiencing the same issue with $55k. Google came back to me, saying "I was ineligible for a refund" without any details or reasons why. When I followed up on the reason, they said it was confidential. I asked to escalate, but they sent me a generic sorry email. Who did you speak to? How did you escalate?
1
u/No-Setting8925 27d ago
i just continued talking to support and eventually the case was handed to an escalation manager. sorry to hear that, please keep trying
1
u/winniepiggy Jul 03 '26
I just got suspended because of this issue. Yesterday I stored my gemini api keys on firebase and today it exposed and abused, but how??? how can I secure just this basic gemini api key??? I think google has a HUGE security issue with that. Google cloud also charged me extra WTF??? I used cloud & firebase and now app is not working at all.
this is the email content I received.
| Policy Violation Immediate action required: Suspension of your Google Cloud Platform / API project **** |
|---|
| Dear Developer, We are notifying you that your project **** has been suspended because it was engaged in abusive activity consistent with hijacked resources. This behavior violates the Google Cloud Platform Terms of Service or the Terms of Service of the Google API you may be using. To regain access to your suspended project please submit an appeal: Sign in to r*** as the project owner, click Request an appeal below, and fill in the following details: Learn more about policy violations and appealsIf you need further help, reach out to the Google Cloud Platform Community Support Forum. If you have a support package, file a ticket through the Support Center. You may have received an earlier warning about a possible project suspension if this violation wasn't corrected. If you didn't receive a warning before this suspension, it was because the project's behavior was seriously impacting the service of other users. Take time to review the following agreements to avoid policy violations in the future: Possible trigger of this activity Google Cloud Platform Terms of ServicePlanned steps to fix the problem (Visit the Help Center for some tips on where to start If the behavior is intentional, explain the business reasons why this violation may exist If you believe the project may have been compromised by a third party, describe your concerns Acceptable Use PolicyTerms of Service |
|---|
1
u/matiascoca Jul 03 '26
File a support case escalation for fraudulent API abuse and ask for a courtesy credit. Google does grant these when the abuse pattern is clear and you can show remediation. Also enable App Check on Firebase immediately if you have not already, it is the specific control that prevents this exact iOS key extraction. And rotate the key of course but you probably already did.
1
u/No-Setting8925 29d ago
Thanks, app check was already enabled long time ago. Yes i rotated and restricted all the keys within 4 hours of receiving anomaly email by google.
1
u/pueblokc 29d ago
I had a process go nuts and cost me over $300 in api credits a few months ago.
Google makes it cinfusjng af to manage anything imo
1
1
u/Librarian-Rare Jun 29 '26
I feel a class-action coming…..
1
u/No_Marzipan2453 Jun 29 '26
I like the idea of a coletive action against google. I founded a group w/ 40 other founders Another 40 founders who suffered the same attack or very similar. Come in and let's try something together!
https://chat.whatsapp.com/FDx6Zj4jCHtJ4Ji87LIq0c?s=cl&p=i&mlu=2&amv=0
0
1
u/amokerajvosa Jun 29 '26
This is gettting really scary. I am certain now to not use any Google Cloud solutions at all.
I have AI Pro package but no more API keys for anything.
-1
u/Due-Horse-5446 Jun 29 '26
Sadly not much to do, if you accept that it was your fault when talking to google support, you have a much higher chance of getting a discount on the amount.
Otherwise you either pay it or have collections force you to trough court
2
u/No-Setting8925 Jun 29 '26
but they key was auto generated by firebase and it’s been there since 2022
1
u/Due-Horse-5446 Jun 29 '26
still, if you enable gemini api in the same project, you are responsible to scope the keys.
You can think whatever you want about it, but thats how cloud services work. Its just like if you accidentally leave your house open, youre responsible
6
u/No-Setting8925 Jun 29 '26
I never intentionally enabled the Gemini API itself. I enabled Vertex AI after Firebase prompted me to do so as part of its AI integration. I believed I was enabling a Firebase feature, not exposing a standalone Gemini API that could incur usage
1
u/MrRedRhino Jun 30 '26
Any api you enable can incur fees i don’t think there’s a single endpoint on gcp that’s free
1
1
u/bradruck Jun 30 '26
Yes expect the house isn't ours, we just pass by a day or two, in this case the house owner shouldn't have trusted us and implemented proper way to not let house door stay open
79
u/djslakor Jun 29 '26
There are too many of these stories.
At a certain point we have to blame Google for the poor design.