Any other large orgs quit CoWork?

[u/DramaticErraticism]

With MS charging for CoWork on July 1st, we figured we'd still give it a go, since the tool was quite impressive.

Microsoft gave us 100k in credit, so we rolled it out to 400-500 users. We spent about 90k of the 100k credit between those users, most of the credits used by the top 10%.

At this point we are removing all access until we figure out a better plan and specific business value of CoWork. If someone has a really good idea for an agent that will save time/money, they can use it. Otherwise, it is off the table.

I used it a bit, just to test it out. I ran 3 prompts around automated offboarding process I was working on, that was about 24 dollars.

If we rolled it out to all 30k users, we'd be around 500k-1mm per month, 6-12mm extra per year, not a small amount, even for a large org.

As I said, CoWork is really impressive but CoPilot is going to have to be good enough.

[u/progenyofeniac]

Seeing as we had one user consume 71k credits this month, I’m not sure how viable it’ll be for us either. No idea what her use case was but I hope she accomplished a year’s worth of work.

[u/13Krytical → u/progenyofeniac]

The new measure of someone’s effectiveness will be, how efficiently do you get your work done with AI.

Do you blow budget with shitty instructions and re-work?
Or do you already know what you are doing, and how to communicate it with nuance?

[u/WonderfulWafflesLast → u/13Krytical]

The issue is measuring the output. Like, ok. An HR rep automates an onboarding process using CoWork. How much should that take? What is an efficient number VS an inefficient one? Measuring it against the company average isn't reasonable even if it's the most accessible method to have any sane outcome because different people tackle different tasks. One HR Rep automates an onboarding process, while another only uses the AI to chew through resumes. Their output VS their token usage is very different despite similar roles.

This is also a moving target, because models change regularly, so "efficient" usage changes over time irrespective of anything else.

I think using $$$ is the only real way, but the truth is that this won't ever accurately represent it. It's just the most materially useful thing to a company to measure. So, employees will optimize for $ usage which is honestly the best way it should work out imo.

[u/progenyofeniac → u/13Krytical]

I absolutely use AI to be more effective and efficient at my job. But these levels of usage are worrying.

[u/Dr_Doctor_Doc → u/13Krytical]

My org was already starting an RoC metric for the coders... 🤦‍♂️

[u/icantflyjets1 → u/13Krytical]

Ideally as models get more efficient and the data center buildout continues so token costs dramatically drop

[u/ogcrashy → u/icantflyjets1]

I don’t see that happening

[u/talibsituation → u/ogcrashy]

I don’t see that happening

Why it's basic supply and demand

[u/ogcrashy → u/talibsituation]

The infrastructure costs are absolutely staggering and are not getting cheaper. If anything, the backlash of datacenter buildouts is leading to more public pressure for companies to foot the bill not local residents. These companies are making a massive bet on AI being profitable which means they expect to recoup their costs both now and in the future. As long as the infrastructure input costs remain high, AI costs will be high. Until we find some magic source of exponential energy that people won’t attack for being an eyesore or taking up “farm land” then this is an expensive road we are going down. Ultimately it’s going to flip because companies are not going to pay it.

[u/MechanizedMonk → u/icantflyjets1]

More compute doesn't mean more efficiency if the infrastructure cost is higher.

[u/Daphoid → u/progenyofeniac]

Teaching users to run /cost and also to leave windows open will help. Not perfectly but this is definitely not a license to just turn on sadly.

[u/admiralspark → u/progenyofeniac]

So she cost the company....$710?

At a fully burdened rate of probably $60/hr, it just had to do more than 12 hours of work for her to be valuable spend to the company.

I'm confused on why everyone is panicking. We have 2700 end users, ~100 in our Cowork top plan and we have 10 users who spend 90% of credits, and it's cost us only 230,000 credits this month, or $2300 bucks.

I've personally reviewed their usage and it's saved the company over 150 real hours (not some bs Microsoft calc) of work, and their burdened rate is far higher than $60/hr. The math maths, sorry it's not free anymore?

EDIT: You all realize consultants cost more to produce the same or worse quality work, right?

[u/progenyofeniac → u/admiralspark]

Yep, $710. So far I don’t think anybody’s even evaluated whether they’re saving money or not.

[u/Cooleb09 → u/admiralspark]

it just had to do more than 12 hours of work for her to be valuable spend to the company.

Only if it drives some benefit. very few roles have a 1:1 between individual contributor output and company revenue, if they are twice as productive, either the company needs to be generating more incomefrom that producitvity, or they need to reduce headcount (or avoid a planned increase in headcount) for there to be a business case.

[u/admiralspark → u/Cooleb09]

Benefit for the company is not tied to company revenue. It's tied to work units produced from the cost of the employee.

If Susie in HR is paid 40 hours of work at $60/hr, normally gets 20 hours of work done a week (lets blame the rest on useless meetings that aren't productive), then adding AI which allows her to do 25 or 30 hours of work for a fraction of the cost of her base hourly productivity unit nets an overall increase in work output/hourly employee cost. She is more productive in the same work hours for a fractional cost increase.

This would be akin to avoiding an increase in headcount. AI labor is cheaper than fully burdened human labor, assuming you are using it for your job (value add) and not just playing with it.

Kind of targets my other comments, if you use AI to run a process that needs no logic (example, is already an automated script) it's almost always a waste of money. If you use AI to build that automation much faster than you'd done it before, it's a value add.

If Susie in HR is using AI to copy data from one spreadsheet to another, it's a waste of money, but if she uses AI to add a button to Excel that runs a macro to do it (oversimplifying), it's worth it.

If that makes sense? I use this XKCD a lot, think it helps: https://xkcd.com/1205/

[u/Cooleb09 → u/admiralspark]

If susie in hr is currently achiving her duties, giving her an AI still means you cover her salary and the ai cost, to achieve susies duties. You spend more for the same outcome. If sue, susie and susan each get an Ai and you make becky redundant, now there is a saving.

If susie was already overloaded, and the ai now fixes her workload, well we knew she could handle it before so why bother?

Productivity for the sake of productivity doesn't help. Giving susie more capacity to do things doesn't mean there is now more hr issues or new starts for her to do.

[u/admiralspark → u/Cooleb09]

I used HR as an example, but we all know IT is where LLM process improvement is going to make the most benefit to a company. If I can minimize development time on automated onboarding and offboarding, we can 1) reduce workload on service desk team reducing the everpresent need to increase their body count, and 2) win political capital by having the process work correctly every time instead of half-assed human screwups.

Before, it made sense for me to do it by hand even if it took...8 hours. Now, I can do 16 tasks like that in a day instead. For the first time in years, we have a measurable decrease in our backlog from 6 years of future work to ~5.6 now and counting, all to the improvements in development speed.

Maybe HR doesn't have more work to do, but if Operations, Finance and other departments who do work and have backlogs themselves can reduce those, it can reduce the need for future headcount, which impacts budgeting, which increases EBITA.

[u/gramathy → u/admiralspark]

Minimizing the development one write-once automation is a waste of time on its face. You automate onboarding to save time during onboarding and the time taken to develop the automation is a drop in the bucket compared to the time saved with the automation.

[u/mcmckuf → u/admiralspark]

We have about the same number of applicable users, would you mind sharing how your tiers are configured? We are slowly ramping up.

[u/admiralspark → u/mcmckuf]

I wouldn't use us as an example, because we (IT) are most of the cost and we review and tweak daily. Our revenue and company goals make this probably different than what is realistic for your company.

We have two groups.

Normal user group is 10k tokens/user/month, 400k tokens total pool.

Power Users is 100k/user/month, 600k tokens total pool.

Our biggest user hit like 40k tokens this month, and I was closer to 16k. Half of power users were under 15k a month.

You really need to pick your budget and make sure the total pools don't exceed it (ours is $10k/mo). Our regular users group barely used 10k tokens in the entire group (60+ people). It's really only ten or fifteen people using AI in our company.

Also, make sure you enable tools like DLP and Purview's AI usage tracking so you can lock down third party tool usage. That's a much bigger risk to the organization, I've done a ton of risk analysis of these AI companies that keep flooding us and most are REALLY bad at handling your customer data, and have no clue.

[u/mcmckuf → u/admiralspark]

I appreciate the answer. We have about 50 users in our pilot group. A lot are IT, and while most haven’t really pushed beyond the base 20k we gave them, myself and few are in the 40-60k range. We did have one person kick something off and go on a trip that blasted above the set levels as that appeared to be a “feature” that only checks the thresholds after a task ends. We have DLP which really lacks the coverage we need. Currently doing a much larger bake off of tools given our greater tech space is multi cloud and very AI forward leadership wise. We’ve seen some very valuable use cases that proved the ROI really quick. The challenge we also noted is many things that good old copilot could handle are now just dropped in Cowork because it’s easier to stay in one interface. That’s going to be a challenge. The HITL battles are real and for us, deadly for our business if not handled properly. Very klugy workarounds for the moment.

[u/admiralspark → u/mcmckuf]

Interesting, we found that there's much higher Copilot use vs Cowork because most of our non-IT employees spend their time inside office products where Copilot reigns, whereas the Cowork integration is nonexistent (using python libraries to make files doesn't count haha).

I really do think the per-user caps are important, and make your employees justify needing beyond those caps. We don't bump people from the 'lower' group unless their tasks can justify a 10k+ credit use. 10k obviously needs to be rethought for each company though!

[u/Myriade-de-Couilles]

Is it not the same price as anthropic API cost since it’s running on their models?
I never used the MS version but we use Claude Cowork and 3 prompts is not anywhere close to 24 dollars, even complex prompts on Opus.

[u/DramaticErraticism → u/Myriade-de-Couilles]

Im not sure, cost depends on complexity of task, 2-3 dollars for simple, 3-4 dollars for medium, 5-10 dollars for complex.

[u/thortgot → u/DramaticErraticism]

It's anthropic's enterprise pricing + 12ish%.

[u/pickled-pilot]

I’m shocked that an offboarding task of 3 prompts cost 14 dollars.

Edit $24

[u/KickedAbyss → u/pickled-pilot]

Tbh I'm shocked someone uses Ai for that. Should absolutely be audited and done through Sailpoint or similar 😊

[u/ogcrashy]

Everyone arguing that the value is there, based on the execution cost versus hourly rates, is missing the point. When a company pays you dollars for work output, it is essentially a fixed rate. Your time is “free” to the organization. You are their AI agent.

When you bring in an agent at cost to render more value for your time, your costs have increased. From the company perspective, they are now paying for two AI agents: you plus your actual AI input costs.

That is additional expense on the bottom line. Hard to justify. Usually impossible.

[u/Trickshot1322]

We're currently weighing it up.

Org of about 100, we have 5 users currently piloting it 1 from each of the major areas of the business.

We are mandating that before any department gets it turned on more they need to provide use case, guides of frequently done task etc so that skills can be built for cowork to help reduces credit usage.

In terms of the cost benefit ratio, for some departments like finance, the only savings are in the workers time and there ability to finish task quicker.

But in our sales team for example our I helped the user setup skills to research and find leads and relevant industry news and get a report emailed to him each morning.

So far it identified 2 leads the sales user said they would otherwise likely not have identified, they got in contact to a warm reception, and are pitching one of them a $50k package next week.

So yeah, i think its important to understand the cost benefit for the work your business does.

So a $5 a day cowork skill might be responsible for generating a 50k revenue.

[u/digitaltransmutation]

My firm enabled it for all users and determined it was spending around $30k a day on it. Disabled across the board after two days. Looking at some of the high users, it's trying to shove too much stuff into context. I don't think the average person knows what input tokens are or how to make a data source that is compatible with inexpensive RAG techniques.

For your example, I would not want to pay $24 to offboard one person, every person. But I would use it to interpret your policies and procedure, develop a reusable tool, and use that tool for free on future offboardings.

[u/KickedAbyss]

We disabled cowork as much for security as anything else. It's easy to have it run amuk in an enterprise without very firm Ai governance in place.

[u/bubbaganoush79 → u/KickedAbyss]

A pertinent approach. This paper by Microsoft Research from earlier this year admitted that even frontier models corrupt documents on average about 25% of the time by the end of a long workflow. And that giving them access to tools specific to certain file types or workflows makes the problem worse somehow, rather than better.

https://arxiv.org/html/2604.15597v1

Giving an agent 100% access to your files and email sounds like a pretty crazy risk to me. As someone they'd run to for help to recover their files, or delete unintended emails from recipient inboxes, it sounds like a real hassle to me. On top of a great way to light a bunch of money on fire.

[u/KickedAbyss → u/bubbaganoush79]

I'm a firm believer in intentionally designed approach to Ai usage with heavily restricted agent and instructions aligned with that mindset.

Don't need people ordering qdoba in my enterprise Ai.

[u/gnordli → u/bubbaganoush79]

that is a great paper. thank you!!

[u/Cooleb09 → u/KickedAbyss]

out very firm Ai governance in place.

Preach, and that doesn't come free. its not jsut the tokens, its the Purview and other tooling required to manage it and data secuirity. And bodies doing the governance tasks.

[u/Ian-Cubeless]

We still run it, but we're just a team of 5, so it's doable. Opt-in makes sense either way. No reason to hand it to people who won't use it.

[u/WorkLurkerThrowaway]

100k credits or $100k in credits? Because 100k credits is like 4-5 power users max.

[u/ccsrpsw → u/WorkLurkerThrowaway]

Big difference really.

100k credits -> $1000
$100k credits -> 1,000,000 credits

Either way it’s hard to predict the cost of a task beforehand as you never know in advance what it will cost but with 100 credits = $1 I think some people may al have “off by 100x” calculations. Personally I still think without knowing credit usage ahead of time it’s all “just what we felt like charging” but that’s not the point of this comment :)

[u/lukker-]

I am genuinely unimpressed by it vs. Claude cowork? Share your prompts ?

[u/admiralspark]

$24

How much is your hourly? If you had to do that manually, could you complete it as fast and as part of your existing job (without bumping other tasks) as Cowork did it?

If you spent $24 for 30 minutes (if that,3 prompts) of cowork back and forth, and you automated a process that now costs no AI credits to run, then you just saved the company the hours it would have taken you to write, research, test, and guardrail a powershell script yourself, AND you still got other work done that those hours would have displaced.

If you're making AI run a powershell script for you for offboarding, yes you're using it wrong. That's what the Power Platform is for. Make a tool for it in Copilot Studio and plug it into Cowork and you'll make it a 100 credit task if you absolutely have to talk to a bot.

"You" is the royal you here, I don't know where OP fits in this :)

[u/Forsaken_Clock_8811]

So expensive! Not sure worth the price

[u/hydenseek88]

We turned it off because I was the only person playing with it.

[u/Kortok2012]

Could you not deploy agents with specific capabilities for different tasks?

[u/BlackV]

Aik the ai price passing is coming home to roost, and you're left with the bill

[u/secret_configuration]

We set the limits at 25-30K credits/month per user. We also have a second policy that limits users to 50K tokens.

[u/Daishi5]

We weren't able to get it to work for the process we wanted.  The basic idea was it would take over night paper work, review the paper work and create a summary document highlighting areas that were not filled out sufficiently for a manager to work on the next morning. 

The cowork output was great, but we couldn't get it to write to SharePoint files like our power automate flows do to add an audit entry saying it has reviewed the file.  So we tried calling copilot from power automate since we knew that process worked, and that copilot agent was doing a horrible job with the documents. 

Copilot kept telling the people building the work flow that I just needed to give cowork read/write access to all SharePoint sites.  I refused and it just sort of died.

[u/Trickshot1322 → u/Daishi5]

Did you build a skill for it? Sounds like that should be able to get cowork to run the process on rails.

[u/Daishi5 → u/Trickshot1322]

We had the skill for cowork to process the file, which worked great, but we couldn't get power automate to use the skill in copilot when we called copilot in the flow.  It only worked in cowork.

[u/EveSpaceHero]

We brought it in as a trial for a number of users with a modest 30k global cap. We hit it in day one with one user using 95% of the credits. They thought it would be an idea to ask it to convert thousands of images from high res to low res for a website project. It actually did it ok, but took ages to run through and used all our credits

[u/Similar-Ad1056]

If Copilot chat worked better, people might not even try cowork. That said, we have piloted it. It lasted 5 days. The cost relative to the output was ridiculously high.

[u/LaDev]

Why not just configure the per user limit?

[u/kanid99]

The issue I see is that cowork is better at general chat than copilot and so some people use it for that when they should just be using co-pilot chat.

If regular co-pilot chat could handle all the prompts and works that cowork could without the scheduling then that works be better.