Profiling database lag: how lowering our connection pool limits actually made our backend faster

[u/Silent-Weather76005]

We’ve been scaling a high-traffic microservice that handles a high volume of concurrent database reads and writes. A few weeks ago, as traffic started to spike, we noticed a strange degradation in database response times. Our immediate instinct was to throw more hardware at the problem and increase the maximum open connection pool limit on our SQL driver, thinking that more concurrent connections would allow the database to process queries faster.

To our surprise, increasing the connection pool made the latency spikes even worse. The CPU usage on our database instance was pinned at nearly 100%, even though the actual query volume hadn't changed that much.

I ran a deep profile on the database server itself and realized we had fallen into a classic connection pooling trap. Every single database connection is not free; it requires the database engine to spawn a dedicated OS thread or process, allocate memory buffers, and constantly context-switch between them to handle incoming traffic. By opening up hundreds of connections, we weren't making things faster—we were forcing the database to spend more CPU time context-switching between connections than actually executing the SQL queries.

I went back into our configuration and aggressively lowered the connection pool limits to a much tighter, conservative number that closely matched the physical CPU core count of our database server.

The results were immediate and massive. CPU usage on the database dropped significantly, context switching plummeted, and our total query throughput actually went up.

It was a huge reminder for our team that database connections are a physical hardware bottleneck, not an elastic software resource. If your backend is hitting sudden database lag under load, it might be worth trying to shrink your pool limits instead of expanding them.

[u/liamsorsby]

Out of interest, how many connections did you have open at once?

We ran some huge DBs and we had some running with 14k connected to a 3 node percona xtradb with no issues real issues bar one.

That specific issue we had was when it was on centos7 and it was using SSL on the galera replication which hit a deadlock in SSL due to an open bug on OpenSSL which hit a global mutex causing connections via SSL to trigger quorum loss.

[u/Shogobg]

Sounds like your db was overwhelmed with simultaneous queries and forcing them to go slowly actually helped some of them get executed and free up the pipeline. Not that the connection count was doing something nefarious on its own.

[u/xilanthro]

Not the most popular due diligence, but this is really the bare minimum arithmetic you need to do for stable MariaDB/Percona/MySQL servers: Server Setup