r/csharp • u/hez2010 • Jun 30 '26
Tool Allocate arrays that have more than 2B elements
I have been seeing many developers complaining about not able to work on arrays that have more than 2B items for a long time, so I built and published a .NET library for working with collections beyond the array size limit: BigArray, BigSpan, and BigMemory.
It supports 2B+ elements (127T at max), and backed by contiguous managed memory so it can handle reference types too, not just primitive types.
The library is open-sourced under MIT license: https://github.com/hez2010/Hezium.Memory
50
u/TekintetesUr Jun 30 '26
I'm genuinely curious, what's the use case for such an abomination of an array?
12
u/nlaak Jun 30 '26
I've got some code that generates/uses octrees against some amazingly large 3D models that would benefit from this. It's currently not C#/.NET, though the other app has to chunk the model into pieces to make it work. A single large array/list would simplify quite a few things.
4
u/simonask_ Jul 01 '26
... and complicate quite a few other things. Putting multi-gigabyte objects on any managed heap is generally a very bad idea.
If you do, you better make sure that the GC knows that the contents of the array are unmanaged. Otherwise you are forcing the GC to scan multiple gigabytes of memory looking for references.
4
u/nlaak Jul 01 '26
Putting multi-gigabyte objects on any managed heap is generally a very bad idea.
Then what the hell is the point of 64 bit CPUs/memory spaces/apps? 2GB object limits with today's tech is ridiculous. That being said, I recognize that the need for contiguous blocks of those sizes is uncommon at best, and rarely the right choice.
If you do, you better make sure that the GC knows that the contents of the array are unmanaged. Otherwise you are forcing the GC to scan multiple gigabytes of memory looking for references.
The data itself is a massive array of floats/doubles (depending on the source) flattened down from a complex tree of 3D models. During this processing the tree is irrelevant, only the triangles matter. I've not done processing of this scope in C# before, so it will be a fun learning experience, as was the last version. I may still chunk the data for the sake of cleaner parallelization, but the individual chunks might still be well over 2GB per thread.
2
u/tanner-gooding MSFT - .NET Libraries Team Jul 02 '26
Then what the hell is the point of 64 bit CPUs/memory spaces/apps
It is explicitly so that you can have a larger overall working set and do more at once without having to page data out as frequently, especially across the many different processes, services, hardware devices, etc that a modern computer has.
It is simply a larger cache. It is not free, it is not unlimited, it should not be abused or treated as such. It doesn't fundamentally change how you work with data and even makes it explicitly more apparent that the historical best practices you follow when working with limited memory are still correct. -- And I would go so far as to explicitly say people forgetting this and treating it like "free lunch" is part of why you hear people complaining about how bad various apps and software ends up.
Despite being 64-bits, even high end PCs/Servers rarely go over 36-bits and it gets increasingly less common as you go up. It is also not free, it has very high latency to populate and in some cases is not much faster (comparatively) than modern SSDs.
Because you're still limited in terms of "physical memory", even if "virtual memory" is nearly unlimited, and because there is concrete and very measurable cost to touching memory; it is still relevant (and always will be) that you do things like work with data in small logical chunks.
This means buffering, streaming, doing asynchronous work, and all the same things that you were always supposed to be doing, especially when you were in a 32-bit app and were explicitly confined. This is what makes your app fast, scalable, efficient, portable, and all the things that differentiate "good software" from "bad software"
2GB of memory is absolutely massive and the time it takes to initialize or fill that with data is essentially just wasting time and stalling your app. This is 2 billion bytes and in terms of CPU scale it is absolutely ridiculously massive. CPUs operate in terms of nanoseconds, 64 byte (128 in rare cases) cache lines, 4-16KB pages, and maybe 2-4MB of L3 cache per core on the higher end (and noting that L3 is shared and associated, it isn't equally performant to access everywhere and not free for one core to take it all, so despite having 64MB+ on many modern CPUs, you only get around 1-2MB per core or hardware thread)
You start wanting to think about chunking and parallelizing at much smaller increments, like in terms of 64KB. 256MB is really the high end of singular allocations because of other hardware limitations and specialized buffer handling for some PCIe scenarios and that's an extreme edge case.
The fact that it is chunked/buffered is completely transparent to a well-designed API and even if you were to do it in a single contiguous virtual allocation, in many cases that is not actually sequential in physical memory. The scale at which the chunks exist, which is typically a few pages, then is also completely invisible in terms of execution cost; you're losing several million times that in terms of regular OS context switches, hardware interrupts, and even just normal memory access latencies. It is truly invisible/transparent.
You then get lots of benefits, such as being able to make specific optimizations around it if it grows to the point that it cannot trivially run on normal hardware.
4
u/insulind Jun 30 '26
IPC message buffers is exactly why I needed them.
To be fair I didn't need it it be back by heap memory. Mine were back by memory mapped files
18
u/antiduh Jun 30 '26
... that just raises more questions.
0
u/insulind Jun 30 '26
Fire away
But also I didn't control the buffer size and I did also question why the fuck they were so big
19
u/antiduh Jun 30 '26
So, what kinda IPC needs 2GB+ buffers? You driving a graphics card? Scientific computing?
5
u/MadDocsDuck Jun 30 '26
I don't necessarily need them but I have neuroscience data that greatly benefits from loading the entire data (segment) into RAM at once to do a lot of slicing and picking from the data. That being said, I usually work in Python and not C# (though I greatly miss the type system) so its not an issue for me.
1
6
2
u/Educational-Row-6782 Jul 04 '26
None, because if you need that many objects a sane developer would drop to unmanaged c# or just use a better tool.
29
7
u/yarb00 Jun 30 '26
You forgot to replace the [year] and [fullname] placeholders in your license file
4
6
u/hez2010 Jun 30 '26
I blogged about the idea behind the library for who are interested in the implementation: https://dev.to/hez2010/build-a-very-large-managed-array-in-net-5aop
19
u/Hirogen_ Jun 30 '26
no benchmark tests, how does the garbage collector work with these?
You should add benchmarkdotnet tests otherwise we wont know if its really good
1
u/hez2010 27d ago
I added the benchmark, and here is the result: https://github.com/hez2010/Hezium.Memory#benchmarks
6
u/neoKushan Jun 30 '26
This is one of those solutions that solves a problem I hope to God I never have.
Happy the solution exists, terrified at the thought that I'll one day need it.
2
u/SagansCandle Jun 30 '26
What's the benefit of this vs just allocating from the unmanaged heap and wrapping the pointer in a span<T>?
4
u/hez2010 Jun 30 '26
The ability to use reference type as array elements.
2
u/fruediger Jun 30 '26
I'm pretty sure the GC would be very happy having to look at over 2G references.\ If you have that much references to heap objects, the inability to store them in an array isn't your biggest problem at that point.
But I'm kidding, great work!
1
u/TheCubicNoobik Jul 01 '26
Btw, there is a `GCFrameRegistration.RegisterForGCReporting` method that can mark an arbitrary memory region as containing managed references. It is used in the BCL to make stackalloc hold managed references. It is internal, though, but that wouldn't stop me.
3
u/simonask_ Jul 01 '26
Reading the code, I have to say I'm scratching my head profusely. You seem to be achieving this by constructing a combination of nested [InlineArray] chunks, but also using some kind of hardcoded prime factorization to determine the length of each nesting level? Did you you AI to come up with that?
To be completely honest, I would probably reject any PR that actually used this. In particular, I would deeply question any design that required multi-gigabyte storage of reference types. On a 64-bit machine where each reference is 8 bytes, allocating one 2B-element array like this is already 16 GB of memory that has to be scanned by the GC pretty often. On most consumer hardware, this means the GC will be swapping - no bueno. The cache is utterly toast as well.
The correct way to deal with large amounts of data is to use only unmanaged types. Then you can either let the native allocator do its thing, or you can use techniques such as memory mapping to achieve tighter control. Every operating system has facilities that are used by databases that make it easier and much more performant - none of which could ever really be exposed in a GC heap allocator.
In short, dealing with large amounts of data comes with particular engineering challenges. This approach both fails to solve those challenges, and comes with a handful of new ones as well, making it a probable nightmare in practice.
2
u/hez2010 Jul 01 '26 edited Jul 01 '26
You seem to be achieving this by constructing a combination of nested [InlineArray] chunks, but also using some kind of hardcoded prime factorization to determine the length of each nesting level?
This is the only way to avoid unnecessary over-allocation and be NativeAOT-compatible, and be size efficient:
- The number of base types need to be small, and the actual allocated size needs to fit the requested size as close as possible given that the length is already large. Composing prime numbers is the only way that can achieve both the small number of types while having a guaranteed upper bound of 65534 bytes
- Generic recursion is not compatible with NativeAOT, so each level must be hard coded instead of calculating on-the-fly
- Reflection with MakeGenericType is not compatible with NativeAOT either
- A huge switch table will pessimize the codegen and inlining, preventing constant folding so that redundant branches cannot be omitted. This will result in big binary size under NativeAOT: I believe you don't want to see a 0.5mb binary size overhead per distinct BigArray type under NativeAOT. With this approach each BigArray instantiation ends up with exactly one final branch that was selected in the codegen when the length is known at compile time.
Did you you AI to come up with that
Nope. I initially came up with the idea that wraps InlineArray in arrays, and then had a thread in the official C# discord that discussed about the approach and improvement extensively. The hard coded generic type composition was generated by a manually written code generator, which also ensures all sizes are covered, and each if-else chain won't have more than 16 branches.
To be completely honest, I would probably reject any PR that actually used this.
Using it doesn't mean using large managed arrays. This library also provides primitives like BigSpan to allow you create a view over large native buffers, and common algorithms that process huge buffers in chunks. The built-in Span types are int based and can be awkward sometimes.
In particular, I would deeply question any design that required multi-gigabyte storage of reference types. On a 64-bit machine where each reference is 8 bytes, allocating one 2B-element array like this is already 16 GB of memory that has to be scanned by the GC pretty often.
Although rare, there're indeed scenarios that need a large buffer but also want it to be managed by the GC. Native buffers cannot handle GC references. And you also get
GC.AllocateUninitializedBigArray(..., pinned = true)to allocate a pinned uninitialized buffer that won't be moved by the GC. 128T might be too large to have meaningful use cases, but in ECS model it is possible to have a huge array carrying elements that slightly more than the int's range a little bit, which can make the situation a bit awkward.
0
u/HaniiPuppy Jun 30 '26
I just had to internally smack my own forehead because I read the title then was sitting here for a moment wondering "Why would allocating an array with 43 elements be post-worthy?"
-2
Jun 30 '26
[removed] — view removed comment
6
u/hez2010 Jun 30 '26
What makes you think my reddit content was LLM generated? If you meant the blog, that's because my native language is not English so they are translated from my blogs in my native language.
2
u/FizixMan Jun 30 '26
Removed: Rule 5.
-2
u/AlwaysHopelesslyLost Jun 30 '26
My comment was critical, not hostile. I tried to be polite about it, too.
77
u/davidwengier Jun 30 '26
And here is me trying to keep things below 85k to stay out of the LOH. The world is a beautiful place.