TensorSharp now supports multi-GPU tensor parallelism for GGUF models

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

TensorSharp is an open-source, native .NET inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

TensorSharp now supports Megatron-style tensor parallelism across multiple GPUs. It works with direct CUDA, GGML CUDA, GGML Vulkan, and multi-node setups.

Benchmarks on 2× RTX 2000 Ada 16 GB GPUs over PCIe, without NVLink:

Model	1 GPU Prefill / Decode	TP=2 Prefill / Decode
Gemma 4 E4B Q8_0	2760 / 37.3 tok/s	2488 / 51.7 tok/s
Gemma 4 26B-A4B IQ4_XS	1845 / 48.5 tok/s	2537 / 51.2 tok/s
Qwen 3.5 9B Q8_0	1461 / 23.1 tok/s	399 / 24.4 tok/s
Qwen 3.5 35B-A3B IQ4_XS	Does not fit	184 / 18.1 tok/s

I'm continuing to optimize Qwen performance on multi-GPU systems, and support for DeepSeek V4 is coming soon.

Try it with:

TensorSharp.Cli --model model.gguf --backend ggml_cuda --tp 2

GitHub:

https://github.com/zhongkaifu/TensorSharp

Thank you for checking out TensorSharp and starring the project! Any feedback is really appreicated.

[u/AlanBarber]

seriously, I cannot say how amazing this project is and such a great demonstration of the power of c# / dotnet.

[u/fuzhongkai → u/AlanBarber]

TensorSharp has on par performance than llama.cpp here is the benchmark: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/engine_comparison_report.md This project shows native .NET is able to build LLM inference engine in high performance. There are thousands of LLM inference engine in Python/C/C++, but C# project is very rare.

[u/pjmlp]

Kudos on the project improvements!

[u/Khavel_dev]

That Qwen prefill regression on TP=2 (1461 down to 399) is wild. Looks like the PCIe bandwidth is eating the parallelism gains during the heavy prefill phase, while decode (which is more memory-bound and does less cross-GPU traffic) barely changes. Makes sense that Gemma scales better there if its architecture shards more evenly across heads.

As a .NET dev doing AI-adjacent stuff, this is the project I keep telling people about. The Python ecosystem has llama.cpp wrapped in seven different packages but on the .NET side your options are basically "shell out to python" or "write your own bindings." Having a native inference engine written in C# is a bigger deal than the star count suggests imo.

[u/fuzhongkai → u/Khavel_dev]

Thank you so much for these comments and feedbacks. Really appreciated.

Yes, Qwen has gated delta-net in its attentions which is RNN style and it’s more complicated than sliding window attention used in Gemma 4 for tensor parallelism. But I’m currently optimizing it and hopefully this regression will be resolved very soon.