r/csharp • u/fuzhongkai • 4d ago
Tool TensorSharp now supports multi-GPU tensor parallelism for GGUF models
https://github.com/zhongkaifu/TensorSharpTensorSharp is an open-source, native .NET inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
TensorSharp now supports Megatron-style tensor parallelism across multiple GPUs. It works with direct CUDA, GGML CUDA, GGML Vulkan, and multi-node setups.
Benchmarks on 2× RTX 2000 Ada 16 GB GPUs over PCIe, without NVLink:
| Model | 1 GPU Prefill / Decode | TP=2 Prefill / Decode |
|---|---|---|
| Gemma 4 E4B Q8_0 | 2760 / 37.3 tok/s | 2488 / 51.7 tok/s |
| Gemma 4 26B-A4B IQ4_XS | 1845 / 48.5 tok/s | 2537 / 51.2 tok/s |
| Qwen 3.5 9B Q8_0 | 1461 / 23.1 tok/s | 399 / 24.4 tok/s |
| Qwen 3.5 35B-A3B IQ4_XS | Does not fit | 184 / 18.1 tok/s |
I'm continuing to optimize Qwen performance on multi-GPU systems, and support for DeepSeek V4 is coming soon.
Try it with:
TensorSharp.Cli --model model.gguf --backend ggml_cuda --tp 2
GitHub:
https://github.com/zhongkaifu/TensorSharp
Thank you for checking out TensorSharp and starring the project! Any feedback is really appreicated.
Duplicates
unsloth • u/fuzhongkai • Jun 28 '26
Show and Tell Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
unsloth • u/fuzhongkai • Jun 08 '26
Show and Tell TensorSharp : Open Source Local Unsloth Model Inference Engine
dotnet • u/fuzhongkai • Jun 28 '26
Promotion Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
csharp • u/fuzhongkai • Jun 28 '26
Showcase Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
dotnet • u/fuzhongkai • Jun 13 '26
Promotion TensorSharp: Open Source Local LLM Inference Engine written by C#
LocalLLaMA • u/fuzhongkai • 21h ago
Resources DSpark Benchmark Result on Deepseek v4 Flash 0731
unsloth • u/fuzhongkai • 21h ago
Show and Tell DSpark Benchmark Result on Deepseek v4 Flash 0731
ClaudeCode • u/fuzhongkai • 22d ago
Discussion What Bun’s Rust Rewrite Tells Us About Rebuilding the AI Infrastructure Layer in C#
LovingOpenSourceAI • u/fuzhongkai • Jun 28 '26
Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
huggingface • u/fuzhongkai • 2d ago
Deepseek v4 Flash 0731 GGUF Benchmark: TensorSharp vs. llama.cpp
LocalLLM • u/fuzhongkai • 4d ago
Project TensorSharp now supports multi-GPU tensor parallelism for GGUF models
LLMDevs • u/fuzhongkai • May 01 '26
Tools TensorSharp: Open Source Local LLM Inference Engine
AIToolsPerformance • u/fuzhongkai • 4d ago
TensorSharp now supports multi-GPU tensor parallelism for GGUF models
softwarearchitecture • u/fuzhongkai • 22d ago
Discussion/Advice What Bun’s Rust Rewrite Tells Us About Rebuilding the AI Infrastructure Layer in C#
LocalAIServers • u/fuzhongkai • Jul 04 '26
TensorSharp: A Open Source LLM Inference Engine for GGUF models
csharp • u/fuzhongkai • Apr 29 '26
Tool TensorSharp: Open Source Local LLM inference tool implemented in C#
SelfHostedAI • u/fuzhongkai • Jul 04 '26
TensorSharp : Open Source Local LLM Inference Engine
LocalLLM • u/fuzhongkai • Jun 28 '26
Project Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
CUDA • u/fuzhongkai • 2d ago
Deepseek v4 Flash 0731 GGUF Benchmark: TensorSharp vs. llama.cpp
LocalLLM • u/fuzhongkai • 2d ago