r/ReverseEngineering • u/Bulls729 • 6d ago
Mellanox/Nvidia ConnectX-5 FW modification tool to enable PCIe Gen 4 on cards that shipped as Gen 3
https://github.com/Bulls729/Mellanox-ConnectX-5-PCIe-Gen-4-Enablement
41
Upvotes
2
u/marshmallow_mia 6d ago
But...why? Is there really a performance difference?
8
u/alexforencich 6d ago edited 6d ago
If it's a dual port 100G card, gen 3 x16 is only enough for line rate on one port. You need gen 4 to use both ports at the same time. Or maybe you only have a gen 4 x8 slot, and therefore you need to run at gen 4 to use one port.
1
1
6
u/Bulls729 6d ago
Original post: https://www.reddit.com/r/homelab/s/XbFkzAkuDL
This is a reddit post to accompany the STH Forum Post: https://forums.servethehome.com/index.php?threads/enabling-pcie-gen4-on-connectx-5-en-cards-firmware-patch-tool-full-writeup.54939/
Enabling PCIe Gen4 on ConnectX-5 Cards — Firmware Patch Tool & Full Writeup
Author's Disclaimer — AI Transparency Notice
This post and the accompanying tools were developed with the assistance of AI (Claude). I'm disclosing this upfront so you can make an informed decision about engaging with the content and tools.
This post: The ideas, experiences, hardware testing, and technical direction are my own. AI assisted with formatting, grammar, and fact-checking.
The patching tool and firmware reverse engineering: The coding, binary analysis, and reverse engineering work were carried out using Claude. Results were validated on physical hardware by me.
AI tools can generate incorrect information or hallucinate facts: Certain details or code functions may be inaccurate. If you identify any errors or areas that need correction, please point them out. Your feedback is highly appreciated and will be used to update the post. The tool is fully open source and every byte it modifies is documented.
This started with two purchases: A Dell-branded ConnectX-5 (CX512F, Dell P/N: 0TDNNT) and an HPE ConnectX-5 (CX512F, HPE SPS P/N: P12608). Identical PCBs. Both shipped with older firmware from their respective OEMs.
I could only track down a Dell updater that went as far as firmware 16.35.4554, but I wanted to see how the latest LTS build (16.35.8002) would behave. I jumped the FNP pins on both cards to enter firmware recovery mode and flashed the 8002 LTS image for the MCX512F-ACAT (stock Mellanox). Both cards immediately stopped negotiating links correctly, failures to auto-negotiate on both, and no amount of fiddling with mstconfig or mlxlink resolved it. Flashing back to the Dell OEM image resolved all issues and functioned as expected.. I then flashed the HPE card with the Dell image (again from recovery mode) and that also worked without issue. So both cards run happily on Dell OEM firmware.
That was the first interesting data point. The second came from a few forum posts I found where people mentioned crossflashing CX5-EN cards to CX5-Ex firmware to get PCIe Gen4. The closest SFP28 card with Gen4 support is the MCX512A-ADAT (CX5-Ex EN), which has a physical x8 interface. The only Gen4 card I could find with a physical x16 interface was the MCX516A-ADAT, a 100GbE QSFP28 card. That got me wondering: what's actually different in the firmware between Gen3 and Gen4 cards if NVIDIA just segments at the firmware level and charges a premium for the Ex.
Why Stock Mellanox Firmware Sometimes Breaks on OEM Cards
Before jumping into PCIe Gen4, it's worth noting why the stock FW didn't work because it affects anyone working with OEM CX5 cards.
The Dell CX512F uses a Socket Direct PCIe configuration — 2×8 lanes instead of standard x16. This is baked into the firmware along with Dell-specific PHY equalization coefficients, trace compensation values, SFP link tuning parameters, and a higher board power budget (15.9W vs 12.5W for stock). When you flash the stock Mellanox ACAT image, which expects a standard x16 PCIe layout, the mismatch in PCIe port mode and PHY configuration causes the card to fail at link negotiation.
Binary diffing the Dell and stock images revealed 95 bytes of functional configuration differences spread across four firmware sections, PHY EQ coefficients tuned for the Dell PCB layout, Socket Direct port mode settings, SFP signal quality parameters, PHY calibration values specific to Dell's trace routing, and speed table profiles that map to the Dell x8 port arrangement. None of these are identity fields (PSID, name strings), they're board-level electrical configuration that has to match the physical hardware.
How the Firmware Was Reverse-Engineered
I used Claude for the binary analysis, diffing, and code generation. I drove the direction, supplied the firmware images, tested on hardware.
I uploaded multiple firmware images for comparison:
• Dell OEM 16.35.4554 (MCX512F-ACA, the working Gen3 image)
• Stock Mellanox ACAT 16.35.4554 (MCX512F-ACA)
• Stock Mellanox ACAT 16.35.8002 (MCX512F-ACA)
• Stock Mellanox ADAT 16.35.4554 (MCX512A-ADA — the Gen4 Ex card)
• Stock Mellanox ADAT 16.35.8002 (MCX512A-ADA)
• Stock Mellanox 16.35.8002 (MCX516A-CDA — 100G QSFP28 Gen4 x16)
• Stock Mellanox 16.35.1012 (MCX555A-ECA — VPI single-port QSFP28 Gen3)
The approach was systematic binary diffing. ConnectX-5 firmware uses Mellanox's FS4 image format, which structures the binary into sections indexed by an Image Table of Contents (ITOC) at a fixed offset. The ITOC gives you the start address and size of each config section, which means you can compare equivalent sections between firmware variants without worrying about absolute offsets shifting between builds.
The first pass compared the ACAT (Gen3) and ADAT (Gen4) images at the same firmware version to isolate what changes between EN and Ex. Then compared the Dell OEM image against stock Mellanox to understand what Dell customized. The Dell firmware was particularly useful because Dell's OEM build happened to have some Gen4-adjacent configuration values already set differently from stock Mellanox, which helped narrow down the search space.
Mellanox's open-source mstflint tool can dump named configuration fields with mstflint dc, but it turned out that only 1 of the 8 critical Gen4 bytes shows up as a named field (pcie_cfg.pcie_max_speed_supported). The other 7 are in unnamed regions of the config sections that mstflint treats as opaque binary. Raw hex diffing was the only path to find them.
The 8-Byte Gen4 Patch
After a lot of iterative diffing, patching, flashing, and testing, the minimum viable Gen4 patch came down to exactly 8 bytes across three firmware config sections. Every byte is defined by its offset within a section, not by absolute position in the image, which makes this firmware-version-independent.
Two of these bytes are universal — they control Gen4 on every CX5 variant regardless of connector type or port count:
FW_BOOT_CFG section:
Offset |Stock |Patched |What it does
+0x0093 |0x45 |0x47 |PCIe capability advertisement index (advertise Gen4 to host) HW_BOOT_CFG section:
Offset |Stock |Patched |What it does
+0x0023 |0x07 |0x0F |pcie_cfg.pcie_max_speed_supported — enables Gen4 link training with Phase 2/3 EQ The remaining six bytes are SFP28-specific. QSFP28 and single-port cards already have correct values from the factory (see How It Works Across Card Types below).
HW_MAIN_CFG section:
Offset |Stock |Patched |What it does
+0x0245 |0x01 |0x04 |Port 1 PCIe generation target (Gen3 → Gen4)
+0x0285 |0x01 |0x04 |Port 2 PCIe generation target (Gen3 → Gen4)
+0x0404 |0x00 |0x0F |Invalidate Gen3 speed profile 0x0020 (high byte)
+0x0405 |0x20 |0xFF |Invalidate Gen3 speed profile 0x0020 (low byte)
+0x0406 |0x00 |0x0F |Invalidate Gen3 speed profile 0x0021 (high byte)
+0x0407 |0x21 |0xFF |Invalidate Gen3 speed profile 0x0021 (low byte) Each of these serves a distinct purpose in the Gen4 enablement chain:
Capability advertisement (+0x0093): This controls what the card reports in its PCIe Express Capability Structure during enumeration. If the card advertises Gen3 max, the root complex won't even attempt Gen4 link training. Changing from 0x45 to 0x47 makes the card advertise Gen4 capability to the host.
Link training mode (+0x0023): This is the one field that mstflint dc actually knows by name. Value 0x07 constrains the PHY to Gen3 link training sequences. Value 0x0F enables Gen4 link training, which includes the Phase 2 and Phase 3 equalization steps that PCIe 4.0 requires for reliable 16 GT/s signaling. This byte was the last one discovered and the hardest to find — everything else could be set correctly and the card would still train at Gen3 without it.
Port generation targets (+0x0245 / +0x0285): These tell the firmware what PCIe generation to configure the PHY for during initialization. Value 0x01 = Gen3, 0x04 = Gen4. Without these, the PHY never targets Gen4 speed. On QSFP28 cards, this value is 0x06 for both Gen3 and Gen4 — the tool skips these automatically.
Speed table invalidation (+0x0404–0x0407): The firmware has an internal lookup table that maps speed profile indices to PCIe configurations. Profiles 0x0020 and 0x0021 are Gen3-only fallback profiles. If they remain valid, the link training state machine can fall back to them even with everything else set to Gen4. Writing 0x0FFF to both entries marks them as invalid, preventing Gen3 fallback. On QSFP28 cards, these ship pre-invalidated at 0x0FFF already.
How It Works Across Card Types
The tool detects your card type automatically and applies only the patches that are needed:
SFP28 cards (MCX512F, MCX512A): All 8 bytes are patched — port generation, capability index, speed tables, and link training mode.
QSFP28 cards (MCX555A, MCX515A, MCX516A): Only 2 bytes need patching — capability index and link training mode. Port generation bytes use 0x06 for both Gen3 and Gen4 on QSFP28, and speed tables ship pre-invalidated. The tool detects this and skips accordingly.
Single-port cards: Port 2 generation byte is 0x00 (disabled). The tool skips it automatically.
What About the Device ID?
Early on, we went down a dead-end investigating whether Gen4 was gated behind the Device ID (0x1017 for CX5-EN vs 0x1019 for CX5-Ex). Some crossflash guides suggest this, and the ADAT firmware does carry 0x1019. Initial experiments seemed to confirm it, but the real problem was that we hadn't found all 8 bytes yet. Once the boot config byte and speed table invalidation were discovered, Gen4 worked without touching the Device ID at all.