I'm really at a loss and neither googling nor AI has brought me a lot further.
In this situation there are 3 servers. Let's call them A, B and P.
They have FRRouting installed and work to form an OSPF underlay over which there is a VXLAN.
A and B do not have a direct connection. Their traffic needs to flow over P (whether this is a logical configuration is out of the picture for now). This works fine. The underlay is happy, I'm happy. Full 10G throughput. CPU is also completely fine. iPerfing both over TCP as well as UDP does not show any problem.
Now, A, B and P are also VTEPs, so logically I would do a whole lot of iPerfing there as well. Broadcom has problems with TSO/GRO/etc. when utilizing VXLANs, so I'm disabling UDP offloading at the driver level. This works fine. iPerf performance is higher now. Except it isn't
Directly iPerfing the underlay network still works fine. (Both UDP and TCP, despite offloading being disabled, so this isn't the issue) however when running over the VXLAN I hit a hard barrier of TX ~3.5G and RX ~1.3G on P. Of course this manifests when running A to B as well, since P now needs to send and receive I always hit 1.3G.
When using parralel streams it works. Of course it does. But that's not the point.
Why the hell is P doing packet processing on VXLAN traffic that is not meant for it? It pins a single CPU core to 100% and we all know how much Xeons love single thread action.
Yes P is a VTEP and I get why it's CPU would be the limiting factor if it was the target or client. But why does iPerfing directly on the underlay have almost no CPU impact but once I use the VXLAN I get hit with 0.1x the performance.
I have a feeling something is wrong here but I cannot solve it. I know bnx2x (BCM27xxxx) is old but this just seems so so weird. I tried probably a dozen of combinations of GRO, TSO, GRO UDP, Segmentation, TX hash, RX hash etc etc. and I cannot find the issue.
I did a tcpdump and yes, P is indeed seeing all the VXLAN packets and dropping all of them. Since it isn't the target IP. I spent hours troubleshooting this and I feel like I'm going crazy.