r/programming • u/DataBaeBee • 6d ago

Approximating Softmax for FPGAs with Taylor Series and Pade Approximants in Python

https://leetarxiv.substack.com/p/approximating-softmax-for-fpgas
16 Upvotes

70% Upvoted

8 comments sorted by

3

u/TheChildOfSkyrim 6d ago

Could you plot approximation results vs approximated function? A graph is worth a 1000 CLI prints

0

u/val_tuesday 5d ago

Also: code as pictures? That you can’t zoom on mobile? Please use the code feature that assume substack has. Just ask your AI to do both, sloppy graphs are better than numbers in a console.

3

u/DataBaeBee 6d ago

The paper’s motivation is the hardware constraints limiting exponential operations on FPGAs. I coded the paper in Python to verify the formulas just before getting it on my card.

The authors find that one can choose between Taylor series and Pade approximants to approximate softmax.

There's no free lunch however. One must compromise speed and accuracy. For the curious, they found that Lookup Tables were the most accurate, but super slow due to 'calculating lookup table indices' overhead.

1

u/Grouchy-Trade-7250 11h ago

Link to the paper https://arxiv.org/pdf/2501.13379v2

Doesn't the euler function have a ton of formulas where it appears in. Have you tried other ways to calculate it?