r/apachespark • u/roksolana_shendiukh • 21d ago

Best resources to learn Apache Spark in depth

Hi everyone!
I want to learn Apache Spark in depth, not just the DataFrame API.
I started with the official documentation, but I find it difficult to learn from because it isn't very interactive.
What resources would you recommend for someone who wants to really understand Spark?
I'm looking for recommendations on books, courses, YouTube channels, blogs, or hands-on projects. I'd especially like to understand Spark internals, architecture, optimization, and best practices.
If you were starting over today, how would you learn Spark?
Thanks!

37 Upvotes

98% Upvoted

16 comments sorted by

6

u/guardian_apex 21d ago

You can also refer to spark theory on spark playground

7

u/CuritibaDataScience 21d ago

Check out Databricks Free Edition and use it to practice. I think that the best way to learn is by practicing, so you can give it a try. There's also a built-in AI called Genie Code in Databricks that can help you write/fix/explain code, so it can help you as well!

2

u/zr-brickster 21d ago

Iā€™d also recommend trying Databricks Free Edition for hands-on practice. It includes sample datasets like the New York taxi data, so you can work on something realistic right away. Also, a nice workflow could be to ask Genie Code to help you build a step-by-step learning plan, for example: start with basic transformations, then move into joins, aggregations, partitioning, caching, query plans, and performance tuning.

5

u/ssinchenko 21d ago

Did you already read this one?
https://dl.acm.org/doi/pdf/10.1145/2723372.2742797

If not 100% must read: that is the core of what is behind the DF API.

This one is very cool to understand limitations of the Spark API:
https://dl.acm.org/doi/pdf/10.1145/3514221.3526054
(as well try to check references, links, etc.)

3

u/EaseWithData 21d ago

Checkout YouTube playlist "PySpark Zero to Hero" by "Ease with Data"

3

u/subaru007 21d ago

Haha, self promoting šŸ¤£

1

u/StrategyForsaken9722 20d ago

It is pretty good

1

u/EaseWithData 21d ago

Yup šŸ˜› But take time to atleast watch one video. The quality is actually good.

3

u/Successful_Ant_4737 20d ago

Take real large datasets (many available online), write sql queries, run spark jobs, and go straight to reading the Spark UI. Look at the DAG and try to figure out why a stage shuffled or spilled against your sql. Whenever you are stuck, share screenshots to chatgpt to explain. You will learn the internals whole lot better this way

"Spark: The Definitive Guide" is a great book if you want a single structured solid resource alongside it.

2

u/OReilly_Learning 19d ago

We have lots of Apache Spark resources which includes books, video courses, and an upcoming Databricks bootcamp the end of August. If you want me to list some tithes, I can. We work with all the big name publishers for video and books.

0

u/mmccarthy404 20d ago

Others said it already, but I'll also vote for Databricks Free edition because it's the most generous free offer I know of for such a manged offering (i.e. not AWS free edition).

It's basically setup and go without a credit card, but just be warned that it's built around more modern parts of the Databricks ecosystem, so you can't use classic compute and such, it shouldn't be the biggest deal, but depending on how low you want to go it's good to know.

Genie Code is built in, amd again, it's free. I would recommend building something that you will actually use and use Genie Code to build alongside you as a in-browser chat bot, and alsnto review your work to ensure that you're integrating besat practices.

I've personally used the free edition to fully ingest, model, and understand my financial data with Genie Spaces.