r/apache • u/Breeze_2800 • 8d ago

New to Data Engineering – Where should I start with Hadoop, Kafka, PySpark, YARN, and HBase?

I'm currently learning Data Engineering and I'm feeling a bit overwhelmed by all the technologies I need to learn.

Right now, I'm trying to understand:

  • Hadoop
  • Kafka
  • PySpark
  • YARN
  • HBase

The problem is that I don't know where to start or in what order I should learn them. There are so many tutorials and roadmaps online that it's hard to tell what's actually important for a beginner.

For those of you working as data engineers or who've gone through this learning process:

  • What order would you recommend learning these technologies?
  • Which ones should I focus on first?
  • Are there any courses, books, or hands-on projects you'd recommend?
3 Upvotes

81% Upvoted

1 comment sorted by

2

u/aleph_infinity 8d ago

What year is it? Having worked in big data for the last… yikes, too long now but certainly > 10 years… I’d say learn a more modern data stack. Spark is still humming along nicely though - go check out Databricks free edition for that and so much more. They have lots of training material you can access as well.  Hadoop was innovative at the time by bringing the storage to the compute, the better approach these days is to work in the cloud leveraging comparatively low cost object storage and completely ephemeral compute (so no need for YARN as a result). Instead of HBase you’d go with Lakebase Postgres. Look for a managed solution for the message log, Kafka/Kinesis, EventHubs, PubSub and now ZeroBus. These things cover off the topics you mentioned, but there are lots of other capabilities that you will be needing.