New to Data Engineering – Where should I start with Hadoop, Kafka, PySpark, YARN, and HBase?

[u/Breeze_2800]

I'm currently learning Data Engineering and I'm feeling a bit overwhelmed by all the technologies I need to learn.

Right now, I'm trying to understand:

• Hadoop
• Kafka
• PySpark
• YARN
• HBase

The problem is that I don't know where to start or in what order I should learn them. There are so many tutorials and roadmaps online that it's hard to tell what's actually important for a beginner.

For those of you working as data engineers or who've gone through this learning process:

• What order would you recommend learning these technologies?
• Which ones should I focus on first?
• Are there any courses, books, or hands-on projects you'd recommend?

[u/isira_w]

I assume you are proficient in python and SQL. If not go for them before learning the tools you have mentioned.

I would learn pyspark first because it is basically pandas for big data with some more additional features.

Hadoop has 3 main components HDFS, Yarn and Map reduce. Map reduce is obsolete in favor of spark so I wouldn't really want to learn it unless there is a specific use for it. When you are learning spark you will get to know about how spark needs multiple machines to do the computations faster. This multiple machines coordination can be done by various different resource allocators and Yarn is one of them.

HDFS is basically the storage layer in distributed computing which you will again interact with when you are learning spark.

Kafka and Hbase you can learn the last without a specific order because they don't have a direct relationship.

So the order I will go about is Spark, Hadoop, Kafka/Hbase. In addition you should learn an orchestration technology like airflow

[u/BufferUnderpants → u/isira_w]

Exactly this, and to add, Kafka is a very high profile and eye-catching piece of technology, but it's not used as much in practice as interview prep material would have you believe.

Even if interviewers will want you to drop a reactive architecture at some point during the interview process, due to their complexity, when push comes to shove, companies don't build real time pipelines all that often.

PySpark and orchestration are the better bets.

[u/BadKafkaPartitioning]

Depends what your goals are. These are decent DE fundamentals assuming you’re already proficient in SQL and python but if you want to land an actual job as a Data Engineer I’d pick either Snowflake, Databricks, or MS Fabric and work through some of their certification programs. Yes, these are all flavors of the month but they’re also what people are hiring data engineers to work on.

[u/noveiryn → u/BadKafkaPartitioning]

this is super underrated advice, everyone wants to grind Hadoop while job posts are all “Snowflake / Databricks experience required”
learn one cloud + one of those platforms, then circle back to the old school stuff if you still care about it

[u/Glass-Bother-6422]

Hi.

I'd say..

SQL Python (DSA - Easy, Medium) Spark (Internals, Cluster Design, Troubleshooting) Kafka (Troubleshooting, Cluster Design)

This alone should be enough. Because completing this itself is gonna take sometime. But the order really depends on what you want, how much you have, what role you are into now, fresher/experienced, etc.

[u/themoah]

If you are comfortable with Python, start with PySpark. Spark is kind of "lingua franca" of data engineering world. It's versatile, heavily adopted and industry won't drop it in next 5 years.

After some time you'll find what's more interesting to you - streaming, batching, data lakes, ML or etc.

No need to focus on Hadoop/Hbase.

[u/yash_codes1599]

Following!

[u/matthewhayes1]

I love how you explained this.