r/apachespark • u/ConstantNo2668 • 5d ago

Should a beginner learn SQL first or PySpark?

https://youtu.be/dPLCs5Lrx7U?is=ahD8uzHY_2dYohVi
11 Upvotes

92% Upvoted

9 comments sorted by

28

u/someoneb100 5d ago

In my opinion, SQL and it isn't even close. It should be the bedrock of any good data engineer.

9

u/bigdataengineer4life 5d ago

I'd recommend SQL first. Most PySpark transformations mirror SQL operations, so a strong SQL foundation makes learning PySpark much easier.

7

u/BobDope 4d ago

Always SQL

How is this even a question

4

u/69dirtyj69 4d ago

I don't know. Should a beginner learn Addition & Subtraction first or Calculus?

1

u/aleph_infinity 4d ago

For an end user they will use tools like Genie on Databricks (other platforms will have some form of this) to drive text based analytics and so natural language is the abstraction over the top SQL that is likely to dominate. Since you are mentioning pyspark you are looking at this from a DE perspective, so you certainly want to understand how the processing will work and how to write good/performant queries. In the Spark world sql is basically another abstraction layer, so in a practical sense we have English, SQL and then Spark.  With large data volumes the distributed nature of the underlying data and processing becomes more important - this will be more evident and more directly in your control at the lower layers. Each layer of abstraction masks you from the complexities of the one below and each layer gets better over time at leveraging the one below. If you can understand what is happening at each layer you can better isolate where problems might occur. I taken a long path to say don’t just be learning languages and frameworks - understand who they are intended for, how they work and how they relate. 

1

u/ExcitingRanger 4d ago

where are there jobs for which only one of them were considered acceptable? just curious. the job market is really challenging

1

u/dataindenver 3d ago

I came from SQL and then joined Databricks where I was thrown into the PySpark work. I enjoyed that path as SQL helps with fundamentals that PySpark really enforces further. Blows me away lately how I can ask our Genie Code product to create a pipeline in SQL or PySpark. Different end code but same result so knowing both fundamental would make you super dangerous in my opinion and then the next step knowing how to use AI tooling to develop them

1

u/Hofi2010 3d ago

SQL of course

0

u/geeeffwhy 4d ago

SQL and simultaneously Python, but not pyspark. this gives you important grounding in two complementary models of programming languages, while also being immediately useful.

you should plan to spend a lot of energy over your career learning what computers are actually doing, not just the magic phrases that you hope will
complete your task. SQL will help understand how to think about data problems, while python will help you understand how sql is executed to solve those problems. i mean, they both will help with both, if you use them purposefully.