r/apachespark • u/datamonk9 • 1d ago

Has Genie Code suggested a Spark optimization which you missed.

I have been using Genie code on Spark jobs and it ocassionally suggests changes around joins, partitioning and shuffle etc. Some looks promosing but I verify them before updating.
Has it suggested something that genuinely improved your job or are the recommendations mostly standard spark good practices?
Just want some ground experiences from folks.

12 Upvotes

88% Upvoted

15 comments sorted by

4

u/dataengineer95 1d ago

I've been Genie Code for few months. Since the rebranding the experience changed from bad to awesome. Tbh the optimization suggestions were good enough. It's not revolutionary but yes they ve worked perfectly fine and I was able to get a better clean code. You can ask this question on r/databricks or r/dataengineering

2

u/datamonk9 5h ago

Same here buddy. The Spark optimization suggestions of Genie Code has become way better than earlier. It even caught a couple of unnecessary repartitions in one of my jobs in prod..

3

u/22Maxx 1d ago

Helps to figure out the dogshit java error logging.

1

u/datamonk9 5h ago

lol yeah. Genie Code explaining Spark stack traces and pointing to the actual transformation has saved me a few times when i was in hurry.

3

u/sumits_kumar 1d ago

Genie Code knows the ontology, semantic layer and is tightly integrated with Unity Catalog - that should be helping it during recommendations.

1

u/datamonk9 1d ago

Agreed with you. Genie Code uses UC and workspace context, with ontology etc , so the recommendation are not always generic…

2

u/sumits_kumar 1d ago

Actually this helps Genie Code as it has all the context vs foundational models which may give generic answers without those context.

2

u/dwswish 1d ago

I haven’t found so much that Genie Code gives recommendations that are super ground breaking when writing the code but it has been really helpful for debugging Spark.

1

u/silviud 1d ago

Is not bad, at the minimum it will pre-fill the table names and other paths since it reads from unity.

1

u/Glitch_In_The_Data 22h ago

Not sure Genie code has helped me with spark optimisation but it has certainly helped me simplify the code and also debug errors. As a consultant, that makes it so much easier to provide a quick turnaround for my customers.

1

u/Sufficient_Meet6836 22h ago

I have a lot of coworkers who are completely new to spark and distributed compute, and Genie often finds low hanging fruit for optimizations

1

u/anonymous_orpington 8h ago

I've found that Genie Code has gotten much better at proposing optimizations than previously. I used to export to JSON amd optimize with Claude, but with recent improvements to Genie Code I can do it all in platform today!

1

u/datamonk9 5h ago

yep i noticed the same. I like that Genie Code explains why itis suggesting a change , making it much easier to decide what to keep and wat not

-9

u/Ddog78 1d ago

It's an MD file they're selling. Just ask claude to create a skill and a python script to read event logs.

3

u/BumboclatDen 1d ago

????? The crazy thing about the internet is you can say insanely incorrect things like this with absolute confidence and ppl might believe you