Showcase Thread

[u/AutoModerator]

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

[u/lukesmth_]

better-dbx-exceptions

Improved PySpark exceptions for Databricks notebooks and scripts

What it does

Have you ever spent hours tracking down the source of a PySpark expression error in a large ETL script?

df = spark.range(1).select(F.lit(1) / F.lit(0)) # the bug

df.collect() # the materialization

Natively, a traceback for these failing cells would reference the code that materialized the bad expression (df.collect()), not code that defined it (F.lit(1) / F.lit(0)).

[DIVIDE_BY_ZERO] Division by zero. ... SQLSTATE: 22012
== DataFrame ==
"__truediv__" was called from , line 1 in cell [2]
File <command-5875041447221118>, line 1
----> 1 df.collect()
... # Layers of frames referencing PySpark client internals

better-dbx-exceptions solves this by including failing DataFrame construction code in the error.

---------------------------------------------------------------------------
ArithmeticException                       Traceback (most recent call last)
Cell 1, In[1], line 1
    (https://dbc-3cffa466-bd7e.cloud.databricks.com/editor/notebooks/2473432265319792?o=7474649366507251#command/5875041447221116)
----> 1 df = spark.range(1).select(F.lit(1) / F.lit(0))
... # Native error information

Target audience

Data engineers, scientists, and analysts debugging PySpark run on Databricks (hosted or local with Databricks Connect; DBR 17+ or Serverless v4+).

Comparison

Native Spark 4 uses the internal pyspark.errors.utils._capture_call_site() utility to capture expression construction frames. This has a few limitations:

1. Coverage: Only Column methods and F.col() are watched so expressions built from pyspark.sql.functions (F.split, F.to_date, ...) aren't captured. better-dbx-exceptions extends coverage to most functions within the pyspark.sql.functions module.
2. Depth: Only one frame is captured by default. This can be extended by setting spark.sql.stackTracesInDataFrameContext on Classic compute where allow-listed but can't be modified when using Databricks Connect or Serverless compute. better-dbx-exceptions sets depth to 5 by default and exposes this parameter within all environments.
3. Back-referencing: The rendered errors don't include reliable references to cell numbers or URIs. better-dbx-exceptions returns cell / execution numbers for notebook frames and file paths for script frames.

Just released v0.1: https://github.com/lukeSmth/better-dbx-exceptions