Simplifying Testing of Spark Applications

### Overview In this presentation, Megan (data scientist at Spotify) and Han Wang (staff machine learning engineer at Lyft and author of Fugue) discuss the challenges of testing big data applications written in Apache Spark and introduce Fugue as an abstraction layer to simplify the development and testing lifecycle. While testing reduces production risks, running tests directly on Spark frequently incurs high CPU costs, slow iteration loops, cluster spin-up delays, and cumbersome serialization logic. The speakers demonstrate how decoupling core data transformation logic from Spark's execution engine enables fast, local, test-driven development (TDD) that scales seamlessly to production clusters.

### Key Topics - **Pitfalls of PySpark and Spark UDFs:** PySpark runs on top of the Java Virtual Machine (JVM) via Python APIs. Using vanilla PySpark user-defined functions (UDFs) requires transferring pickled row-by-row data between the JVM and Python workers, introducing high serialization overhead. While Pandas UDFs (introduced in Spark 2.3) leverage Apache Arrow for zero-copy memory transfer, they introduce complex and varying semantics (such as differing type hints, series, iterators, and group-by behaviors). - **Testing Friction in Distributed Environments:** Unit testing native PySpark code typically forces developers to spin up local or remote Spark clusters. When using client tools like Databricks Connect, local PySpark environments are overridden, forcing remote executions that can take several minutes per test run. Additionally, APIs like `mapInPandas` require spark-specific iterator signatures, explicit schema definitions, and multiple wrapper functions, making logic difficult to isolate and parameterize. - **Fugue Abstraction Layer:** Fugue acts as an engine-agnostic abstraction layer that preserves native Python and Pandas execution locally while allowing code to execute across Spark or Dask without modification. It eliminates the need for cluster spin-up during unit tests while guaranteeing behavioral consistency across engines. - **Test-Driven Development and Downsampling:** Han Wang highlights downsampling as a fundamental principle in big data development. By downsampling data deterministically and writing tests against expected business logic first, developers can rapidly catch bugs using local Python debuggers and test frameworks before scaling workloads to full datasets.

### Technical Details - **Tokenization NLP Pipeline:** Megan illustrates a text tokenization workflow using a Kaggle toxic comment classification dataset. Using PySpark's native `mapInPandas` required custom wrapper functions, iterator handling, explicit Spark schema strings, and boilerplate parameter passing. Re-implementing the function in Fugue allowed the logic to remain entirely native Python/Pandas with simple schema hints. Changing execution engines from local Python to Spark required only altering the execution engine parameter. - **Fugue SQL and TDD Pipeline:** Han demonstrates sentiment analysis on the 2020 US Election Tweets Kaggle dataset (1.7 million tweets across 16 Parquet files). Using Fugue SQL and Fugue's notebook integration (`fugue-jupyter`), the workflow downsamples the dataset by 0.01% to run quick iterations. A modular pipeline is built using NLTK (lemmatization, stop words) and scoring (VADER and TextBlob) followed by state- and hashtag-based aggregations. Every component is developed using strict TDD, asserting output schemas, sign agreements, and filtering rules. The complete workflow is encapsulated in an engine-agnostic function that can be tested locally on Pandas mock data or passed a SparkSession fixture via pytest without altering internal business logic.

### Takeaways - Decoupling computational logic from Spark objects (such as `DataFrame`, `RDD`, and `Schema`) removes unnecessary dependencies and accelerates unit testing. - Downsampling large datasets enables fast, local iteration, full debugger usage, and immediate feedback cycles. - Using an abstraction layer like Fugue allows developers to achieve near-100% unit test coverage using lightweight local execution engines, while retaining the ability to deploy unchanged code to production Spark clusters.