Outline:

• Why focus on performance? numbers, comparison
• How did we speed things up? selective computation
• Splitting data into min-batches by type
• Processing batches of data in datasketches
• Putting it all together: executing in pyspark on a Databricks cluster

About the speaker:

Jamie Broomall, Senior Software Development Engineer at Whylabs, Technical lead for whylogs

Github whylogs: https://github.com/whylabs/whylogs

Python examples of whylogs: https://github.com/whylabs/whylogs/tree/mainline/python/examples

Whylabs: https://whylabs.ai/

DataTalks.Club is the place to talk about data. Join our slack community!