Lightning Talk Intermediate

Boosting Data Processing: Performance Tune Pandas

Rejected

Session Description

This talk will be a in-depth exploration of techniques to enhance the performance of Pandas, the powerful data analysis library in Python. It will cover strategies, tips, and best practices for optimizing data processing workflows, leading to faster and more efficient analysis.

Recognizing the challenges posed by big data, we're well aware that Pandas can struggle with large datasets. Given that optimization is integral to tech, this talk delves into effective strategies for accelerating Pandas operations be it simple transformations on data or data export/imports to databases. I will also cover alternate supporting libraries to use and simple modifications to existing code to speed up execution.

We will have a live demo with code snippets demonstrating the usage and performance comparison as opposed to traditional methods which are used widespread.

This session aims to address 4 key points:

1. Why pandas is slow when it comes to handling big data?

2. Slight code modifications to existing pandas code syntax

3. Using different libraries to speed up execution - like SQLAlchemy, NVIDIA’s RAPIDS cuDF library among others

4. Performance comparison between proposed and existing methods

Drawing from personal experience, I'll share tried-and-tested methods to optimize Python scripts using Pandas and reduce pipeline execution time, ultimately enhancing resource efficiency.

References

https://pandas.pydata.org/docs/user_guide/enhancingperf.html

Session Categories

FOSS

Speakers

Asha Holla Analytics Developer | Bloom Value

Asha is a seasoned data professional with over two years of experience in Big data and AI. Currently, She's proud to be a part of Bloom Value, a US based AI startup focused on delivering innovative solutions to the US healthcare market.

Asha's journey into this field began as she found the title of data scientist intriguing, but it was working with data that truly captured her imagination. She sees data not only as a science but also as a creative outlet with endless possibilities waiting to be explored.

Beyond her professional role, Asha is deeply passionate about advocating for women in tech. Actively participating in community events and initiatives, she find immense satisfaction in giving back to the community. She also enjoy sharing my knowledge and experiences through tech blogs and posts, sharing my experiences, learnings on the day to day, mixed in with bit of humor. Her goal is to bridge the gap in spaces where information is sparse, based on my firsthand experiences in the field.

For Asha, being able to make a difference, both within her company and in the broader tech community, is incredibly fulfilling.