Learning and Implementing the PyData Stack
Hey there!
Python has been part of my toolkit for a while, but this year I actually sat down and learned the PyData stack properly — NumPy, pandas, and the surrounding tooling that turns raw data into something useful.
Why I needed this
My MSc dissertation is an AI-assisted cyber defence system for IoT-based smart networks — comparing rule-based detection against machine learning models like Random Forest and SVM on intrusion datasets. That’s not a project you can do with for loops over lists. I needed proper data wrangling.
What I picked up
- NumPy for the array operations underneath everything else — much faster and much less painful than plain Python lists once the dataset gets large.
- pandas for loading, cleaning, and reshaping intrusion detection datasets and generated telemetry before it goes anywhere near a model.
- Basic feature engineering — turning raw network traffic fields into something a classifier can actually learn from.
- Comparing model output — structuring results from rule-based detection vs. ML models side by side instead of eyeballing separate scripts.
The honest part
The hardest part wasn’t the syntax, it was the mindset shift — thinking in terms of vectorised operations on whole columns instead of looping row by row. Once that clicked, cleaning and reshaping data stopped being the most tedious part of the project.
This is very much a “still learning” post rather than a “mastered it” post, but it’s already paying off in how I’m building the detection pipeline for my dissertation.
Happy coding! 🚀