My Transition from SE to Data Engineering

After years in Software Engineering, I decided it was time for a change. With AI rapidly automating most of the coding work, I felt the field was losing the spark it once had for me. So I'm making the switch to Data Engineering, a field I've always been drawn to, and this is the story of my plan, my progress, and what's driving me forward.

My Background and Why I'm Making the Switch

I used to work as a Software Engineer for a Singapore-based company, and I eventually left that job. While I sometimes second-guess that decision, the core reason I left is still very much there: software development just doesn't excite me the way it used to.

With the rise of LLMs, I genuinely believe coding as we knew it is largely becoming automated. Most of the new applications being built today can have their coding done by AI, faster and often better than most humans. What still requires human judgment is the design side: understanding client requirements and architecting solutions. The parts left for humans in traditional software development are mostly high-sensitivity systems or legacy applications. That's one of the main reasons I decided to move into Data Engineering.

There are a few other reasons too. I always wanted to work in data or ML, but after getting into software engineering, I got tunnel-visioned and gradually drifted away from that path. I also think Data Engineering has strong demand right now, and it's a relatively future-proof field with a less saturated market compared to Software Engineering. I've seen many Software Engineers make this transition successfully, including a few close friends, so it feels like a natural path to follow.

The Plan

My plan started with getting a few cloud certifications to have something concrete to show, and honestly, to give myself a confidence boost. That's why I sat for the AWS Solutions Architect Associate and AWS AI Practitioner exams. It did take me three months to complete both, which was a bit slower than I'd have liked, but it's done.

From there, I built out a roadmap. The foundation comes first: solid SQL and Python skills before diving deep into Data Engineering concepts. So the first few weeks are dedicated to that.

After the foundation, the original plan was to jump straight into core Data Engineering concepts. But things shifted a little. I found out about the Microsoft AI Festival and the Databricks Learning Festival happening in June, both offering certification vouchers. Microsoft is giving away 100% Azure certification vouchers, and Databricks is offering 50% off. That's hard to pass up.

Originally, I hadn't planned on any Azure certifications at all. But since the vouchers are essentially free, I decided to go for the Azure Fabric Data Engineer certification. The catch is that these vouchers expire: Microsoft's within two months and Databricks' within three, so I've had to accelerate things a bit.

So here's how the revised timeline looks. Through early June, the focus is on completing the SQL and Python foundations. In mid-June, I'll join both festivals and claim the Databricks and Azure certification vouchers. The last three weeks of June will be spent studying core Data Engineering concepts. Early to mid-July is for working on small DE projects, studying Azure Fabric, and attempting the Fabric certification.

After that, I'll take four weeks off from mid-July to mid-August to focus on my MSc end-semester exams. Once the exams are done, three weeks go toward preparing for the Databricks Data Engineer Associate exam. And if time permits, I'll take a shot at the AWS Data Engineer Associate certification as well.

The timeline is tight, and I may have to compromise on a few things. But more importantly, I don't want this journey to just be about collecting certifications. Personal projects are critical. I need real work to showcase, not just badges on a resume. So alongside all of this, I'll be building projects that demonstrate what I actually know.

Doing all of this while keeping up with MSc assignments is probably the hardest part of this whole thing. But I think it's manageable.

One tool that's been really valuable throughout this process is Claude. It's been like having a personal teacher available at any time. I can ask even the smallest questions and get a reliable, thoughtful answer. I'm also planning to use Claude Code to guide and review my project work, and I'll write an article about that experience once I've tested it out properly.

How It's Going So Far

I'm still in the very early stages. I've covered SQL and part of Python so far, and the Microsoft AI Festival starts next week, so I'll be joining that while finishing up the Python foundation.

I'm not sure exactly how everything will play out, but I'm genuinely excited to find out. I have a good feeling this will mostly go according to plan.

The one thing I need to be careful about is the Databricks certification. Even with the 50% voucher, it still costs around $100, so I need to be well-prepared before attempting it, unlike the Fabric cert where the risk is lower. I also need to make sure my MSc doesn't slowly slip through the cracks. That's equally important.

Let's see how things go.