Three years. Three year’s I’ve been building the Internal Developer Platform, based on Kubernetes. I wanted to share my leanings with you. Cause it takes way more than slapping few pieces of technology together.
This as the next part of my career. Back in 2022 I made a video about transition from Sysadmin to DevOps - check it out.
I Little Bit About Myself and Where I’m Coming From
Making computers works has been my thing since my teenage years. I was this village computer fixing guy - during beautiful times when Windows 98 or XP could just break itself, without any particular reason.
My enterprise story started in 2012 and continues to this day: I went from managing single SBS 2008 Windows Server, to managing dozens of them. From VMware Virtualisation to Azure. From coding in PowerShell to C# Golang and many other languages. Instrumenting CI/CD pipelines.
My latest experience was about building Engineering Platform on Kubernetes. This post documents what I learnt in the process. What Platform Engineering is and what it means to you and your users.
If you want to learn about my career, check my LinkedIn
What Is Platform Engineering?
Platform Engineering is the discipline of developing and maintaining an engineering platform.
The platform gets consumed by some users - in my case our users were internal employees. Although sometimes the app developed internally was shared with employees from outside of the organisation.
Guardrails and security should be at the centre of the platform - the last thing we want is for data or IP to leak!
Audience and Personas
It’s important to understand who your audience is - it’s different to prepare the platform for someone who just learnt how to put some Python code together, and a seasoned developer. Both personas have same goal - to deliver value to their customers in the form of an app.
But their experience differs. E.g. Aiming for citizen developers might require creating extra step-by-step tutorials and explanation how each piece of tech comes together. Creating checklists might give them confidence boost of how to follow the deployment process.
Professional developers might require some extra advanced feature or less restriction as they know what they need. Checklists might slow them down - as they had years to develop the sense of what’s right and how to achieve it.
It’s important to realise AI often blurs the line of someone’s experience, and it can only be discovered during later phases of deployment.
Generally you want to be close to your users to understand their needs and get direct unfiltered feedback. The last thing you want to do is to build something in isolation that nobody will use.
Engineering Platform should empower its users with self service. The last thing your users want is to fill another form. The last thing you want is to process another form, manually. You must remember that every manual task put on your team, is time taken away from core development.
But It Works on My Machine
You will find angry users whose app works perfectly in their pure local environment. The moment they deploy - nothing works.
Pipelines failing. App not starting up. Or starting and crashing due to memory shortage.
It’s easy to get into never ending support cycle. Trying to fix somebody’s code. Dealing with somebody’s ambition that “my app takes 10 seconds to start up on your platform” where there’s no business justification for speed.
You must set clear boundaries, of what’s your - the platform team - responsibility and what’s your users. You must be very strict about this split of responsibility.
Now there’s no harm in having some support time with your users from time to time. You will learn how they use your platform. But it cannot take your entire time.
Your job is to build the platform. Not to fix somebody’s shoddy app.
Be Selective
Here’s the ugly part - you must be very selective of which features you implement, which feedback is truly important. The problem here is getting the architecture right. If you implement something wrong in your application, you, and your team will need to work with until it gets fixed.
If you implement something wrong on the users side, it often cannot be easily fixed. Usually you will need to migrate them (and migration is the middle name of platform engineering, like it or not) off that feature eventually. Nobody likes migrations. I truly believe we should be calling it “Platform Migrations Engineering”
It’s easy to fall into the HIPPO (Highest Paid Person’s Opinion) trap, or the loudest complainer. You must see through that, and ruthlessily decide what will benefit the platform, not a single person.
Example of bad decision: imagine in AKS you want to provision storage accounts in Azure using private endpoints. You decide you will use /24 network for it. At best case you will be able to create around 200 of storage accounts. At worst around 50. When you hit the limit, that’s kind of game over if you cannot extend the network. Probably you will end with some workaround, or yet another migration…
Your decisions matter, and you need to own them.
It’s generally good idea to ask how the feature can be scaled out 10 or 100 times, and how can it be monitored. Before the implementation starts.
How Does Platform Engineering Differ from DevOps?
DevOps in theory should have been culture change to the entire organisation. Everyone working together to empower its employees and customers.
In practice we’ve got DevOps Engineers surrounded by DevOps tooling. Working in isolation.
DevOps enabled teams to build and ship faster than before. It created another problem: bespoke pipelines per each project. It was almost the distributed effort.
At one of the meetings somebody said: “Give DevOps a story, and you’ll get 5 pipelines in return”.
Platform Engineering tries to unify all the effort to create ubiquitous work. It reduces mental load of repetitive tasks of how to configure this pipeline, and how to deploy and secure the app.
Right executed platform will save your project time and money. If you’ve got budget scheduled for 1 year, and platform enables you to deliver within 10 months - you’ve saved 2 months of your budget. You can build another thing or improve current processes.
The longer I think about it, the more I’m convinced Platform Engineering is what DevOps tried to be at its core principals.
Should Every Organisation Have Its Own Engineering Platform?
No. Building and maintaining your own platform requires significant people element to it: developers, architects, technical writers, product managers. You also need enough number of customers/internal developers who will build on the platform.
You can think of a platform a soil, and your developers as farmers. Without developers, the platform won’t be worth much. Think what happened to Windows Mobile - a great platform which nobody wanted to develop for. It just died - despite huge investment from Microsoft.
One indication of the need of a platform is shadow IT - you’ll see (if you’re lucky heh) your employees using some unapproved services to deploy their apps. The intellectual property will be at danger. What happens if the employee leaves - who will gain access to such unmanaged app? Who will pay for it, once the free tier is gone?
This is especially accelerated in the times of AI, when a person can develop the app by prompting, and having agent to deploy it somewhere.
You might also notice some similar problems multiple teams need such as requesting a server to run some app. Failing builds across the board for some reason. Hundreds of unpatched Dependabot issues.
You need to look, carefully, and know what’s going on in your estate.
Summary
Platform Engineering is both exciting and difficult. It requires long term thinking and architectural sanity. You must understand what platform needs, and what your users truly need.
You will inevitably run into migrations - something what makes sense today, won’t make anymore after X amount of time. This is assuming you had time to develop the feature. Often you will work against time pressure - where everything cannot be engineered as good as you might have wanted to.
And most importantly, platform is worth nothing without its users. Without stakeholders buy-in. This is initiative which will take serious amount of resources before you start getting any return on investment.
But once you went through the pain. Once you see the number of deployment going up. Clusters scaling out. Your colleagues saying your platform enabled them to deploy the app, and that it was fun.
Then you know it was all worth it. And it’s only the beginning.