Table of Contents
Last week I unpacked the core components and building blocks of a GitHub Workflow, the YAML structure, what a job is and how it works, and touched on variables and secrets. This week we’re taking that a step further and building a reusable pipeline focused on platform delivery.
I want to do something a little different. Coming from a platform and infrastructure background, the examples you often find online for Azure DevOps and GitHub workflows are focused on developers. Platform engineering is different, we’re not building an application, we’re building a platform for developers to consume. A platform workflow needs to focus on different areas than a developer-focused workflow deploying an application.
In this post I want to show you how to build a reusable workflow that enforces your policy and governance, and more importantly doesn’t blow up your production environments. One caveat worth stating upfront though: the workflow is just one part of the puzzle. The governance model and platform design that sits around it matters just as much. I’ve covered that in more detail across my other posts on building the platform and governance.
There are four key pillars that our workflow is built on:
- Consistency – a reliable, dependable system your team can trust
- Security – security first, no exposed secrets, no shortcuts
- Auditability – governance and change control is foundational
- Reusability – build it once, use it everywhere
Let’s walk through how each of these shapes the workflow we’re building.
The Workflow Anatomy
I walked you through the building blocks of the pipeline structure in last week’s post, GitHub Workflow Fundamentals.
Trigger
Not every workflow should trigger automatically, and for platform delivery, I’d argue that’s often the wrong default.
Although focused on Azure DevOps, I previously wrote a guide on building pipelines that run on pull requests, and doing so gives us some valuable process automation and governance controls that form part of our standard work process. But when deploying a platform using a reusable workflow, automatic triggers aren’t always the right call. Sometimes you need an extra level of control, and more often than not you want to pass in parameters to customise the pipeline run before it executes. Automatic triggers make that significantly harder.
That said, we don’t use a single trigger type across all our workflows. We use different triggers depending on what the workflow is doing, and I’ll cover each as we build out the full picture.
Environment and Secrets
The golden rule is never expose secrets in code. GitHub gives us a few ways to handle this properly.
At the most basic level, secrets can be stored at the repository or organisation level. As a general rule, if a secret needs to be used across multiple workflows, store it at the organisation level. The Infracost API key is a good example of this, one key stored once, available to every workflow that needs it. Even at the org level, secrets can be restricted to specific repositories, which is good practice. You don’t always want every new repository in the organisation to have access to your shared credentials.
For workflow or workload specific secrets, store them at the repository level. Beyond that, GitHub also supports environment level secrets. A GitHub Environment represents a deployment target, think DEV, UAT, PROD. If you have a separate Terraform state backend per environment, each with its own storage account, environment secrets are where those credentials live.
That covers the mechanics, but where possible we should be going a step further and avoiding secrets altogether. Secret management carries overhead, and that overhead is increasingly unnecessary when secretless options exist. If you can use federated identity, use it.
OIDC and Federated Identity
I won’t go deep on OIDC and federated identities here as I’ve covered this in detail in my earlier post on Entra Workload Identities. For the purposes of this post, the key point is that we use federated identity to connect our GitHub workflows to Azure without storing a client secret anywhere.
The setup involves three steps:
- Create an application registration and service principal in your Entra ID tenant
- Add federated credentials to the service principal, scoped to your GitHub repository and branch or environment
- Store the tenant ID, subscription ID and client ID as GitHub secrets, but no client secret
That last point is what makes this approach powerful. There is no secret to rotate, no secret to leak, and no secret to expire at an inconvenient moment.
Actions
GitHub Actions form the primary building blocks of our workflows, but there is a security consideration worth being aware of before we go further.
Actions are essentially reusable scripts created and maintained by third parties. When you call an action you reference it like this:
- uses: actions/checkout@v4
We tend not to think too hard about what that actually means. We are pulling a script from a remote repository and executing it as part of our pipeline. You should have validated what that action does before using it, but the version tag itself is the problem. Tags like @v4 are mutable, a maintainer can move that tag to point at a completely different commit at any time. If that repository were compromised, or a developer pushed updated code under the same tag, your workflow would run it without any warning.
The protection against this is to pin your actions to a specific commit SHA rather than a version tag:
- uses: azure/login@8c334a195cbb38e46038007b304988d888bf676a
A commit SHA is immutable. Nobody can move it. The tradeoff is readability and maintenance, which is where Dependabot comes in. Dependabot can automatically open pull requests to keep your pinned SHAs up to date as new versions are released. I’ll cover that in a separate post, but it’s worth knowing the two go hand in hand.
Concurrency Control
Multiple instances of the same workflow can run at the same time in GitHub, which can cause various issues. This is different to how Azure DevOps handles it, where each pipeline run is queued and runs in sequence. In GitHub we need to prevent this from happening through the use of the concurrency key.
concurrency:
group: production-deploy
cancel-in-progress: false
The group keyword creates a concurrency group name. Each run within the same group will be queued and run in sequence. The cancel-in-progress boolean allows us to define what happens if a second run is triggered while one is already in progress. Set it to true to cancel the running instance, or false to queue the new run and wait for the current one to finish. For Terraform deployments, always use false. Cancelling a running apply midway is dangerous.
A good approach is to make the concurrency group dynamic, scoped to the environment being deployed to rather than a static name:
concurrency:
group: deploy-${{ inputs.environment }}
cancel-in-progress: false
This means two deploys to production will queue behind each other, but a production deploy and a dev deploy can still run simultaneously.
Reusable Workflows
There are two parts to reusable workflows worth understanding separately.
The first is building a workflow that abstracts deployment specifics. When deploying Terraform code we want our workflow to follow the standard steps of init, validate, plan, apply. But there will always be elements that need to be unique per deployment, such as the environment target or the name of the state file. This is where workflow parameters come in. Rather than hardcoding these values, we pass them in at runtime. Hardcode them and the reusability is gone, you end up with multiple near-identical workflow files and a maintenance headache every time something needs to change across all of them.
The second part is reusable code snippets within a workflow. If you come from an Azure DevOps background, this is the equivalent of pipeline templates. A good example would be the Terraform init, validate, plan sequence. This gets called during PR validation and again during deployment. Rather than duplicating that code across multiple workflows, you create a single reusable workflow containing those steps and call it from wherever it’s needed using workflow_call, passing in the inputs it needs.
jobs:
deploy:
uses: your-org/platform-workflows/.github/workflows/terraform-plan.yml@main
with:
environment: production
The key point here, and this is where it becomes genuinely powerful for platform teams, is that the reusable workflow can live in a central repository owned and maintained by the platform team. Every team across the organisation references it. You make a change once and it propagates everywhere. That is the reusability pillar in practice.
Job Outputs and Dependencies
I touched on dependencies in my previous post, but a quick refresher. Jobs within a workflow run in parallel by default. If you have a job that runs Terraform validate, another for Terraform plan, and one for Terraform apply, all three will start at the same time, which is far from ideal. To prevent this we explicitly set dependencies between jobs using the needs keyword, so they run in order rather than in parallel.
jobs:
validate:
runs-on: ubuntu-latest
steps:
...
plan:
needs: validate
runs-on: ubuntu-latest
steps:
...
apply:
needs: plan
runs-on: ubuntu-latest
steps:
...
The other problem to solve is passing data between jobs. Each job runs independently in its own environment, so variables and outputs from one job don’t automatically carry over to the next. We address this by explicitly defining job outputs.
jobs:
plan:
runs-on: ubuntu-latest
outputs:
tfplan: ${{ steps.plan.outputs.stdout }}
steps:
- id: plan
run: terraform plan
apply:
needs: plan
runs-on: ubuntu-latest
steps:
- run: echo "Plan output was ${{ needs.plan.outputs.tfplan }}"
The output from the plan job is referenced in the apply job using needs.plan.outputs.tfplan. For a Terraform workflow this is important from an auditability perspective. Passing the plan output through to the apply job means what gets reviewed and approved is exactly what gets deployed, not a fresh plan generated at apply time.
The Pitfalls and What to Avoid
Walk Before You Run
The main issue I see when teams adopt new workflows is that they run before they walk. They try to put everything in at once and then wonder why it doesn’t work, spending days troubleshooting and rebuilding. The fix is simple, build and iterate progressively. A great workflow is built over time and will have gone through tens of iterations before it really starts to shine. But when it does, your work isn’t finished. New technologies emerge, changes are unavoidable, and what was a great workflow can quickly become outdated. Live and learn from your experiences, and get early feedback at every stage.
Treat Your Workflows as Production Code
Close behind building too fast comes the teams that build their workflows, push them out, and never maintain them. Worse still, some make updates on the fly without any consideration for the impact. This always leads to disaster and is a great way to lose the trust you are trying to build in your platform. Workflows should be treated as production code. That means version control, governance and security.
The Thousand Lines of YAML
Another pattern I see regularly is workflows that start well but gradually become far too complex. Thousands of lines of YAML, multiple jobs, stages and approvals, dependencies all over the place, outputs at every level. Troubleshooting becomes painful and making changes becomes a risk in itself.
The golden rule is keep it simple. Build what you need, nothing more and nothing less. If you’re familiar with the SpaceX engineering philosophy, the principle is that every requirement, every component, every step should be challenged. If you can’t make a good case for why it’s there, remove it. Apply the same thinking to your workflows. It’s easy to get carried away plugging in the next cool tool, but more often than not it doesn’t belong in your pipeline.
The Full Workflow
I’ve put together two complete workflows as an example, but here’s the deal, don’t just copy and paste them. Take the time to read through them, understand how they work, and then go and make them your own.
The two workflows work together as a pair. The main reusable workflow is owned by the platform team, lives in a central repository, and is version controlled. The caller workflow lives in the project repository and simply calls it, passing in the parameters at runtime.
If you want a copy of the repo with with these workflow examples to use as a starting point, drop your email in the box below and it’ll be sent to you automatically.
Summary and What’s Next
We’ve covered a lot of ground in this post and I hope you’ve found it useful. Feel free to take the concepts above and run with them, but don’t take the easy route. Don’t copy and paste, make a point to understand what each part is doing, build on it, improve it and make it yours. My platform environment will likely be different from yours, and my working needs won’t be identical either. If you follow the principles I’ve outlined though, you can quickly build your own reusable workflows that address the gaps in your current deployment model.
Next up I’ll be covering OIDC and federated identities in detail, focusing specifically on getting the setup right for Azure.
I’d love to hear about your experiences and what workflows are working in your environment. Reach out in the comments below, or drop your email in the box to stay up to date.