The Anatomy of a Good Azure Pipeline
Table of Contents
- Introduction
-
The Core Components of a Pipeline YAML
-
Triggers
- Push Trigger
- Pull Request (PR) Trigger
- Chaining Pipelines
- Variables: Your Configuration Hub
-
The Structure: Stages, Jobs, and Steps
- Stages
- Jobs
- Steps
-
Triggers
- Putting It All Together
Introduction
You’ve probably heard about CI/CD and how it’s supposed to make your life easier. At the heart of this automation in the Azure world is the Azure Pipeline. But what exactly makes a pipeline good? It’s not just about making it work; it’s about making it clean, understandable, and maintainable.
At its core, a pipeline is just a YAML file that describes your entire CI/CD process.
-
A pipeline is one or more stages.
-
Stages are the major sections, these are things like “Build the app,” “Run tests,” and “Deploy to Pre-Prod.”
-
A Job is a linear series of steps that run on an agent (a server). For example, within a “Build” stage, you might have separate jobs like “Build Backend” and “Build Frontend” that can run in parallel to speed things up.
-
Steps are the individual commands or tasks within a job. They can be simple scripts, built-in tasks (like building a .NET project), or even references to other templates.
The Core Components of a Pipeline YAML
Your azure-pipelines.yml file has a few key top-level sections that define how it behaves.
| Component | Description |
|---|---|
trigger |
Specifies which branches will trigger the pipeline when code is pushed. |
pr |
Defines which target branches will trigger the pipeline on a pull request. |
resources.pipelines.pipeline |
Enables one pipeline to trigger another or consume its artifacts. |
variables |
Defines variables you can use throughout the pipeline. |
stages |
A collection of related jobs. The main way to structure your pipeline. |
jobs |
A simpler structure with one implicit stage. Good for basic pipelines. |
steps |
The simplest structure with one implicit job. Great for quick tasks. |
Triggers:
How does a pipeline start? Usually, it’s a push to a repo or the creation of a pull request.
Push Trigger
The trigger keyword controls runs based on pushes. You can get pretty specific here.
-
batch: true: If you push multiple commits quickly, this waits for the current run to finish and then starts one new run with all the changes, instead of queuing a run for every single commit. Super handy. -
branches: Lets youincludeorexcludebranches. -
paths: Want to only trigger a build if the front-end code changes? paths is your friend. It lets you include or exclude file paths.
# This pipeline runs for pushes to master and release/* branches,
# but only when changes affect the 'apps/frontend' directory.
# Changes outside this directory (e.g. other 'apps' or 'local-setup') do not trigger the pipeline.
trigger:
batch: true
branches:
include:
- master
- release/*
exclude:
- features/wip-*
paths:
include:
- apps/frontend/*
exclude:
- README.md
Pull Request (PR) Trigger
The pr trigger is for validating changes before they get merged.
-
autoCancel: true: When a new commit is pushed to the PR’s source branch, it automatically cancels the pipeline run that was in progress for the previous commit. Keeps your agent queue clean. -
drafts: false: By default, pipelines run even for draft PRs. Set this to false to save build minutes until the PR is ready for review.
# This PR trigger runs for PRs targeting master and release/* branches.
# It won't run for draft PRs.
pr:
autoCancel: true
drafts: false
branches:
include:
- master
- release/*
paths:
include:
- apps/frontend/*
Chaining Pipelines
Sometimes, one pipeline depends on another. An example is a “Deploy” pipeline that needs the artifacts from a “Build” pipeline. resources makes this easy.
# In your deploy-pipeline.yml
# Disable CI/PR triggers for this pipeline, we only want it to be triggered by another pipeline.
trigger: none
pr: none
resources:
pipelines:
# Give the resource a friendly ID, like 'build'.
- pipeline: build
# The name of the source pipeline in Azure DevOps.
source: my-app-build-pipeline
# trigger this deploy pipeline when the source pipeline completes.
trigger:
branches:
include:
- master
- release/*
Variables: Your Configuration Hub
Variables let you store and reuse strings and secrets. You can define them at different levels (pipeline, stage, or job) or pull them from Variable Groups created in the Azure DevOps UI. Using Variable Groups is the best practice for secrets like connection strings and API keys!
variables:
# A simple key-value pair
- name: buildConfiguration
value: 'Release'
# Reference a variable group from the UI
- group: 'My-App-Secrets' # Contains things like database passwords
The Structure: Stages, Jobs, and Steps
A multi-stage pipeline is the gold standard for any serious project.
Stages
Stages are the top-level divisions of work. A common pattern is Build -> QA Deploy -> Prod Deploy.
-
displayName: A friendly name that shows up in the UI. -
dependsOn: Makes a stage wait for another one to complete. -
condition: This lets you control if a stage runs. For example, you might only want your “Deploy” stage to run if the build stage succeeded and the change was on the master branch. -
pool: Specifies which agent pool (and VM image) to use for the jobs in this stage.
Here’s an example of a condition for a deploy stage:
stages:
- stage: Build
displayName: 'Build the Application'
jobs:
# ... build jobs go here ...
- stage: Deploy
displayName: 'Deploy to Dev'
dependsOn: Build
# This stage only runs if:
# 1. The Build stage succeeded.
# 2. The pipeline was NOT triggered by a pull request.
# 3. The pipeline was either run manually OR it was triggered by a push to the master branch.
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'), or(eq(variables['Build.Reason'], 'Manual'), eq(variables['Build.SourceBranchName'], 'master')))
jobs:
# ... deploy jobs go here ...
Jobs
Inside a stage, you have one or more jobs. Jobs within a stage can run in parallel by default, which can seriously speed things up! For instance, you could build your frontend and backend code at the same time in separate jobs.
Steps
Steps are the smallest unit of work. They run sequentially inside a job. This is where you call tasks, run scripts, and get things done.
-
task: A pre-built task from the marketplace (e.g., DotNetCoreCLI@2, NodeTool@0). -
script: A simple command-line script. -
checkout: Controls how source code is downloaded.
Putting It All Together
Let’s imagine we’re deploying a full-stack application with a .NET backend, a SQL database (using a DACPAC project), and a React frontend.
Here’s what a good, clean pipeline for that might look like.
# azure-pipelines.yml
trigger:
- master
pr:
- master
variables:
- group: 'MyWebApp-Dev-Secrets' # Store secrets in a Variable Group in the Azure DevOps UI
- name: buildConfiguration
value: 'Release'
stages:
# ========================================================================================
# BUILD STAGE: Compile all parts of the app and publish artifacts
# ========================================================================================
- stage: Build
displayName: 'Build & Package'
jobs:
- job: BuildBackend
displayName: 'Build .NET Backend'
pool:
vmImage: 'windows-latest'
steps:
- task: DotNetCoreCLI@2
displayName: 'Restore, Build, Publish'
inputs:
command: 'publish'
publishWebProjects: true
arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)/backend'
zipAfterPublish: true
- task: PublishBuildArtifacts@1
displayName: 'Publish Backend Artifact'
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/backend'
ArtifactName: 'backend'
- job: BuildFrontend
displayName: 'Build React Frontend'
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool@0
inputs:
versionSpec: '18.x'
displayName: 'Install Node.js'
- script: |
cd src/MyWebApp.Frontend
npm install
npm run build
displayName: 'npm install and build'
- task: PublishBuildArtifacts@1
displayName: 'Publish Frontend Artifact'
inputs:
PathtoPublish: 'src/MyWebApp.Frontend/build'
ArtifactName: 'frontend'
- job: BuildDatabase
displayName: 'Build Database Project'
pool:
vmImage: 'windows-latest'
steps:
- task: VSBuild@1
displayName: 'Build DACPAC'
inputs:
solution: 'src/MyWebApp.Database/MyWebApp.Database.sqlproj'
msbuildArgs: '/p:SqlPublishProfilePath=MyWebApp.Database.publish.xml /p:Configuration=$(buildConfiguration) /p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:OutDir=$(Build.ArtifactStagingDirectory)/db'
- task: PublishBuildArtifacts@1
displayName: 'Publish Database Artifact'
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/db'
ArtifactName: 'database'
# ========================================================================================
# DEPLOY STAGE: Deploy all the artifacts to Azure
# ========================================================================================
- stage: Deploy
displayName: 'Deploy to Production'
dependsOn: Build
# Only deploy on successful builds to master, not on PRs
condition: and(succeeded(), eq(variables['Build.SourceBranchName'], 'master'))
jobs:
# A deployment job gives you better tracking and control for deployments
- deployment: DeployToProduction
displayName: 'Deploy to Azure'
environment: 'Production'
pool:
vmImage: 'windows-latest'
strategy:
runOnce:
deploy:
steps:
# Download all the artifacts from the Build stage
- task: DownloadBuildArtifacts@1
displayName: 'Download all artifacts'
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'backend'
downloadPath: '$(System.ArtifactsDirectory)'
- task: DownloadBuildArtifacts@1
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'frontend'
downloadPath: '$(System.ArtifactsDirectory)'
- task: DownloadBuildArtifacts@1
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'database'
downloadPath: '$(System.ArtifactsDirectory)'
# Deploy the database
- task: SqlAzureDacpacDeployment@1
displayName: 'Deploy Database DACPAC'
inputs:
azureSubscription: 'My-Service-Connection'
AuthenticationType: 'server'
ServerName: '$(dbServerName)' # From Variable Group
DatabaseName: 'MyWebAppDB'
SqlUsername: '$(dbUser)' # From Variable Group
SqlPassword: '$(dbPassword)' # From Variable Group
DacpacFile: '$(System.ArtifactsDirectory)/database/MyWebApp.Database.dacpac'
# Deploy the .NET backend to an App Service
- task: AzureWebApp@1
displayName: 'Deploy Backend to App Service'
inputs:
azureSubscription: 'My-Service-Connection'
appType: 'webApp'
appName: 'my-webapp-api-prod'
package: '$(System.ArtifactsDirectory)/backend/**/*.zip'
# Deploy the React frontend to a Static Website in a Storage Account
- task: AzureCLI@2
displayName: 'Deploy Frontend to Blob Storage'
inputs:
azureSubscription: 'My-Service-Connection'
scriptType: 'bash'
workingDirectory: '$(System.ArtifactsDirectory)/frontend'
scriptLocation: 'inlineScript'
inlineScript: |
az storage blob upload-batch
--account-name mywebappstaticstorage
--source '$(System.ArtifactsDirectory)/frontend'
--destination '$web'
--overwrite
