Deployment Pipelines — Staging, Production Approval, and Rollback

Complete Deployment Workflow #

# .github/workflows/deploy.yml — Multi-environment deployment pipeline

name: Deploy

on:
    workflow_run:
        workflows: [CI, Docker Build & Push]
        types:     [completed]
        branches:  [main]
    workflow_dispatch:
        inputs:
            environment:
                description: 'Target environment'
                required:    true
                type:        choice
                options:     [staging, production]
                default:     staging
            image-tag:
                description: 'Image tag to deploy (default: latest main SHA)'
                required:    false

env:
    REGISTRY:     ghcr.io
    API_IMAGE:    ghcr.io/${{ github.repository }}/api
    CLIENT_IMAGE: ghcr.io/${{ github.repository }}/client

jobs:

    # ── Resolve which image tag to deploy ────────────────────────────────
    resolve-tag:
        name:    Resolve Image Tag
        runs-on: ubuntu-latest
        outputs:
            tag: ${{ steps.resolve.outputs.tag }}
        steps:
            - name: Resolve tag
              id:   resolve
              run: |
                  if [ -n "${{ github.event.inputs.image-tag }}" ]; then
                      echo "tag=${{ github.event.inputs.image-tag }}" >> $GITHUB_OUTPUT
                  else
                      echo "tag=sha-$(echo ${{ github.sha }} | cut -c1-7)" >> $GITHUB_OUTPUT
                  fi

    # ── Deploy to Staging ─────────────────────────────────────────────────
    deploy-staging:
        name:        Deploy to Staging
        runs-on:     ubuntu-latest
        needs:       resolve-tag
        environment: staging           # uses staging secrets + deployment tracking
        concurrency:
            group: deploy-staging
            cancel-in-progress: false  # don't cancel in-progress deployments!

        steps:
            - uses: actions/checkout@v4

            - name: Deploy to staging server
              uses: appleboy/ssh-action@v1
              with:
                  host:     ${{ secrets.STAGING_HOST }}
                  username: ${{ secrets.STAGING_USER }}
                  key:      ${{ secrets.STAGING_SSH_KEY }}
                  script: |
                      cd /opt/taskmanager
                      export API_IMAGE=${{ env.API_IMAGE }}:${{ needs.resolve-tag.outputs.tag }}
                      export CLIENT_IMAGE=${{ env.CLIENT_IMAGE }}:${{ needs.resolve-tag.outputs.tag }}
                      export GIT_SHA=${{ github.sha }}

                      # Pull new images
                      docker pull $API_IMAGE
                      docker pull $CLIENT_IMAGE

                      # Deploy with rolling update
                      docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d \
                          --no-build \
                          api angular

                      # Verify deployment health
                      sleep 10
                      curl -f http://localhost:3000/api/v1/health || exit 1

                      echo "Staging deployment successful: $GIT_SHA"

            - name: Run smoke tests against staging
              run: |
                  response=$(curl -s -o /dev/null -w "%{http_code}" \
                      https://staging.taskmanager.io/api/v1/health)
                  if [ "$response" != "200" ]; then
                      echo "Smoke test failed: HTTP $response"
                      exit 1
                  fi
                  echo "Smoke test passed"

            - name: Notify staging deployment
              uses: slackapi/slack-github-action@v1
              with:
                  channel-id: ${{ secrets.SLACK_CHANNEL_ID }}
                  slack-message: |
                      ✅ Staging deployed: `${{ needs.resolve-tag.outputs.tag }}`
                      Commit: ${{ github.event.head_commit.message }}
              env:
                  SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

    # ── Deploy to Production (requires manual approval) ───────────────────
    deploy-production:
        name:        Deploy to Production
        runs-on:     ubuntu-latest
        needs:       [resolve-tag, deploy-staging]  # staging must succeed first
        environment: production                      # has required reviewers configured
        concurrency:
            group: deploy-production
            cancel-in-progress: false

        steps:
            - uses: actions/checkout@v4

            - name: Create GitHub deployment
              uses: actions/github-script@v7
              id:   create-deployment
              with:
                  script: |
                      const deployment = await github.rest.repos.createDeployment({
                          owner:       context.repo.owner,
                          repo:        context.repo.repo,
                          ref:         '${{ github.sha }}',
                          environment: 'production',
                          description: 'Deploying ${{ needs.resolve-tag.outputs.tag }}',
                          auto_merge:  false,
                      });
                      core.setOutput('deployment_id', deployment.data.id);

            - name: Deploy to production
              uses: appleboy/ssh-action@v1
              with:
                  host:     ${{ secrets.PROD_HOST }}
                  username: ${{ secrets.PROD_USER }}
                  key:      ${{ secrets.PROD_SSH_KEY }}
                  script: |
                      cd /opt/taskmanager
                      export API_IMAGE=${{ env.API_IMAGE }}:${{ needs.resolve-tag.outputs.tag }}
                      export CLIENT_IMAGE=${{ env.CLIENT_IMAGE }}:${{ needs.resolve-tag.outputs.tag }}

                      # Save current tag for rollback
                      docker inspect $(docker compose ps -q api) \
                          --format '{{.Config.Image}}' > /opt/taskmanager/.last-deployed-tag

                      docker pull $API_IMAGE
                      docker pull $CLIENT_IMAGE

                      docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d \
                          --no-build api angular

                      sleep 15
                      curl -f https://api.taskmanager.io/api/v1/health || exit 1

            - name: Update deployment status — success
              if:   success()
              uses: actions/github-script@v7
              with:
                  script: |
                      github.rest.repos.createDeploymentStatus({
                          owner:         context.repo.owner,
                          repo:          context.repo.repo,
                          deployment_id: ${{ steps.create-deployment.outputs.deployment_id }},
                          state:         'success',
                          environment_url: 'https://app.taskmanager.io',
                      });

            - name: Update deployment status — failure
              if:   failure()
              uses: actions/github-script@v7
              with:
                  script: |
                      github.rest.repos.createDeploymentStatus({
                          owner:         context.repo.owner,
                          repo:          context.repo.repo,
                          deployment_id: ${{ steps.create-deployment.outputs.deployment_id }},
                          state:         'failure',
                      });

How It Works #

Step 1 — Environment Protection Rules Gate Production #

When a job targets environment: production, GitHub checks the environment’s protection rules. If “Required reviewers” is configured (in Settings → Environments → production), the job pauses and GitHub sends review requests to the designated reviewers. The workflow shows a “Review pending” status in the UI. Only after an authorised reviewer clicks “Approve” does the deployment job continue. This human checkpoint is the last line of defence before code reaches users.

Step 2 — Staging Must Pass Before Production is Available #

needs: [resolve-tag, deploy-staging] on the production job means: production deployment only becomes available after staging deployment succeeds. If the staging deployment fails (smoke test fails, SSH error, Docker pull fails), the production job is skipped entirely. This enforces the invariant: code must work in staging before it can reach production.

Step 3 — concurrency: cancel-in-progress: false Protects Deployments #

For deployment jobs, cancel-in-progress: false is critical. If two deployments are triggered in quick succession, you do not want to cancel a deployment mid-flight — the server could be left in a half-deployed state. The default for CI jobs is to cancel on new pushes (saving compute), but for deployments, queue them instead. The second deployment waits for the first to complete before starting.

Step 4 — Saving the Previous Tag Enables Fast Rollback #

Saving the current image tag before deploying the new one provides instant rollback capability: if the new deployment fails or causes errors, run the deployment workflow with the saved previous tag. Storing it in a file on the server (.last-deployed-tag) means the rollback information survives CI pipeline failures. The GitHub Deployments API also tracks this — the deployment history in the GitHub UI shows which SHA is deployed where.

Step 5 — Smoke Tests Catch Deployment Failures Immediately #

A smoke test runs immediately after deployment to verify the application is actually responding correctly — not just that the Docker commands succeeded. A simple curl -f https://api.taskmanager.io/api/v1/health verifies the API container started, connected to MongoDB, and is responding to HTTP requests. A failed smoke test triggers an immediate alert and blocks the production deployment gate, preventing broken deployments from being confirmed as successful.

Common Mistakes #

Mistake 1 — Deploying to production without staging first #

❌ Wrong — code goes directly to production without environment verification:

deploy-production:
    needs: resolve-tag   # no staging dependency!
    environment: production

✅ Correct — staging must succeed before production is available:

deploy-production:
    needs: [resolve-tag, deploy-staging]   # staging required
    environment: production

Mistake 2 — Using cancel-in-progress: true on deployments #

❌ Wrong — mid-flight deployment is cancelled, server in unknown state:

concurrency:
    group: deploy-production
    cancel-in-progress: true   # cancels deployment mid-flight!

✅ Correct — queue deployments, never cancel in flight:

concurrency:
    group: deploy-production
    cancel-in-progress: false   # queue — second waits for first to complete

Mistake 3 — No smoke test after deployment #

❌ Wrong — deployment “succeeds” even if app is broken:

- name: Deploy
  run: docker compose up -d
  # Docker started the container — but is the app actually working?
  # No verification — deployment marked successful even if it crashes

✅ Correct — verify health after deployment:

sleep 10
curl -f https://api.example.com/health || exit 1  # fail job if unhealthy

Quick Reference #