Are you still using keys?

aws cicd cloud github github actions terraform zero trust

This second post explains more advanced practices of deploying keyless with GitHub Actions to AWS. In the first post explains the basics of deploying to AWS with GitHub Actions using OIDC. In this second post you learn how to debug the OIDC claim, and how to allow shared workflows to deploy resources to AWS.

Source code for this post

The problem

In the first post we have seen that deploying with GitHub Action and OIDC to Amazon is straightforward. Defining a trust condition for the repository allows you to safely deploy without using keys. What if we want to do some more generic deployments. For example we want to deploy files from several repositories to a single S3 bucket, where each repository has its own path to deploy files. A use-case here is deploying Backstage techdocs to S3. Another use-case is publishing docker images from several repositories to Amazon ECR.

You can replicate the setup for each repository by duplicating the workflow and updating the IAM policy. This pattern is hard to maintain, and in most cases creates a central bottleneck. Each team having a repository is getting blocked by a central team that needs to update IAM permission. This last part could be automated, but let’s explore what our other alternatives are.

With GitHub Action you have the option to create a shared workflow in your organisation, this shared workflow can take care of those common deployments. Using shared workflows removes duplication of your workflows, but not the complexity of maintaining IAM policies to allow access via OIDC to AWS. Let’s see how to simplify this.

The solutions

The problem that we need to solve is: how can we allow the caller of the workflow to assume a role in AWS. And can we check what the caller is allowed to with a dynamic policy based on the repository name? Before exploring the options, we have a closer look at the information in the claim. You can check the GitHub OIDC configuration here. Another option to understand the claim; is debugging the OIDC claim running in an Action. Add the following workflow to debug the OIDC claim to your repository.

name: Debug OIDC
on:
  workflow_dispatch:
   
jobs:
  oidc_debug:
    permissions:
      id-token: write
    runs-on: ubuntu-latest
    name: A test of the oidc debugger
    steps:
      - name: Debug OIDC Claims
        uses: github/actions-oidc-debugger@main
        with:
          audience: 'https://github.com/github'

This workflow uses the GitHub Action to debug OIDC. Run the workflow and check the log for the details in the claim.

{
  "actor": "npalm",
  "aud": "https://github.com/040code",
  "base_ref": "",
  "event_name": "workflow_dispatch",
  "job_workflow_ref": "040code/blog-oidc-github-actions-aws/.github/workflows/debug-oidc.yml@refs/heads/main",
  "ref": "refs/heads/main",
  "ref_type": "branch",
  "repository": "040code/blog-oidc-github-actions-aws",
  "repository_owner": "040code",
  "repository_visibility": "public",
  "sha": "369522825551840e804852c0c529d819479c910e",
  "sub": "repo:040code/blog-oidc-github-actions-aws:ref:refs/heads/main",
  "workflow": "Debug OIDC",
  ...
}

This looks promising, the claim contains an attribute repository. It would be nice if we can use this in AWS to dynamically check if the workflow caller has access to a location in the bucket that matches the repository name. Or publish an ECR container matching the repository name to the registry name. But this is not possible because AWS does not support those custom fields, but instead only allows a limited set of keys. Circumventing this by writing dynamic policies is also not allowed. So unfortunately we cannot use the OIDC field repository to check if a role is allowed to invoke the operation.

Amazon does support setting principal details via session tags. A claim would look like this:

{
  "sub": "repo:040code/blog-oidc-github-actions-aws:ref:refs/heads/main",
  "workflow": "Debug OIDC"
    "https://aws.amazon.com/tags": {
        "principal_tags": {
            "Repository": ["040code/blog-oidc-github-actions-aws"],
            "Actor": ["npalm"],
            "Workflow": ["Debug OIDC"]
            ...
        }
    }
}

Now we can check the principal in IAM dynamic with the dynamic field ${aws:PrincipalTag/Repository} and match it against a path in the bucket or registry name in ECR. But this requires that GitHub sets this information in the JWT token, which is currently not the case. This means that also this second option is a no go. Hopefully support will be added soon, follow this issue to track the status.

Only one option is left at the moment. GitHub allows you to change the default subject (sub) and set any of the available custom fields, or combinations. We can set for example the workflow reference in the subject field. By adding the workflow reference to the subject, we can define a trust relation for the shared workflow. And allow the shared workflow to run the cloud deploy. Checking or enforcing the caller in accessing the correct path in the bucket has to be done in the workflow. Doing the check in the IAM policy is of course preferably. The other problem is that a subject claim needs to be altered for any repository calling the shared workflow, which increases the complexity. The subject claim can be set on repo or org level, doing this on org level can cause unwanted and unexpected side-effects in case the repository uses OIDC for other connections.

Altering the subject claim

Time to work out an example. We create a shared workflow that syncs files to a location in a S3 bucket based on the repository name. The first step is to update the subject field in the OIDC claim with the workflow reference. This is required to define a trust based on our shared workflow. The only way to change the fields is via the REST API. The GitHub cli seems not to support this PUT operation, we have to fall back to invoke the REST API directly. Below an example of the REST calls, set the GITHUB_REPOSITORY to org/repo and the GITHUB_REPOSITORY to a GitHub access token.

Retrieve the current configuration:

curl -X GET \
  -H 'Accept: application/vnd.github+json' \
  -H 'Authorization: Bearer $GITHUB_TOKEN' \
  https://api.github.com/repos/$GITHUB_REPOSITORY/actions/oidc/customization/sub

Customise the claim to add the workflow reference.

curl -X PUT \
  -H 'Accept: application/vnd.github+json' \
  -H 'Authorization: Bearer $GITHUB_TOKEN' \
  https://api.github.com/repos/$GITHUB_REPOSITORY/actions/oidc/customization/sub -d \
  '{"use_default":false,"include_claim_keys":["repo","context","job_workflow_ref"]}'

Retrieve the new configuration by running the GET command again, you should get a response like.

{
  "use_default": false,
  "include_claim_keys": [
    "repo",
    "context",
    "job_workflow_ref"
  ]
}

The setup above can also be provided via a shareable workflow, see here an example. The workflow can be called from a caller repository to customise the OIDC claim.

For setting up the OIDC provider on the cloud, we use almost the same Terraform configuration as in part 1. In case you have used the setup in part 1, ensure you remove the setup by running terraform destroy. Next you run the setup for part 2.

We add an extra condition to the AWS IAM policy for assuming the role. This condition is allowing the shared workflow to assume the deployment role. Any repository that can call the shared workflow, can deploy to AWS using this workflow. But since we lock down the workflow the caller cannot assume the role directly. Ensure the inputs you inject are safe, and cannot cause unwanted access to your cloud account.

For the repository we only check it matches one of the repositories in our Org. Update the conditions to fit your needs.

condition {
  test     = "StringLike"
  variable = "token.actions.githubusercontent.com:sub"
  values   = ["repo:my-org/*:job_workflow_ref:my-org/my-repo/.github/workflows/s3-template.yml@refs/head/main"]
}

When replaying the blog example, you can simply run the Terraform code provided here and set the variables for the repository and the workflow.

To deploy files per repository to a central bucket we create a shared workflow. The shared workflow is running in a separate job, so we cannot use any build output directly. Therefore we store our generated file in a temporary artifact. Remember for this blog the workflow is not hardened, no extra checks are done!

name: Template deploy S3 (part 2)
on:
  workflow_call:
    inputs:
      artifact:
        required: true
        type: string
        default: dist
  
jobs:
  s3:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - name: Download math result for job 1
        uses: actions/download-artifact@v3
        with:
          name: ${{ inputs.artifact }}
          path: tmp

      - name: configure aws credentials
        uses: aws-actions/configure-aws-credentials@v1-node16
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions/blog
          role-session-name: gh-actions
          aws-region: eu-west-1

      - name: deploy
        run: |
          aws s3 sync --no-progress --delete tmp s3://${{ github.repository_owner }}-${{ github.event.repository.name }}/${{ github.repository }}/

The final step is creating a workflow that calls the shared workflow to deploy. The workflow below is generating some data, stores it as an artifact, and calls the shared workflow. The shared workflow can then deploy the generated artifact to S3. The shared workflow is allowed to assume the role because you have updated the claim on the calling repository before.

name: Blog example S3 (part 2)
on:
  workflow_dispatch:
  push:

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v3
        with:
          node-version: 16

      - name: Generate dist
        run: |
          mkdir dist
          npx cowsay -f ghostbusters "Running ${{ github.workflow }}" > message.txt
     
      - name: Upload dist
        uses: actions/upload-artifact@v3
        with:
          name: dist
          retention-days: 1
          path: |
            message.txt

  deploy:
    needs: generate
    permissions:
      id-token: write
    uses: ./.github/workflows/s3-template.yml
    secrets: inherit
    with:
      artifact: dist

That was all, next run you workflow by pushing or manually triggering.

deploy

Final thoughts

We have explored how we can run deployments with GitHub Actions to AWS using shared workflows. There is currently not a good way to check the caller dynamic to match permissions in AWS. With the customised OIDC claim we can deploy using shared workflows, but it still requires the caller to alter the subject claim. And we need to run checks in the shared workflow instead of using IAM policies when we want to avoid a maintenance burden on our IAM policies. It would be great when AWS and GitHub could work out a way to support session tags to check the principal.

Are you still using keys?

Deploy with GitHub Action using shared workflows to AWS

The problem

The solutions

Altering the subject claim

Final thoughts