Just In Time

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Maybe you remember this warning, the warning was shown to a user when using the sudo command to run tasks as administrator on Unix/Linux. As a GitHub Org admin, you do not explicitly switch with sudo to an admin view, you are admin all the time and to each action, so you should keep the sudo warning in mind. In this blog, we explore how we can become just-in-time admin, like using sudo on Linux.

The problem

Are you an admin for a GitHub Org? Then you likely use the same user that you use for day-to-day work, like developing code, creating pull requests, reviewing pull requests, and more. When you perform a task for changing privileged settings you have to authenticate yourself like when you do a sudo operation.

But how do you then differentiate between the access you have as a normal user because you are a member of a team or collaborator on a repository, and the full access you are granted as administrator? The answer is; there is no easy way to see the difference.

Because your user is always admininistrator, you can always do anything in your Org. You can see any repository, make any change, operate on any pull request, push commits to any repository, and much more. For most of those operations no explict authentication like sudo is required. Only when you change settings with high impact an extra authenticaton is triggered. And normal members in the Org can only act on repositories to which they are explicit granted acces.

Just in Time Admin with IssueOps

A solution for the problem is using two users. One user you use as admin, and the second as a normal member. Besides the costs of an extra license, a mistake can be made still easily. And with the standard SSO including MFA having two users is in many organizations not trivial. Simply because you are logged in with the wrong user. Another way of solving the problem is becoming just-in-time (JIT) admin, similar to sudo.

At the moment of writing GitHub does not provide an option to become just-in-time admin. The GitHub Service team open-sourced an action and JavaScript CLI which automate the promoting and demoting of normal Org members to admin based on the JIT principle.

What is IssueOps? IssueOps is automating the process around GitHub Issues with GitHub Action. With optional approvals before automation kicks in.

We are going to use this action to automate “just-in-time” promotion to Org admins with IssueOps. The action will execute highly privileged operations on our GitHub Org and thus it requires adminstration permissions for your Org. We will be very careful to who we provide the token because malicious actions could misuse the token. When trusting third-party actions it is strongly recommended that you lock the action on the Git SHA. Locking the action to the exact version is just the first step in hardening. Think also about checking the source code, libraries used, and what you run. For many JavaScript actions, you simply don’t know if the generated runtime bundle is matching the sources. The topic of securing your software supply chain is a topic on itself and we not cover in detail in this post.

Fork

To ensure we keep our admin token safe we take a copy of the repository, check the code and ensure we generate the execution bundle ourselves. We fork the admin action repository and make a few modifications. It is up to you if you fork or copy the repository. But make sure you always check the license. In this case, the license is MIT which means no limitations in usages or rules to contribute back.

The readme of the admin-support-issueops-actions repository is a good guide for setting up the automation, including directions on how to extend it. Creating your own extensions is great, but we believe those should be contributed back as open source. Because it is much better to use each other’s power than build the same in our closed environments over and over again.

- name: Update dist
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: |
    gh pr checkout ${{ github.event.pull_request.number }}
    git config --global user.name "action-bot"
    git config --global user.email "action-bot@040code.github.io"
    DIFF=$(git diff-index HEAD dist/)
    if [ "$DIFF" ]
    then
      git commit -m "chore: Update action dist" dist/index.js
      git push
    else
      echo "Distribution is up-to-date."
    fi

name: Release
on:
  push:
    branches:
      - main

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: philips-software/app-token-action@v1.1.2
        id: token
        with:
          app_id: ${{ secrets.APP_ID }}
          app_base64_private_key: ${{ secrets.APP_PRIVATE_KEY_BASE64 }}
          auth_type: installation

      - name: Release
        uses: google-github-actions/release-please-action@v3
        with:
          release-type: simple
          token: ${{ steps.token.outputs.token }}

Promotion and demotion workflow

The admin support action implements a simple promotion/demotion process. Key in the process is that only members have access to the repository for admin request issues. This means the repository has to be ⚠️ private ⚠️. The diagram illustrates the process of demotion and demotion.

A member of the Org that has access to the admin’s repository creates an issue to request admin add access. Next, an action starts and handles the issue to promote the user. The issue is kept open as long the user is busy with his or her admin task. Once the admin task is finished the user closes the issue. The closure of the issue will start the demotion workflow to demote the user back to a normal member. As a fallback, scheduled workflow ensures that after a time-out the user is always demoted back to a standard member. The user can set the time-out in the issue which should be between 1 and 8 hours.

Action setup

Now we have discussed the working of the action it is time to set up the actual workflows. First, we need to create a repository to handle the issues for requesting admin promotion. Remember this repository must be private and you should only grant access to members in your Org who are allowed to become an admin. Create a new repository, for example via the GitHub CLI:

gh repo create my-org/admins --private

We have already discussed the process for promotion and demotion. This process requires three different workflows:

• Promotion workflow that acts on issue creation
• Demotion workflow that acts on issue closure
• Timeout check to ensure a user is demoted after a maximum duration that acts on a schedule (cron).

PAT vs App token

The admin support action suggests using a PAT (personal access token). We prefer to use an App token instead of the PAT. It avoids user accounts that are required to run automation. With the old PAT, the scope of the token was far too wide, with the new PAT, a token can be limited in the same way as an App token. A second issue of the PAT is the rate limit, although that may not be a problem for this use case but could be in general a challenge. Finally, it saves you a license seat. The action can run with an App token but the template workflows require a few modifications.

Create the Admin support App

As discussed we will use an App token for authorization against the GitHub API. Go to your Org developer’s settings and create a GitHub App. Set the following settings for the new App:

• Disable the webhook
• Repository permissions: contents read/write
• Organization permissions: Administration read/write, Members read/write

Save your App, download the SSH key of the App, and make a note of the App id. The last step is to install the App to the created repository to manage admins. This grants the App access to the repository.

Repository configuration

The admin support action requires a configuration file config.yml in the root of the repository. We set up the admin promotion/demotion for a single Org. If needed you can set up the action for multiple Orgs. Add a similar file like below to your admin’s repository.

org: 040code
repository: admins
supportedOrgs:
  - 040code
reportPath: reports

The automation we are going to run requires labels on the repository. The example workflows are using the actions-ecosystem/action-add-labels action. This action is not well maintained, therefore we replace the label created by simple gh CLI commands. Drawback: we have to create the labels first. You can create the labels with this simple script.

for l in "automation-running", "user-promoted", "promotion-error", "user-demoted", "manual-demotion", "automatic-demotion"
do
  gh label create $l
done

Finally, the workflow needs to get a token for the created GitHub App. This requires you to define the following two secrets. First add the id of the App as APP_ID. Next, add the base64 encoded string of the private ssh key as APP_PRIVATE_KEY_BASE64. You can convert the ssh key to a base encoded string by running cat key-file.pem | base64 | pbcopy..

Issue template

We are going to promote and demote users with IssueOps which requires that we can automatically process the issue with Actions. To help the user to create an issue in a predictable format we use an issue template. Note that issue templates are not supported in private repositories for Orgs on the free plan. Create the directory .github/ISSUE_TEMPLATE and add create an issue template yaml file.

---
name: Request administrator permission in the organization
about: Allows the support team to request temporary admin permission in an organization
title: Request administrator permission
labels: ''
assignees: ''
---

Organization: my-org
Description:
Duration: 2
Ticket: 0

As you can see several fields are required. You can use the setup for multiple organizations, for now, we use it for one Org. The Duration can be set to max eight hours but we will set the default to two hours. The tickets are meant to refer to an external ticket system and because we use GitHub issues we can ignore this field.

Promotion workflow

The promotion workflow is triggered once an issue is created. After parsing the issue the workflow promotes the user to admin. This and the following workflows are using third-party action based on version. Be aware that a version is mutable; it is strongly recommended to lock your actions on SHA instead of a tag.

name: Promotion workflow
on:
  issues:
    types: [opened]

jobs:
  promote-workflow:
    name: Promote @${{ github.event.issue.user.login }} to admin
    runs-on: ubuntu-latest
    permissions:
      issues: write
      contents: read
    env:
      GH_TOKEN: ${{ github.token }}

    steps:
      - uses: actions/checkout@v3
      - uses: philips-software/app-token-action@v1.1.2 #1
        id: token
        with:
          app_id: ${{ secrets.APP_ID }}
          app_base64_private_key: ${{ secrets.APP_PRIVATE_KEY_BASE64 }}
          auth_type: installation

      - name: Add label automation-running
        if: always()
        run: gh issue edit --add-label automation-running ${{ github.event.issue.number }}

      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Parse the issue submitted
        id: issue_parser
        uses: 040code/admin-support-issueops-actions/admin-support-cli@v1.0.0
        with:
          action: "parse_issue"
          issue_number: ${{ github.event.issue.number }}
          ticket: ${{ github.event.issue.number }}

      - name: Parse issue parser output
        id: parse_issue_output
        run: |
          target_org=$(echo ${{toJSON(steps.issue_parser.outputs.output)}} | jq -r .target_org)
          echo "target_org=$target_org" >> $GITHUB_OUTPUT

      - name: Grant admin access #2
        id: grant_admin
        uses: 040code/admin-support-issueops-actions/admin-support-cli@v1.0.0
        with:
          action: "promote_demote"
          username: ${{ github.event.issue.user.login }}
          target_org: ${{ steps.parse_issue_output.outputs.target_org }}
          role: "admin"
          admin_token: ${{ steps.token.outputs.token }}

      - name: Add a comment on the issue
        uses: actions/github-script@v6
        if: success()
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }} #2
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `✅   We have executed the request and now the user **@${{github.event.issue.user.login}}** is an admin on ${{steps.parse_issue_output.outputs.target_org}}. When you finish the operations required by the support ticket, close this issue to demote your permissions.

              <sub>
                Find details of the automation <a href="https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${{github.run_id}}">here</a>.
              </sub>
              `
            })

      - name: Add label user-promoted
        if: success()
        run: gh issue edit --add-label user-promoted ${{ github.event.issue.number }}

      - name: Add label promotion-error
        if: failure()
        run: gh issue edit --add-label promotion-error ${{ github.event.issue.number }}

      - name: Close issue if the promotion fails
        uses: actions/github-script@v6
        if: failure()
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            github.rest.issues.update({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              state: 'closed'
            })
            github.rest.issues.lock({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo
            })

      - name: Remove label automation-running
        if: always()
        run: gh issue edit --remove-label automation-running ${{ github.event.issue.number }}

The workflow should be easy to read, but lets discuss the most interesting details. Since we are using an App for executing API calls to promote users, we first need to obtain a token for the App. We use an action that can get a token for an app [#1]. For all API calls that don’t need the admin token, we use the repo-level secret injected by GitHub Actions, aka GITHUB_TOKEN. After parsing the issue and setting labels the user is granted access, here we use the token from the App [#3]. Once the user is promoted, a comment is made on the issue to inform the user.

We can already test our promotion. Create an issue based on the template. You should shortly see that the issue is updated like below. `

Demotion

Now we can promote the user, the next step is to dethrone the user and revoke the admin privileges. The second workflow acts on closing the issue.

name: Demote a user
on:
  issues:
    types: [closed]

jobs:
  demote-workflow:
    name: Demoting a user for closing an issue
    runs-on: ubuntu-latest
    permissions:
      issues: write
      contents: write
    env:
      GH_TOKEN: ${{ github.token }}
      DEMOTION_ERROR_NOTIFY: "@npalm"

    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - uses: philips-software/app-token-action@v1.1.2
        id: token
        with:
          app_id: ${{ secrets.APP_ID }}
          app_base64_private_key: ${{ secrets.APP_PRIVATE_KEY_BASE64 }}
          auth_type: installation

      - name: Add label automation-running
        if: always()
        run: gh issue edit --add-label automation-running ${{ github.event.issue.number }}

      - name: Parse the issue submitted
        id: issue_parser
        uses: 040code/admin-support-issueops-actions/admin-support-cli@v1.0.0
        with:
          action: "parse_issue"
          issue_number: ${{ github.event.issue.number }}
          ticket: ${{ github.event.issue.number }}

      - name: Parse issue_parser json output
        id: parse_issue_output
        run: |
          target_org=$(echo ${{toJSON(steps.issue_parser.outputs.output)}} | jq -r .target_org)
          description=$(echo ${{toJSON(steps.issue_parser.outputs.output)}} | jq -r .description)
          duration=$(echo ${{toJSON(steps.issue_parser.outputs.output)}} | jq -r .duration)
          echo "target_org=$target_org" >> $GITHUB_OUTPUT
          echo "description=$description" >> $GITHUB_OUTPUT
          echo "duration=$duration" >> $GITHUB_OUTPUT

      - name: Demote user
        id: demote_admin
        uses: 040code/admin-support-issueops-actions/admin-support-cli@v1.0.0
        continue-on-error: true
        with:
          action: "promote_demote"
          username: ${{ github.event.issue.user.login }}
          target_org: ${{ steps.parse_issue_output.outputs.target_org }}
          role: "member"
          admin_token: ${{ steps.token.outputs.token }}

      - name: Add a comment on the issue to confirm the demotion
        uses: actions/github-script@v6
        if: success()
        with:
          github-token: ${{secrets.GITHUB_TOKEN}}
          script: |
            await github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `✅   We have executed the request and now the user **@${{github.event.issue.user.login}}** has been demoted from ${{steps.parse_issue_output.outputs.target_org}}. \n\n This issue will be locked to avoid new interactions

              <sub>
                Find details of the automation <a href="https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${{github.run_id}}">here</a>.
              </sub>
              `
            })
            await github.rest.issues.lock({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo
            })

      - name: Add a comment to notify the team that this automation failed
        uses: actions/github-script@v6
        if: failure()
        with:
          github-token: ${{secrets.GITHUB_TOKEN}}
          script: |
            await github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `Demoting the user has failed. ${{env.DEMOTION_ERROR_NOTIFY}} have a look to make sure the user is left in a correct state.

              <sub>
                Find details of the automation <a href="https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${{github.run_id}}">here</a>.
              </sub>
              `
            })

      - name: Add labels user-demoted, manual-demotion
        if: ${{ success() && github.event.sender.login == github.event.issue.user.login }}
        run: |
          gh issue edit --add-label user-demoted ${{ github.event.issue.number }}
          gh issue edit --add-label manual-demotion ${{ github.event.issue.number }}

      - name: Add labels user-demoted, manual-demotion
        if: ${{ success() && github.event.sender.login != github.event.issue.user.login }}
        run: |
          gh issue edit --add-label user-demoted ${{ github.event.issue.number }}
          gh issue edit --add-label automatic-demotion ${{ github.event.issue.number }}

      - name: Remove label user-promoted
        if: success()
        run: gh issue edit --remove-label user-promoted ${{ github.event.issue.number }}

      - name: Remove label automation-running
        if: always()
        run: gh issue edit --remove-label automation-running ${{ github.event.issue.number }}

The demotion workflow is similar to the promotion workflow, but in reverse. After getting a token and parsing the issue, the user is reverted back to normal Org member. Again a comment to the issue is made to inform the user. In case of any failure, a comment is made to trigger a notification to a predefined user. The admin support action also supports retrieving the relevant parts from the audit log, and writing them as an audit record back in the repository.

Now we can demote our user back to normal user by just closing this issue. After closing the issue will be updated like below.

Timeout

The timeout is a safeguard. We run a scheduled workflow every hour to check the open issues. In case the user has reached the maximum duration of being admin, one to eight hours. This workflow will close the issue which triggers the default demotion workflow that runs on the closure of an issue.

name: Provisioning check to see if a user needs to be demoted
on:
  schedule:
    - cron: "0 * * * *"
  workflow_dispatch:

jobs:
  provisioning-check:
    name: Close issues with expired duration
    runs-on: ubuntu-latest
    steps:
      - uses: philips-software/app-token-action@v1.1.2
        id: token
        with:
          app_id: ${{ secrets.APP_ID }}
          app_base64_private_key: ${{ secrets.APP_PRIVATE_KEY_BASE64 }}
          auth_type: installation

      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Run through all the issues and close them if they are expired
        id: issue_parser
        uses: 040code/admin-support-issueops-actions/admin-support-cli@v1.0.0
        with:
          action: "check_auto_demotion"
          ticket: ${{ github.event.issue.number }}
          # Require a non default action token, otherwise it won't trigger a job on issue close
          admin_token: ${{ steps.token.outputs.token }}        

This workflow does not need much explanation. As mention, it checks on hourly based the open issues using the admin-support-issueops-action. When running it once in an hour it could result that the issue being closed almost an hour after the time-out, but nothing holds you back to run the workflow more frequently.

Fireside chat

In this article, we have explained how you can promote members to admin and back to a normal member with IssueOps. We run IssueOps via action, which makes you dependent on GitHub Actions. When GitHub Acton is down your promotion/demotion workflows are broken. An alternative could be hooking into the GitHub events with a webhook. Reduce your dependencies, but you are still dependent now on the GitHub eventing system. When moving to Just In Time access via IssueOps it sounds smart to keep a backup user available, one that normally not is used, but that is available for disaster recovery.

The solution provided by GitHub via the admin support action makes it possible to have just-in-time admin. But it would be much better to have this feature by default in the platform. The setup with a private repo gives you also slightly more control over who can add an admin. By distinguishing user permission levels on this repository.

Now our Org admins are no longer admin anymore by default and are only promoted to admin for doing specific tasks. Should we not ask ourselves, why do we have all the time repository admins? Is this required? Should we not apply a similar mechanism to them as well?

Credits

Thanks for reviewing and suggestions:

• Jeroen Knoops
• Margriet Palm
• Ron Jaegers