Single-use macOS VMs for iOS CI with Anka, AWS EC2 Mac, and Github Actions or Buildkite

In this blog, we’re going to walk you through how you can automate the creation of single-use macOS VMs for iOS CI with AWS EC2 Mac and Github Actions or Buildkite.

Here at Veertu we have a long history with Buildkite and love YAML configured build, test, and deployment automation. Buildkite has many great features and handles the management of your self-hosted agent pools. They provide an agent to install on your machines which then executes different plugins to prepare your environment for the job commands. For example, I can install and register the Buildkite agent to the mac-anka-large-node-fleet queue and also install the Anka Virtualization package onto my Mac Mini, then define the Anka Buildkite Plugin in my repo’s pipeline.yml file like so:

steps:
  - label: "build"
    agents: "queue=mac-anka-large-node-fleet"
    command: "make test"
    plugins:
      - veertuinc/anka#v0.7.0:
          vm-name: macos-base-10.14

This allows me to prepare an Anka VM using a prepared macos-base-10.14 “Anka VM Template” containing all of the specific dependencies and state I need, on my Mac Mini, and then execute make test inside of it, all isolated from the host.

Let’s first chat about one of our new favorites on the CI/CD playground, Github Actions. Similar to buildkite, Github Actions has a YAML file you define actions (vs the buildkite plugins) in. You can expect a similar step definition to Buildkite using our Anka VM action:

     runs-on: [self-hosted, macOS]
     steps:
      - name: build
        uses: veertuinc/[email protected]
        with:
          anka-vm-template-name: "macos-base-10.14"
          vm-commands: "make test"

The self-hosted Github Action Runner you install onto your Apple hardware will register with your repo or organization as self-hosted and macos and allow Github Actions to schedule jobs on it when necessary.

With both Buildkite and Github managing the registered runners, you don’t need the Anka Build Cloud Controller for orchestration and only need to set up the Cloud Registry to store your templates and tags.

Using the GitHub action or Buildkite plugin to achieve ephemeral on-demand single-use macOS VMs for your building and testing works great, but can still leave a bit of burden around maintaining on-premise macOS hardware. This is why we recommend hosting your macOS hardware with AWS EC2 Mac instances. Amazon’s EC2 Mac instances allow you to avoid purchasing macOS hardware to be set up in your team’s datacenter. You simply request a dedicated mac and then start an instance on it — very similar to EC2 Linux! You’ll also notice the experience for creating or choosing an AMI, EBS volume, security group, and ssh keys are the same. Amazon wrote a blog on creating an AWS EC2 Mac instance and installing Anka inside of it, however, we’re going to take a slightly different approach and utilize our getting started scripts to set things up.

Note: The “getting started scripts” will set up both the Controller and Registry inside of an ec2 instance. As explained above, you won’t need to run the Controller when using Buildkite, Github Actions, or even Azure DevOps Pipelines. The Controller is used for creating and managing instances through a UI or REST API, which each of these tools already does. We’ll only need the Registry to store our VM templates/tags and allow our nodes to pull down new versions when necessary. We’ll use the Controller in this guide to visualize and confirm we’re setting things up properly.

Step one: Obtain your Anka license

We will need to obtain an Anka Build license. You can fill out our trial form and we’ll send you one by email immediately.

Step two: Git clone the getting-started repo

Once we have the license, we will be using git to clone the getting-started repo on our local machine:

❯ git clone https://github.com/veertuinc/getting-started.git
Cloning into 'getting-started'...
remote: Enumerating objects: 943, done.
remote: Counting objects: 100% (433/433), done.
remote: Compressing objects: 100% (305/305), done.
remote: Total 943 (delta 271), reused 281 (delta 128), pack-reused 510
Receiving objects: 100% (943/943), 137.97 MiB | 25.92 MiB/s, done.
Resolving deltas: 100% (595/595), done.

❯ cd getting-started 

❯ ls -alht
total 96
drwxr-xr-x  12 nathanpierce  wheel   384B Sep 22 14:25 .git
-rwxr-xr-x   1 nathanpierce  wheel   7.4K Sep 22 14:25 shared.bash
drwxr-xr-x  17 nathanpierce  wheel   544B Sep 22 14:25 .
-rwxr-xr-x   1 nathanpierce  wheel   1.7K Sep 22 14:25 install-anka-virtualization-on-mac.bash
-rwxr-xr-x   1 nathanpierce  wheel   2.6K Sep 22 14:25 create-vm-template.bash
-rwxr-xr-x   1 nathanpierce  wheel   6.7K Sep 22 14:25 create-vm-template-tags.bash
drwxr-xr-x   5 nathanpierce  wheel   160B Sep 22 14:25 TEAMCITY
-rw-r--r--   1 nathanpierce  wheel    13K Sep 22 14:25 README.md
drwxr-xr-x   5 nathanpierce  wheel   160B Sep 22 14:25 PROMETHEUS
-rw-r--r--   1 nathanpierce  wheel   1.0K Sep 22 14:25 LICENSE
drwxr-xr-x   5 nathanpierce  wheel   160B Sep 22 14:25 JENKINS
drwxr-xr-x   5 nathanpierce  wheel   160B Sep 22 14:25 GITLAB
drwxr-xr-x   5 nathanpierce  wheel   160B Sep 22 14:25 AWS
drwxr-xr-x   6 nathanpierce  wheel   192B Sep 22 14:25 ANKA_BUILD_CLOUD
drwxr-xr-x   3 nathanpierce  wheel    96B Sep 22 14:25 .misc
-rw-r--r--   1 nathanpierce  wheel    73B Sep 22 14:25 .gitignore
drwxrwxrwt  24 root          wheel   768B Sep 22 14:25 ..

❯ ls AWS
prepare-anka-node.bash   prepare-build-cloud.bash

You can see within the AWS folder above there are two scripts:

prepare-build-cloud.bash

• Running this script will create everything necessary inside of AWS to run an Anka Build Cloud. This includes a security group, elastic IP, etc.
• Required before running prepare-anka-node.bash
• If the first argument is --delete, it will only remove the instance and other items needed for the build cloud.

prepare-anka-node.bash

Requires you first run prepare-build-cloud.bash. Otherwise, you need to set several necessary environment variables before execution. These ENVs can be found in the script under the script’s source and the comment # Collect all existing ids and instances.

• Running this script will create everything necessary inside of AWS to run a mac1.metal instance and then install Anka inside. You’ll be prompted for the Anka license to use if the ANKA_LICENSE env variable is not set.
• Relies on scripts from https://github.com/veertuinc/aws-ec2-mac-amis
• If the first argument is --delete, it will disjoin the node from the controller, remove the anka license, and terminate the instance. You need to release the dedicated host manually.

Step three: Prepare Anka Build Cloud in EC2

Note: These scripts require a locally configured AWS account, ssh key setup in the region, and proper permissions added to your user in IAM. You’ll need the ability to create/modify/delete security groups, create/modify/describe/delete instances,create/modify/describe/delete dedicated machines,create/assign/describe/delete elastic IPs, and describe availability zones.

Let’s run the first of the script to prepare our Anka Build Cloud, which is where we will store our macOS VM Templates:

❯ AWS_KEY_PAIR_NAME=nathan-us-east-2 AWS_REGION=us-east-2 ./AWS/prepare-build-cloud.bash         
WARNING: This script is tested with AWS CLI v2.2.9. If your version differs (mostly a concern for older versions), there is no guarantee it will function as expected! 
==============================================
]] Creating and setting up Anka Build Cloud [[
==============================================
] AWS Profile: default
] AWS Region: us-east-2
] AWS Key Pair: nathan-us-east-2
] AWS User: Nathan | XXXXXXX
==============================================
 - Using existing Security Group: xxxxxxxx | anka-build-cloud
 - Added XX.XX.XX.XX to Security Group xxxxxxxx (80, 8087, 22)
++++ aws ec2 allocate-address --domain vpc --tag-specifications 'ResourceType=elastic-ip,Tags=[{Key=Name,Value=Anka Build Cloud},{Key=purpose,Value=anka-build-cloud}]'
 - Created Elastic IP: eipalloc-009cf4f41e1edc0a3 | 18.116.138.X
++++ aws ec2 run-instances --image-id ami-077e31c4939f6a2f3 --instance-type t2.small --security-group-ids xxxxxxx --key-name nathan-us-east-2 --count 1 --block-device-mappings '{"DeviceName": "/dev/xvda","VirtualName": "anka-build-cloud","Ebs": { "VolumeType": "io2", "Iops": 20000, "VolumeSize": 100 }}' --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=Anka Build Cloud Controller and Registry},{Key=purpose,Value=anka-build-cloud}]'
Instance still starting... Waiting to associate the Elastic IP...
Instance still starting... Waiting to associate the Elastic IP...
 - Created Instance: i-098a2fc09a0724b5d
++++ aws ec2 associate-address --allocation-id eipalloc-009cf4f41e1edc0a3 --instance-id i-098a2fc09a0724b5d
 - Associated Elastic IP 18.116.138.X to i-098a2fc09a0724b5d: eipassoc-0d883829e3e12eb64
]] Preparing Instance [[
Instance ssh still starting...
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Existing lock /var/run/yum.pid: another copy is running as pid 3355.
Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum
    Memory : 151 M RSS (368 MB VSZ)
    Started: Wed Sep 22 19:48:33 2021 - 00:06 ago
    State  : Running, pid: 3355
Cleaning repos: amzn2-core amzn2extra-docker
12 metadata files removed
4 sqlite files removed
0 metadata files removed
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Resolving Dependencies
. . .
+++ aws ec2 reboot-instances --instance-ids i-098a2fc09a0724b5d

 - Instance rebooted
Instance still booting...
]] Installing with Docker [[
Cloning into 'getting-started'...
]] Cleaning up the previous Anka Cloud installation

]] Downloading anka-controller-registry-1.18.0-b3bb21bf.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  134M  100  134M    0     0  28.0M      0  0:00:AnkaControllerAcknowledgements.pdf
. . .
]] Modifying the docker-compose.yml
]] Starting the Anka Build Cloud Controller & Registry
Creating network "anka-controller-registry-1180-b3bb21bf_default" with the default driver
Pulling anka-etcd (veertu/anka-build-cloud-etcd:v1.18.0)...
v1.18.0: Pulling from veertu/anka-build-cloud-etcd
Digest: sha256:0510e2aadf56e2c85c5ee4452b5c4bcaae58904984c7b5a1a69ee0a3f8619e2d
Status: Downloaded newer image for veertu/anka-build-cloud-etcd:v1.18.0
Pulling anka-registry (veertu/anka-build-cloud-registry:v1.18.0)...
v1.18.0: Pulling from veertu/anka-build-cloud-registry
Digest: sha256:bd718ed53eb23a6f3695f53671c3cd6281944adf90f3ec46586908f7023e9c0e
Status: Downloaded newer image for veertu/anka-build-cloud-registry:v1.18.0
Pulling anka-controller (veertu/anka-build-cloud-controller:v1.18.0)...
v1.18.0: Pulling from veertu/anka-build-cloud-controller
Digest: sha256:18c524769047c5e24c3734ca82151b471e976b4a7ffff1067cc3d080a20ec7f3
Status: Downloaded newer image for veertu/anka-build-cloud-controller:v1.18.0
Creating anka.registry ... 
Creating anka.etcd     ... 
Creating anka.etcd     ... done
Creating anka.registry ... done
Creating anka.controller ... 
Creating anka.controller ... done
=============================================================================
Controller UI:  http://18.116.138.X:80
Registry:       http://172.31.25.X:8089
Documentation:  https://ankadocs.veertu.com/docs/anka-build-cloud/

If successful, the script outputs the IPs and ports to access the controller and registry. Try loading both and ensure they’re functional.

According to the controller UI, we have no active Anka nodes, so that’ll be our next step. Execute the prepare-anka-node script:

❯ AWS_KEY_PAIR_NAME=nathan-us-east-2 AWS_REGION=us-east-2 ./AWS/prepare-anka-node.bash
WARNING: This script is tested with AWS CLI v2.2.9. If your version differs (mostly a concern for older versions), there is no guarantee it will function as expected! 
========================================
]] Creating and setting up Anka Nodes [[
========================================
] AWS Profile: default
] AWS Region: us-east-2
] AWS Key Pair: nathan-us-east-2
] AWS User: Nathan | XXXXXXXXXXXXXX
========================================
 - Added 24.170.77.X to Security Group XXXXXXXX (22, 5900)
 - Using Dedicated Host: h-05eedacedc535da34
]] Creating Instance
++++ aws ec2 run-instances --image-id ami-03b7daa9a004e5049 --instance-type=mac1.metal --security-group-ids XXXXXXXXX --placement HostId=h-05eedacedc535da34 --key-name nathan-us-east-2 --count 1 --associate-public-ip-address --ebs-optimized --user-data 'export ANKA_CONTROLLER_ADDRESS="http://172.31.25.X"' --block-device-mappings '[{ "DeviceName": "/dev/sda1", "Ebs": { "VolumeSize": 400, "VolumeType": "gp3" }}]' --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=Anka Build Node},{Key=purpose,Value=anka-build-cloud}]'
Instance still starting...
 - Created Instance: i-0c26d1c0884c8ef6f | 18.191.106.X
Instance still starting...
Instance still starting...
Instance still starting...
Instance still starting...
]] Preparing Instance
Input your Anka license (type "skip" to skip this): XXXXXXX
License activated. The fulfillment ID: XXXXXX
+ echo 'Thu Sep 23 21:54:06 GMT 2021 (root): Attempting join...'
Thu Sep 23 21:54:06 GMT 2021 (root): Attempting join...
++ curl -s http://169.254.169.254/latest/user-data
++ grep 404
+ [[ ! 0 -eq 0 ]]
+++ dirname /Users/ec2-user/aws-ec2-mac-amis/cloud-connect.bash
++ cd /Users/ec2-user/aws-ec2-mac-amis
++ pwd
+ SCRIPT_DIR=/Users/ec2-user/aws-ec2-mac-amis
+ cd /Users/ec2-user/aws-ec2-mac-amis
+ echo 'Waiting for networking...'
Waiting for networking...
+ ping -c 1 -n github.com
+ . ./_helpers.bash
++ LAUNCH_LOCATION=/Library/LaunchDaemons/
++ CLOUD_CONNECT_PLIST_PATH=/Library/LaunchDaemons/com.veertu.aws-ec2-mac-amis.cloud-connect.plist
++ RESIZE_DISK_PLIST_PATH=/Library/LaunchDaemons/com.veertu.aws-ec2-mac-amis.resize-disk.plist
++ PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
++ AWS_INSTANCE_USER=ec2-user
++ true
+ [[ ! -e /Library/LaunchDaemons/com.veertu.aws-ec2-mac-amis.cloud-connect.plist ]]
++ date
++ whoami
+ echo 'Thu Sep 23 21:55:20 GMT 2021 (root): Attempting join...'
Thu Sep 23 21:55:20 GMT 2021 (root): Attempting join...
++ curl -s http://169.254.169.254/latest/user-data
++ grep 404
+ [[ ! -z '' ]]
++ curl -s http://169.254.169.254/latest/user-data
++ grep ANKA_
++ sed 's/\"//g'
+ export ANKA_CONTROLLER_ADDRESS=http://172.31.25.X
+ ANKA_CONTROLLER_ADDRESS=http://172.31.25.X
+ [[ ! -z '' ]]
+ ANKA_JOIN_ARGS='--host 18.191.106.X --name node1-us-east-2'
+ curl -O http://172.31.25.X/pkg/AnkaAgent.pkg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 17.2M  100 17.2M    0     0  43.8M      0 --:--:-- --:--:-- --:--:-- 43.7M
+ installer -pkg AnkaAgent.pkg -tgt /
installer: Package name is Anka Agent
installer: Upgrading at base path /
installer: The upgrade was successful.
+ rm -f AnkaAgent.pkg
+ anka license accept-eula
EULA accepted
+ /usr/local/bin/ankacluster join http://172.31.25.X --host 18.191.106.X --name node1-us-east-2
Testing connection to the controller...: Ok
Testing connection to the registry...: Ok
Success!
You can now access your Anka Node with:
   ssh -i "/Users/nathanpierce/.ssh/nathan-us-east-2.pem" "[email protected]"

You will find a getting-started directory under the user's home folder which contains a script to help you generate your first Anka VM Template and Tags.

If successful, this script will request a dedicated host, create an EC2 mac instance on it using our public AMI, then on boot join the instance to our Build Cloud which we can confirm by loading up the Controller UI:

The joining of the node is automatic and only necessary if we were going to use the Controller to manage our Anka VMs. For now, just ignore it.

Step four: Finish instance preparation

Our next step is to finalize the preparation of the Mac EC2 instance so that we can access it and also run the Anka Hypervisor. Apple requires that you have an active UI session and a logged-in user to start the hypervisor, so we’ll need to perform steps 3 – 5 outlined in https://github.com/veertuinc/aws-ec2-mac-amis#prepare-an-ami before we proceed. Once those steps are performed, go over the steps listed in https://docs.veertu.com/anka/anka-build-cloud/getting-started/preparing-and-joining-your-nodes/ and make sure they’re all performed. All of these ensure the best possible performance and usability for the instance.

Step five: Create your Anka VM Template+Tag

We only have one step left before we can start Anka VMs! We’ll need to create our VM Template/Tag for a specific macOS version we want to use. However, we’ll log into the EC2 Mac instance for this to ensure that we’re building a VM Template/Tag fully compatible with the hardware type. The SSH command to log in to the mac was provided at the end of the prepare-anka-node.bash script.

❯ ssh -i "/Users/nathanpierce/.ssh/nathan-us-east-2.pem" "[email protected]"
Last login: Tue Oct  5 17:33:40 2021

    ┌───┬──┐   __|  __|_  )
    │ ╷╭╯╷ │   _|  (     /
    │  └╮  │  ___|\___|___|
    │ ╰─┼╯ │  Amazon EC2
    └───┴──┘  macOS Big Sur 11.6

ec2-user@ip-172-31-25-X ~ % 

Inside you’ll find /Users/ec2-user/getting-started which should contain the same scripts we were running on our local machine. We can use a script inside of it to obtain the macOS .app installer for the specific version we want to create a VM for. It also will automatically create our VM Template and Tag, then push it to the registry for us:

❯ cd /Users/ec2-user/getting-started
❯ ./create-vm-template.bash
]] Downloading Mac Installer .app (requires root) ...

installinstallmacos.py - get macOS installers from the Apple software catalog

This Mac:
Model Identifier : Macmini8,1

Bridge ID        : J174AP
Board ID         : Mac-7BA5B2DFE22DDD8C
OS Version       : 11.6
Build ID         : 20G165

 #  ProductID       Version    Build    Post Date   Title                          
 1  001-15219       10.15.5    19F2200  2020-06-15  macOS Catalina                 
 2  001-68446       10.15.7    19H15    2020-11-11  macOS Catalina                 
 3  071-00696       11.4       20F71    2021-06-02  macOS Big Sur                  
 4  001-36801       10.15.6    19G2021  2020-08-12  macOS Catalina                 
 5  001-04366       10.15.4    19E2269  2020-05-04  macOS Catalina                 
 6  071-97382       11.6       20G165   2021-09-17  macOS Big Sur                  
 7  061-86291       10.15.3    19D2064  2020-03-23  macOS Catalina                 
 8  041-91758       10.13.6    17G66    2019-10-19  macOS High Sierra              
 9  041-88800       10.14.4    18E2034  2019-10-23  macOS Mojave                   
10  061-26589       10.14.6    18G103   2019-10-14  macOS Mojave                   
11  001-51042       10.15.7    19H2     2020-09-24  macOS Catalina                 
12  071-71342       11.5       20G71    2021-07-21  macOS Big Sur                  
13  071-72781       11.5.1     20G80    2021-07-26  macOS Big Sur                  
14  001-57224       10.15.7    19H4     2020-10-27  macOS Catalina                 
15  041-90855       10.13.5    17F66a   2019-10-23  Install macOS High Sierra Beta 
16  061-26578       10.14.5    18F2059  2019-10-14  macOS Mojave                   
17  071-78704       11.5.2     20G95    2021-08-18  macOS Big Sur                  
18  001-36735       10.15.6    19G2006  2020-08-06  macOS Catalina                 

Choose a product to download (1-18): 6
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1611k  100 1611k    0     0  10.4M      0 --:--:-- --:--:-- --:--:-- 10.4M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.5G  100 11.5G    0     0  26.7M      0  0:07:22  0:07:22 --:--:-- 26.2M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1952k  100 1952k    0     0  5759k      0 --:--:-- --:--:-- --:--:-- 5742k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   188  100   188    0     0   2473      0 --:--:-- --:--:-- --:--:--  2473
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5778  100  5778    0     0  79150      0 --:--:-- --:--:-- --:--:-- 79150
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2571k  100 2571k    0     0  7475k      0 --:--:-- --:--:-- --:--:-- 7475k
Making empty sparseimage...
installer: Package name is macOS Big Sur
installer: Installing at base path /private/tmp/dmg.qpHqbC
installer: The install was successful.
*********************************************************
*** Working around a very dumb Apple bug in a package ***
*** postinstall script that fails to correctly target ***
*** the Install macOS.app when installed to a volume  ***
*** other than the current boot volume.               ***
***       Please file feedback with Apple!            ***
*********************************************************
On slow disks this can take a really long time...
Product downloaded and installed to /tmp/anka-mac-resources/Install_macOS_11.6-20G165.sparseimage
]] Mounting Install_macOS_11.6-20G165.sparseimage to /tmp/anka-mac-resources/mount ...
/dev/disk3              GUID_partition_scheme          
/dev/disk3s1            EFI                            
/dev/disk3s2            Apple_HFS                       /private/tmp/anka-mac-resources/mount
"disk3" ejected.
]] Installer placed at /Applications/Install macOS Big Sur.app
]] Creating 11.6.0 using /Applications/Install macOS Big Sur.app ...
Installing macOS 11.6...
Copying AnkaGuestAddons.pkg...
Converting to ANKA format...
100% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]                   
Waiting for installation to complete in the guest (about thirty minutes approx.)...
VM created successfully with uuid: f0872d5c-76b7-4701-9915-a20b58e13476
++
++

local-demo (default)

+--------+---------------+
| host   | 172.31.25.X |
+--------+---------------+
| scheme | http          |
+--------+---------------+
| port   | 8089          |
+--------+---------------+
]] Stopping VM 11.6.0 and pushing with tag: vanilla...
Uploading files  [####################################]  100%             
Upload complete
]] Preparing and pushing VM template 11.6.0 and tag vanilla+port-forward-22
]] Stopping VM 11.6.0 and pushing with tag: vanilla+port-forward-22...
Uploading files  [####################################]  100%
Upload complete
]] Preparing and pushing VM template 11.6.0 and tag vanilla+port-forward-22+brew-git
xcode-select: note: no developer tools were found at '/Applications/Xcode.app', requesting install. Choose an option in the dialog to download the command line developer tools.
==> Checking for `sudo` access (which may request your password).
==> This script will install:
/usr/local/bin/brew
/usr/local/share/doc/homebrew
/usr/local/share/man/man1/brew.1
/usr/local/share/zsh/site-functions/_brew
/usr/local/etc/bash_completion.d/brew
/usr/local/Homebrew
. . .
==> /usr/bin/sudo /usr/bin/touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
==> Installing Command Line Tools for Xcode-13.0
==> /usr/bin/sudo /usr/sbin/softwareupdate -i Command\ Line\ Tools\ for\ Xcode-13.0
Software Update Tool

Finding available software

Downloading Command Line Tools for Xcode
Downloaded Command Line Tools for Xcode
Installing Command Line Tools for Xcode
Done with Command Line Tools for Xcode
Done.
==> /usr/bin/sudo /bin/rm -f /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
==> /usr/bin/sudo /usr/bin/xcode-select --switch /Library/Developer/CommandLineTools
==> Downloading and installing Homebrew...
From https://github.com/Homebrew/brew
 * [new branch]          master     -> origin/master
. . .
]] Stopping VM 11.6.0 and pushing with tag: vanilla+port-forward-22+brew-git...
Uploading files  [####################################]  100%             
Upload complete

ec2-user@ip-172-31-31-X getting-started % sudo anka list              
+-------------------------------------------+--------------------------------------+---------------------+---------+
| name                                      | uuid                                 | creation_date       | status  |
+-------------------------------------------+--------------------------------------+---------------------+---------+
| 11.6.0 (vanilla+port-forward-22+brew-git) | c12ccfa5-8757-411e-9505-128190e9854e | Oct 5 15:10:31 2021 | stopped |
+-------------------------------------------+--------------------------------------+---------------------+---------+

ec2-user@ip-172-31-31-X getting-started % sudo anka show 11.6.0       
+---------+-------------------------------------------+
| uuid    | c12ccfa5-8757-411e-9505-128190e9854e      |
+---------+-------------------------------------------+
| name    | 11.6.0 (vanilla+port-forward-22+brew-git) |
+---------+-------------------------------------------+
| created | Oct 5 15:10:31 2021                       |
+---------+-------------------------------------------+
| vcpu    | 3                                         |
+---------+-------------------------------------------+
| memory  | 8G                                        |
+---------+-------------------------------------------+
| display | 1024x768                                  |
+---------+-------------------------------------------+
| disk    | 100GiB (19.58GiB on disk)                 |
+---------+-------------------------------------------+
| addons  | 2.5.3.135                                 |
+---------+-------------------------------------------+
| network | shared                                    |
+---------+-------------------------------------------+
| status  | stopped Oct 5 16:23:59 2021               |
+---------+-------------------------------------------+

ec2-user@ip-172-31-31-X getting-started % sudo anka show 11.6.0 tags
+----------------------------------+---------------------+---------------------+
| tag                              | creation_date       | last_access         |
+----------------------------------+---------------------+---------------------+
| vanilla+port-forward-22+brew-git | Oct 5 16:24:03 2021 | Oct 5 16:18:06 2021 |
+----------------------------------+---------------------+---------------------+
| vanilla                          | Oct 5 16:12:20 2021 | Oct 5 16:18:06 2021 |
+----------------------------------+---------------------+---------------------+
| vanilla+port-forward-22          | Oct 5 16:18:07 2021 | Oct 5 16:18:06 2021 |
+----------------------------------+---------------------+---------------------+

Take note that there are several tags for this template, all layered on the first tag vanilla. You can find more information about how our templates and tags work as well as the most optimal, efficient, and maintainable approach to creating them in our documentation.

Let’s make sure the registry contains our Template and Tags using curl:

ec2-user@ip-172-31-31-X getting-started % curl -s http://172.31.25.X:8089/registry/status | jq
{
  "status": "OK",
  "body": {
    "status": "Running",
    "version": "1.18.0-04fd94e"
  },
  "message": ""
}

ec2-user@ip-172-31-31-X getting-started % curl -s http://172.31.25.X:8089/registry/v2/vm | jq                          {
  "status": "OK",
  "body": [
    {
      "id": "c12ccfa5-8757-411e-9505-128190e9854e",
      "name": "11.6.0",
      "size": 21024776192
    }
  ],
  "message": ""
}

ec2-user@ip-172-31-31-X getting-started % curl -s "http://172.31.25.X:8089/registry/v2/vm?id=c12ccfa5-8757-411e-9505-128190e9854e" | jq
{
  "status": "OK",
  "body": {
    "id": "c12ccfa5-8757-411e-9505-128190e9854e",
    "name": "11.6.0",
    "versions": [
      {
        "number": 0,
        "tag": "vanilla",
        "config_file": "c12ccfa5-8757-411e-9505-128190e9854e.yaml",
        "nvram": "nvram",
        "images": [
          "6e45c8998fab41d9bbf4a6a6975efda0.ank"
        ],
        "state_files": null,
        "description": "",
        "state_file": "",
        "size": 18645143552
      },
      {
        "number": 1,
        "tag": "vanilla+port-forward-22",
        "config_file": "c12ccfa5-8757-411e-9505-128190e9854e.yaml",
        "nvram": "nvram",
        "images": [
          "6e45c8998fab41d9bbf4a6a6975efda0.ank"
        ],
        "state_files": null,
        "description": "",
        "state_file": "",
        "size": 18645143552
      },
      {
        "number": 2,
        "tag": "vanilla+port-forward-22+brew-git",
        "config_file": "c12ccfa5-8757-411e-9505-128190e9854e.yaml",
        "nvram": "nvram",
        "images": [
          "732787cd099d47e2bcd429f93ba57613.ank",
          "6e45c8998fab41d9bbf4a6a6975efda0.ank"
        ],
        "state_files": null,
        "description": "",
        "state_file": "",
        "size": 21024776192
      }
    ],
    "size": 21024776192
  },
  "message": ""
}

Step six: Install and run the Github Actions Runner

Once confirmed that it’s in the registry, we can now move on to setting up Github Actions runner on our node. To do this, we’ll need to go into our GitHub repository’s actions/runners (https://github.com/{org}/{repoName}/settings/actions/runners) page and click on the New self-hosted runner button. Follow the instructions on the page to install and register the runner. We can finally start the runner and confirm it’s showing up in our repo’s runner list with ./run.sh.

Note: Starting the runner as a service that will automatically start on system boot has a problem under Big Sur. See https://github.com/actions/runner/issues/1056#issuecomment-893920790 for information about the problem.

Step seven: Prepare your YAML and run the job

At this point, we’re ready to set up our first Github Actions YAML which will execute commands we want within the Anka VM on the node we just configured. We’re going to use the existing public simple.yml from the github-actions-examples repo. I’m going to kick it off now that I’ve attached the node/runner to the repo.

ec2-user@ip-172-31-31-X actions-runner % ./run.sh 

√ Connected to GitHub

2021-10-05 18:50:06Z: Listening for Jobs
2021-10-05 18:50:24Z: Running job: ephemeral
2021-10-05 18:50:30Z: Job ephemeral completed with result: Failed

Uh oh, why did we get a failure? The logs for the job are showing STDOUT: -anka: No registry configured which is surprising because we know the registry was working when we created our template and pushed it. Fortunately, this is an easy fix! When we executed the create-anka-template.bash script from getting-started, we created the template under the root user space. Anka will be available for all users on the system by default but will have a unique environment for each. Let’s take a look at the anka list command as both root (sudo) and the ec2-user:

ec2-user@ip-172-31-31-X actions-runner % sudo anka list
+-------------------------------------------+--------------------------------------+---------------------+---------+
| name                                      | uuid                                 | creation_date       | status  |
+-------------------------------------------+--------------------------------------+---------------------+---------+
| 11.6.0 (vanilla+port-forward-22+brew-git) | c12ccfa5-8757-411e-9505-128190e9854e | Oct 5 15:10:31 2021 | stopped |
+-------------------------------------------+--------------------------------------+---------------------+---------+

ec2-user@ip-172-31-31-X actions-runner % anka list     
ec2-user@ip-172-31-31-X actions-runner % 

So the root user has the template, but the ec2-user does not. When I run the Github Actions Runner run.sh as the ec2-user, it will be executing all commands as the ec2-user and not see anything. However, even if we wanted to pull the template from our registry, we can’t. The getting started script we ran has only configured the registry under root. Let’s add it manually for ec2-user:

ec2-user@ip-172-31-31-X actions-runner % sudo anka registry list-repos
++
++

local-demo (default)

+--------+---------------+
| host   | 172.31.25.X   |
+--------+---------------+
| scheme | http          |
+--------+---------------+
| port   | 8089          |
+--------+---------------+
ec2-user@ip-172-31-31-X actions-runner % anka registry list-repos 
++
++
ec2-user@ip-172-31-31-X actions-runner % anka registry add aws http://172.31.25.X:8089
ec2-user@ip-172-31-31-X actions-runner % anka registry list-repos                  
++
++

aws (default)

+--------+---------------+
| host   | 172.31.25.X   |
+--------+---------------+
| scheme | http          |
+--------+---------------+
| port   | 8089          |
+--------+---------------+

We can now re-run our Github job as the anka-vm-github-action will pull the template+tag from the “(default)” repo we’ve set in the Anka CLI if it does not exist before executing commands inside. Be sure to delete the 11.6.0 Template from the root userspace to free up space and execute the ./run.sh again from the terminal.

It may take a while to download the template into the ec2-user space from the registry. You can ensure the pull is happening if necessary:

ec2-user@ip-172-31-31-X ~ % ps uax | grep "[p]ull"
ec2-user         42507  56.7  0.1  4821572  37592 s000  U+    7:05PM   2:31.63 anka registry pull 11.6.0

Once the pull finishes, the logs inside of the actions job will show what commands are running on the host to prepare the VM and then the STDOUT from our execution inside of the VM.

We should then see the status change on our terminal, confirming success:

ec2-user@ip-172-31-31-X actions-runner % ./run.sh 

√ Connected to GitHub

2021-10-05 19:05:13Z: Listening for Jobs
2021-10-05 19:05:17Z: Running job: ephemeral
2021-10-05 19:13:34Z: Job ephemeral completed with result: Succeeded

That’s it! You now have a working Github Actions self-hosted runner that can use ephemeral and templated Anka VMs for your building and testing. If you’re interested in using Buildkite, the steps are the same except for installing the buildkite agent and setting up your YAML to fit the requirements for their platform. You can read our documentation for Buildkite if necessary. We hope you now have a better understanding of how you can automate the creation of single-use macOS VMs for iOS CI with AWS EC2 Mac and Github Actions or Buildkite.