How to create your own real device cloud for iOS CI with Anka Build

Anka Build enables you to configure your private macOS cloud. You can create and run macOS VMs on-demand for iOS CI and manage it as infrastructure-as-a-code. Additionally, with Anka Build Enterprise Tier, you can also configure a real device cloud to augment your simulator based CI testing.

Anka Build enables you to work with real devices inside Anka macOS VMs through the USB interface. After connecting and claiming the real device on the mac host/node machines, dynamically attach the device to provisioned Anka VMs through the Anka Build Cloud Controller REST API interface. Check out detailed steps in our documentation.Register for a trial at www.veertu.com.

Privileged user access macOS environments – locally delivered and centrally managed

A large number of organizations every year suffer breach into their internal data or their customer’s data due to employee access. While employee access to sensitive data can’t be entirely blocked out, it’s possible to manage it as privileged access. There are a lot of solutions available to provide employees secondary windows privileged access work environments. However, there are not any solutions for macOS.

Almost all of the existing solution takes the approach of completely locking down users’ primary macOS work environment to the extent that impacts their productivity. As an example, Macbooks of iOS developers in financial enterprises are generally entirely locked down and affects their ability to download any early/beta versions of third party software for R&D purposes. As more and more employees migrate from Windows-based systems to Macs, there exists a gap in delivering secondary privileged access macOS work environments for these users. Anka Secure fills this gap.

Anka Secure is built on top of Anka macOS native virtualization engine and uses it to create and run lightweight macOS VMs on user’s mac workstations. It exposes services to prepare privileged users/groups, manages their access with VM policies, and delivers the privileged access secondary macOS work environment(VMs) on end-users mac workstations. The privileged access macOS VM based work environments run locally on the user’s primary mac workstations. This approach provides a local secure, sandbox, privileged macOS environments with best user experience and performance.

Who can benefit from Anka Secure :
  1. Technical and customer support who have access to privileged data from their mac workstations.
  2. Developers in enterprises with access to corporate developer/application certificates and other privileged data from their MacBooks.
  3. iOS developers working in secure enterprise setup and need isolated secondary mac environments for research purposes on their MacBooks.
  4. Environments for Malware and other sensitive analysis and testing tasks for macOS.
Discover more and try it at https://veertu.com/anka-secure/.

Anka version 2.0

Anka version major release is now out and available for download. We are very excited to announce the availability of Enterprise and Enterprise Plus tiers of Anka, along with several other exciting improvements and features.Here is a shortlist of some of the new features and improvements. For a complete list, visit the release notes and product documentation.
  • Up to fifteen percent improvement in performance inside Anka VMs – Significantly closes the performance gap between Anka VMs running on 2018 mac mini and running on native hardware.
  • Support for Catalina Beta 3 VMs – Create and test with Catalina beta macOS VMs.
  • Native Mac App package to for controller/registry – Install and run Controller and Registry on Mac.
  • Unified single Anka package – Deploy the same package for Build or Flow and activate features with the license key.
  • New Queue management module in the Controller – Provides more flexibility for future improvements and scales well.
  • Support for single user Controller Portal authentication – Admins user access to the Controller Portal.
  • Support for multi-user access control and management through OAUTH based SSO integration – Multi-user access to the Portal.
  • Secure, policy managed, and encrypted VMs – Configure Security policies and encryption on the Build VMs.
  • Management of testing on real devices attached to the VMs through the cOntroller REST APIs – Enables creation of a local device cloud.
  • Certificate-based authentication between Plugins, Controller, Registry, and Nodes – Enables a secure setup.
To make it extremely easy for users to deploy Anka Build, we now have a Mac Application package to install Controller and registry services on Mac hardware.If you are new to Anka, visit the products page and register for the Anka Build trial. 

T2 in new Mac hardware and its impact on virtualizing and running macOS – Role of SMC

In the last few weeks, we have received a lot of inbound questions on the existence of T2 chip in new Apple Mac hardware models and how it impacts the ability to run macOS VMs. In this blog series, we will try to share our knowledge and insights on this topic.

The focus of this first blog is on the role SMC plays in booting macOS and how T2 chip impacts this function. Apple T2 chip can control many aspects of the macOS platform over a single unified bus. One of them is, performing additional firmware validation in a trusted execution environment before supplying it to the chipset for execution. During macOS booting, macOS is accessing the hardware SMC chip to read key validation of the Mac.

In T2 enabled Mac hardware, T2 chip is acting as gatekeeper to SMC. Hackintosh and KVM  based projects, which use Clover, are using the saved key of the SMC key, and bypassing the SMC validation. And, tools like ESXi access to SMC in this hardware is blocked by T2, unable to boot macOS VMs (https://twitter.com/lamw/status/1120368830427959297).

However, any solution that leverages macOS native hypervisor.framework like Anka, can boot macOS VMs without any issues. In this scenario, SMC calls are placed through the hypervisor.framework APIs and T2 is able to pass those onwards.



T2 chip in new Mac hardware not only acts as a gatekeeper during the boot process, but also prevents unauthorized access to internal SSD and Thunderbolt ports. So, even if any tool somehow manages to boot macOS VMs(using saved SMCkeys), it will not have access to the SSD. The workaround would be to use USB or network attached storage which is slow, not scalable and unreliable. More on this in our next blog.

Let us know if you have additional questions/comments in our slack channel.

References – https://www.apple.com/mac/docs/Apple_T2_Security_Chip_Overview.pdf

Cloud-based continuous integration for macOS with Concourse and Anka

Pix4D creates professional photogrammetry software for drone mapping. The people who use our products include surveyors, farmers, and researchers, and they use the software on a range of platforms including desktop, mobile, and cloud. Pix4D’s continuous integration (CI) team is based in the headquarters in Lausanne, Switzerland, where we design and maintain the systems used by our developers to automatically build, test, and package our suite of products.

Our CI system is designed around the paradigm of infrastructure as code. We manage the system’s resources using text-based, version-controlled configuration files and automation tools such as Terraform and Salt. All resources are located in a virtual private cloud (VPC) and hosted by Amazon Web Services (AWS). We use Concourse CI as our CI software because it is fully configurable by YAML files and scripts that may be shared, reviewed, and tracked in version control as an application code base would be managed. In addition, each task that is run on a Linux platform is executed inside a Docker container which provides the isolation that is necessary for reproducible builds.

Recently our developers began building some of our software products for macOS. This meant that we had to find a way to incorporate macOS builds into our CI environment with Concourse.

Integrating macOS builds into our existing CI

The first challenge we faced in incorporating macOS builds into our CI was finding the right infrastructure. The services provided by our current CI are running as AWS Elastic Cloud Compute (EC2) instances. Unfortunately, macOS images are not available for EC2. This meant that we would need to either subscribe to a separate service or self-host our own machines. We ultimately decided that we would have the most flexibility by hosting a number of on-premises Mac Minis and integrating them into our existing cloud.

The second (and more important) problem was determining how we could create an isolated build environment on the Mac. Concourse uses Docker containers for isolation on Linux. A number of Linux instances are registered with the Concourse task scheduler, and the scheduler sends them jobs to do in the form of Docker containers and volumes. On the Mac, however, Concourse executes builds directly in the host machine. This increases the chances for environmental drift and flakiness, ultimately reducing our ability to maintain the system and scale up our services. The solution to this problem was provided by Anka, a macOS virtualization technology for container DevOps by Veertu, Inc.

Anka and Concourse

Infrastructure – The primary components of our CI system are displayed in the following figure.



Our local cluster of Mac minis is connected to the VPC through a single entry point – the reverse proxy. The individual Macs register directly with the Anka controller and registry inside the VPC, both of which are hosted on a Linux EC2 instance. The Anka Build controller spawns and destroys VM instances on the Mac minis while requests to spawn or destroy an instance are made through the controller’s REST API.

The Anka Build Registry acts as a store of VM images. We pre-bake images with both the Concourse binary and the necessary tools and libraries to build our software, then push the images to the registry. If a Mac mini does not yet have an image saved locally, it will automatically download the image from the registry when a new VM is requested.

A startup script registers each newly created Anka VM instance as a worker with the ATC, which is the term that Concourse uses to refer to its task scheduler. Once registered, the worker is ready to receive data and build instructions from the ATC and stream the artifacts back into the VPC where they are ultimately stored in Amazon S3 buckets.

Creating new macOS VM templates

We streamline the creation of new Anka VMs with Veertu’s own builder extension for Packer by Hashicorp. A single Mac mini in our cluster is reserved for building new images. The creation of a new VM image starts with the execution of a script that handles the transfer of the necessary configuration files to the Mac builder via SSH. The same script then launches the packer build process directly on the Mac builder. Packer derives each new VM image from a base VM that mimics a fresh install of macOS. The result is a new Anka VM image which is pushed to the Anka registry inside the VPC, thereby making it available to each Mac host within the cluster.

This method of creating Anka VMs follows the infrastructure as code paradigm because the provisioning of the VM is configured with files that are parsed and executed by Salt. Another advantage is that the configuration of the VMs is stored in version control and old VM images may be stored on the registry in case we need to revert any changes to our CI.

Dynamic provisioning of the worker VMs

We faced another engineering challenge when it came to provisioning: how should dynamic information be provided to the newly spawned VMs? Such information includes the Concourse worker name and the cryptographic keys that are necessary to register with the ATC. We especially did not want to bake the keys into the VM images as this would pose a security risk. Fortunately, the Anka Controller REST API allows us to pass a startup shell script to each VM. A condensed version of the startup script’s Jinja template follows.

cat << EOF | sudo tee /srv/pillar/bootstrap.sls

{{ anka.anka_vm_pillar }}

EOF

sudo INSIDE_ANKA_VM="TRUE" salt-call --local state.apply


Two important events occur in the lifetime of this template. The first event is the rendering of the template which occurs inside the VPC on the Anka controller instance. The cryptographic keys and ATC endpoint are dynamically pulled from a secure parameter store when the Anka controller is provisioned on AWS. Importantly, this data is interpolated directly into the script at the line {{ anka.anka_vm_pillar }}.

The second event occurs immediately after a new VM is spawned. The script is passed from the controller to the new VM and executed, which directly injects the dynamic data into the VM instance’s pillar. (Pillar is the term used by the Salt provisioner for a static data store.) The Salt states that register the worker with Concourse are then applied to the VM by setting the INSIDE_ANKA_VM environment variable to TRUE and running salt-call –local state.apply.

After a successful salt-call, the new Concourse Mac worker should be registered with the Concourse ATC and ready to work.

Current limitations

There are currently two limitations to the setup described above.
  • On Linux, in which each Concourse task runs in a container, data is streamed between containers within the VPC. In the case of Anka and the Mac, the workers need to receive data from and send data to the VPC. This can incur some latency and bandwidth issues due to the Mac minis’ physical separation from the AWS hardware. We were able to work-around this limitation by shaping the traffic around our Mac cluster in our office’s router. We also expect that changes in streaming behavior in the upcoming Concourse releases to further improve streaming performance.
  • Concourse currently does not support on-demand VMs for the Mac. To prevent drift of the Build environment, a Cron job on the Anka controller respawns the VMs once per day. We found that this frequency was enough to keep our build environment fresh.


  • Summary

    In summary, we use Veertu, Inc.’s Anka virtualization technology to achieve well-isolated build environments for macOS builds. The unique circumstances regarding Apple license agreements meant that we needed to design a custom solution that integrated the Macs into our current CI system with Concourse on AWS. Despite this challenge, we were able to launch Concourse Mac workers inside Anka VMs that are running on an on-site cluster of Mac minis. The VMs are connected to the Concourse ATC inside our private cloud using credentials that are dynamically provisioned by the Anka controller.

    Overall, Anka has provided us symmetry with our Linux builds in Concourse.

    iOS 12 USB pairing with Anka VM for iOS CI

    In iOS 12 the lockdown procedure (mutual authentication of USB host and iOS device on each other) was modified. This lockdown procedure affects the workflow of attaching iOS devices to Anka VMs with USB pass-through for iOS CI real device testing using Anka Build or Anka Flow.

    We have fixed this issue in Anka version 1.4.3 and also determined that a specific set of instructions are required to pair iOS 12 device with the Anka VM Template and then it can be used in a fully automated manner as part of CI.

    Step 1 – If the device in the past, was already connected to host (on which the VM is going to run), then remove host record from the device DB, i.e., make the device forget the host. Execute the following to do this.
  • Detach the device from the host
  • On the device, Select Settings -> General -> Reset -> Reset Network Settings.
  • Step 2 – Attach the device to host and “Don’t Trust” the host in the dialog box that appears on the Device.



    If a dialog box also appears on the host, don’t ‘trust’ the device on the host.



    Step 3 – In the terminal window on the host, claim the device with these commands.

    sudo anka usb list

    sudo anka sub claim -n ios12iphone iphone/location

    “-n” flag can be used to associate a name with the claimed device. You can claim multiple devices connected to the same host with the same name, thus creating a group(name).



    Step 4 – Attach the device to the VM.

    anka start -d iPhone VMNAME

    The trust dialog will appear again on the device. Click on ‘trust’ the VM.



    and allow access to the device in the VM (trust device)



    Now the VM and the device are paired and accessible in Xcode and iTunes. You could detach the device, suspend or stop the VM. Next sessions of the VM or its clones on any other host should silently connect the device without additional prompts.



    Important – To pair another iOS12 device or other VMs, repeat steps 2 – 5, and make sure to “Don’t Trust” host on which you are performing the pairing. Contact us in our slack channel with additional questions on how to use the real devices with Anka Build dynamically provisioned VMs.

    Announcing Anka Build version 1.4.3 – Supports Docker inside Anka VMs

    Anka Build and Anka Flow version 1.4.3 is now available for all users. Some key new features in version 1.4.3 of Anka include support to run Docker and other hypervisors inside Anka VMs, create groups/clusters in Anka Build macOS cloud, priority based provisioning and core-based licensing. For more details see the version 1.4.3 release notes here.

    Run hypervisors inside Anka VMs

    Starting version 1.4.3, you can run Docker and other hypervisors like Anka, Android Emulator inside an Anka VM. See documentation to enable this feature on existing or newly created Anka VMs.

    Anka VM running Android Emulator and VirtualBox hypervisors



    Create groups/clusters to run VMs on specific Anka Build nodes(Host machines)

    When your Anka Build host machines are a mix of different types, you can create groups of similar machine types and provision specific Anka VMs to execute certain iOS CI jobs on specific groups.

    Also, use groups/clusters feature to assign pre-determined capacity to different groups using the Anka Build iOS CI infrastructure.  Grouping and Priority based provisioning features are available in Anka Build Enterprise Tier.

    Groups



    Creating Groups



    Core Based Licensing

    Going forward, Anka Build licensing will be generated for total cores in Anka Build Cloud instead of total machines and machine types. This will make it easier to add/remove and manage license key across multiple machines.

    Machine/Host-based licensing will be supported until the expiration of subscription to maintain compatibility with existing user setup. Sign up at https://veertu.com/getting-started-anka-trials/ to try Anka for iOS and macOS CI. 

    Quick setup of Anka Build macOS cloud on Mac hardware

    In this blog post, we will discuss how you can very quickly set up a macOS cloud using Anka Build exclusively on Mac Hardware. Anka Build cloud management software component is packaged as Linux Docker containers, and most common implementation architecture is to run them on a Linux instance. However, if you don’t have access to a Linux instance, you can set up a test/POC environment for Anka Build all on Mac hardware. The Mac hardware can be a single Mac machine (your MBP) or a cluster of Mac machines. We recommend this setup for quick proof of concept purposes, but you can use it for small-scale iOS CI operations.

    Setup on a single Mac machine

    Anka Build runs on all Mac hardware, including the latest 6-core 2018 mac minis. Follow these steps to get started.
  • Step 1 – Download Anka Build and Anka Build Controller and Registry for Mac packages from the Anka Build download page after trial registration. Note – Make sure there is enough free space on the Mac machine (at least 60GB).
  • Step 2 – Install AnkaBuild.pkg mac application on the Mac. https://ankadoc.bitbucket.io/getting-started/#anka-package-installation-and-upgrade-on-mac-hardware
  • Step 3 – Install the anka-controller-registry-mac-X.X.XX package. This package contains the management pieces of Anka Build macOS Cloud software. The controller gets installed as a docker container and registry is installed as a Mac application, Follow the instructions under Mac installation in https://ankadoc.bitbucket.io/getting-started/#registry-and-controller-installation-and-upgrade
  • Step 4 – Join the AnkaBuild node to Controller using ankacluster join command. https://ankadoc.bitbucket.io/using-controller/. For controller-address[:port], you will use http://localhost:8090.
  • Step 5 – Create your macOS VM template on the mac machine using AnkaBuild CLI. https://ankadoc.bitbucket.io/creating-vms/#using-macos-installer-application-recommended-to-create-mojave-hisierra-sierra-vms.
  • Step 6 – Connect the mac machine to the Registry using anka registry add [OPTIONS] REG_NAME REG_URL command. https://ankadoc.bitbucket.io/using-registry/#adding-a-new-registry. For REG_URL, you will specify the IP of your Mac. It’s the same value that you specified in step 3 in anka-controller. Docker file ENV REGISTRY_ADDR variable.
  • Step 7 – Push your macOS VM template to the registry using anka registry push command. https://ankadoc.bitbucket.io/using-registry/#pushing-vms-to-registry
  • Step 8 – You will now see this VM template under templates in Controller Portal dashboard.
  • You have now completed the setup of a single node macOS cloud and have one macOS VM template that you can use to provision on-demand VMs.
  • Step 9 – Go to Instances page on the portal dashboard and test using “Create Instances”.


  • Setup on Multiple Mac machines

    In this setup, it assumed that you want to set up a multi-node Anka Build macOS cloud. You will install the AnkaBuild package on all the Mac machines.

    Install the anka-controller-registry-mac-X.X.XX package on only one Mac machine, where you want to run the cloud management services. Follow the same instructions as Step3 outlined in the earlier section. Install the anka-controller-registry-mac-X.X.XX package on only one Mac machine, where you want to run the cloud management services. Follow the same instructions as Step3 outlined in the earlier section.

    Join all the Mac machines running AnkaBuild package to the Controller to create a multi-node cloud.

    Select one of the Mac machines to create VM templates from the nodes and execute steps 5.6.7 from the earlier section.

    You should now see multiple nodes in the portal dashboard.

    For additional questions, you can join the slack channel at https://slack.veertu.com/.

    T2 Chip and Anka co-existence

    Since the release of new 2018 mac minis, we have received several questions from users on the impact of T2 security chip included in the new mac minis on Anka Build setup. This blog describes in detail how Anka technology foundation is agnostic to T2 and can co-exist with it.

    There is enough written about the new T2 mac security chip, but as a quick summary, let’s look at the areas where it impacts existing mac administration and management workflows. T2 security chip is a security processor that is in charge of securing the following components of Mac hardware.

  • It validates the boot process (secure boot).
  • It does on-the-fly encryption of storage. Its full encryption with no performance loss due to software encrypt/decrypt I/O.
  • It protects the hardware from malicious usage scenario with hardware disconnect that ensures the microphone disables whenever the lid closes.


  • By default, all the new Mac Hardware including mac mini 2018 and newer iMac Pros have the T2 security chip enabled. So, what does it mean?

    1. Only certified operating systems, macOS and Windows, can boot on this hardware.

    2. NetBoot is not possible with Macs that have T2 hardware even when it is disabled.

    How does this affect the existing workflows?

      Installing other OS on top of Mac bare metal
    Before the introduction of T2 chip, it was possible to install software like ESXi, etc. on the Mac hardware. Now, even after disabling T2, access to SSD is still restricted, so booting from external USB is the only way to install other OS on the hardware. In this scenario, the OS can’t use the fast SSD inside the mac and will run on top of the slow USB device.

      Imaging
    Netboot has primarily been used to implement imaging in order to manage a group of macs. Now, with Netboot not working on the newer Mac hardware, it’s impossible to image a large group of macs consistently. It’s quite challenging now to set up and administer mac infrastructure for iOS CI.

    Anka installation on newer Macs is not impacted with T2 enabled. This is because Anka virtualization for macs is built on top of macOS hypervisor.framework. It works like any other mac application, which leverages and can work with full security on, utilizing all of the Mac hardware resources, including the fast internal SSD for maximum performance. So, Anka VMs can run with no impact of T2 enabled on the host machines.

    Anka VM on 2018 mac mini


    Anka Build is an alternative to explore if you wish to set up an agile and scalable mac infrastructure to provision immutable, container-like macOS VMs on-demand for iOS/macOS CI. Anka Build now supports Mojave and APFS.

    iOS build and test in Anka Build VMs on new 2018 mac mini – No performance compromise compared to running native on Mac Pro and Mac mini

    The new Mac mini 2018 which was released this month, with 8th Generation Intel CPU going up to six cores and support for up to 64GB of memory, is now a powerful contender for enterprise-grade macOS workloads. Here at Veertu, we were looking forward to this new mac hardware release to run Anka Build. Anka Build works out-of-box on the new mac mini hardware because it uses macOS Hypervisor.framework for virtualizing macOS VMS on top of macOS. What’s exciting us for is the performance of Anka Build on this new hardware and how it can change the game for iOS CI infrastructure. With new superpowerful CPUs, virtualization gets a boost, and we are seeing Anka Build VMs perform faster than running the same workload native on the older twelve core Mac Pro.

    Anka virtualized build/test iOS workload on new 2018 mac mini run faster than running it native on a twelve core Mac Pro

    We took the opensource Kickstarter iOS application and executed Xcodebuild build and test job in one Anka VM with 6vCPU (with HTT enabled) on the 6-core new 2018 mac mini and also native on a 12-core Mac Pro host. Anka VM performed ten percent better than native 12 core Mac Pro.
  • One Anka VM with 6 vCPU(HTT) on 2018 6-core mac mini – 1m21s
  • Bare metal Mac Pro 2013 with 12 cores (SSD) – 1m24s


  • Multiple Anka virtualized build/test iOS workloads on the new 2018 mac-mini run as fast as running it native on a quad core mac mini

    Again, we took the same opensource Kickstarter iOS project and executed Xcodebuild build and test job in two concurrent Anka VM with 6vCPU on the 6-core new 2018 mac mini and also native on a quad core mac mini. Both concurrent Anka VMs performed the same as a native quad-core mac mini.
  • Two Anka VM with 6 vCPU on 2018 6-core mac mini – 2m6s
  • Bare metal 4 core mac mini 2012 (2.3ghz SSD) – 2m4s


  • We also noticed that running concurrent Anka VMs on the new 2018 mac mini perform significantly better than running the same workload native on a dual core mac mini.
  • Two Anka VM with 6 vCPU on 2018 6-core mac mini – 2m6s
  • Bare metal dual-core 2014 (2.6 GHz, SSD) – 3m5s


  • So, what’s our takeaway from this?
    1. Devops and iOS software development teams are usually concerned about the performance impact of moving their iOS CI workloads to a virtualized environment. Since Anka Build virtualization on the new 2018 mac minis performs better or as good as native (better than native mac pro, significantly better than the native dual-core and as good as native quad-core), DevOps should explore running their workload on Anka Build macOS cloud on the new Mac minis.
    2. Most teams running virtualized environments for their IOS workloads are using Mac Pros (6 or 12 cores). They should explore the option of gaining a significant increase in the performance of their existing virtualized workloads by running them on Anka Build on top of the new 2018 mac minis.
    Contact us at info@veertu.com or https://slack.veertu.com/ to get more details on these benchmarks or with specific questions on running your iOS/macOS CI workloads.