Performance testing with new Anka release and 2018 minis

Anka macOS virtualization is now at version 2.1.2 with support for Catalina. Since our earlier releases starting from 1.x.x, we have been quietly making changes to improve the performance of iOS and macOS build and test jobs inside Anka VMs. Some of these changes are significant items shared in the release notes, and at other times, the changes are mostly optimizations in the Anka virtualization components. Here are some findings from the latest round of testing with version 2.1.2 and the new 2018 mini hardware.

Anka Version 2.1.2 on 2018 6-Core Mac minis

  • Project – Kickstarter iOS-OSS
  • Xcode – Version 10.3
  • macOS – Catalina (inside Anka VM and on the native hardware)


  • We used command line xcodebuild utility for the builds and ran clean after every execution on both native hardware and inside Anka VM.

    Build and test on 2018, 6-core mac mini, 32GB RAM took 263.6s.

    Build and test inside Anka VM with 12 vCPU, and 28 GB RAM took 282s. Performance degradation inside Anka VM is 7% compared to running it on native hardware. Generally, the performance degradation inside Anka VM compared to the native mac is less than 10%. If you tested with an older version of the Anka package(version 1.x), we recommend that you rerun the same tests with version 2.1.2 on the new 2018, minis. You will see a significant performance jump in running jobs inside Anka VMs.

    Share your findings with us in our slack channel, and we can assist with additional performance optimization tricks.

    Mac pro, mac mini or new mac mini for iOS CI?

    With the rapid growth in the iOS apps market, iOS development teams are moving their build and test setup from under their desks to managed, multi-machine environments. With the increase in team size, app count and complexity also comes business requirements to make building and to test the apps more consistent, predictable, and efficient. iOS development teams are looking for CI build and test infrastructure, which can scale dynamically and provide predictable results with minimal management. Cloud-powered by virtualization technology solves this problem.

    Anka Build solution enables the configuration of such a macOS cloud on mac hardware. The next question is which mac hardware to use for configuring private macOS cloud for iOS CI? There seem to be some preconceived assumptions that machines with large CPU core count are ideally suited for this.

    We ran a set of tests and discovered that the best rackable Mac hardware to configure private cloud for iOS CI is the 2018 mac mini machine. The 2018 mac minis give the most performance for virtualized build environments, which is one of the critical demands of iOS developers.

    These tests were done using the opensource https://github.com/kickstarter/ios-oss project. The results are very self-explanatory. We plan to run the same tests when the new Mac Pro becomes available.

    Model1 Anka VM running2 Concurrent Anka VM running
    2018, 6 core/32GB Mac Mini2m2.754s (12vCPU)2m58s(6vCPU)
    12 core/64GB Mac Pro6m3.718s(12vCPU)6m19s(6vCPU)
    8m41.625s(24vCPU)8m47s(12vCPU)
    2014, 2 core mac mini8m13.372s(4vCPU)NA
    2012, 4 core/16GB mac mini4m38.090s(8vCPU)7m56s(4vCPU)

    How to create your own real device cloud for iOS CI with Anka Build

    Anka Build enables you to configure your private macOS cloud. You can create and run macOS VMs on-demand for iOS CI and manage it as infrastructure-as-a-code. Additionally, with Anka Build Enterprise Tier, you can also configure a real device cloud to augment your simulator based CI testing.

    Anka Build enables you to work with real devices inside Anka macOS VMs through the USB interface. After connecting and claiming the real device on the mac host/node machines, dynamically attach the device to provisioned Anka VMs through the Anka Build Cloud Controller REST API interface. Check out detailed steps in our documentation.Register for a trial at www.veertu.com.

    Privileged user access macOS environments – locally delivered and centrally managed

    A large number of organizations every year suffer breach into their internal data or their customer’s data due to employee access. While employee access to sensitive data can’t be entirely blocked out, it’s possible to manage it as privileged access. There are a lot of solutions available to provide employees secondary windows privileged access work environments. However, there are not any solutions for macOS.

    Almost all of the existing solution takes the approach of completely locking down users’ primary macOS work environment to the extent that impacts their productivity. As an example, Macbooks of iOS developers in financial enterprises are generally entirely locked down and affects their ability to download any early/beta versions of third party software for R&D purposes. As more and more employees migrate from Windows-based systems to Macs, there exists a gap in delivering secondary privileged access macOS work environments for these users. Anka Secure fills this gap.

    Anka Secure is built on top of Anka macOS native virtualization engine and uses it to create and run lightweight macOS VMs on user’s mac workstations. It exposes services to prepare privileged users/groups, manages their access with VM policies, and delivers the privileged access secondary macOS work environment(VMs) on end-users mac workstations. The privileged access macOS VM based work environments run locally on the user’s primary mac workstations. This approach provides a local secure, sandbox, privileged macOS environments with best user experience and performance.

    Who can benefit from Anka Secure :
    1. Technical and customer support who have access to privileged data from their mac workstations.
    2. Developers in enterprises with access to corporate developer/application certificates and other privileged data from their MacBooks.
    3. iOS developers working in secure enterprise setup and need isolated secondary mac environments for research purposes on their MacBooks.
    4. Environments for Malware and other sensitive analysis and testing tasks for macOS.
    Discover more and try it at https://veertu.com/anka-secure/.

    Anka version 2.0

    Anka version major release is now out and available for download. We are very excited to announce the availability of Enterprise and Enterprise Plus tiers of Anka, along with several other exciting improvements and features.Here is a shortlist of some of the new features and improvements. For a complete list, visit the release notes and product documentation.
    • Up to fifteen percent improvement in performance inside Anka VMs – Significantly closes the performance gap between Anka VMs running on 2018 mac mini and running on native hardware.
    • Support for Catalina Beta 3 VMs – Create and test with Catalina beta macOS VMs.
    • Native Mac App package to for controller/registry – Install and run Controller and Registry on Mac.
    • Unified single Anka package – Deploy the same package for Build or Flow and activate features with the license key.
    • New Queue management module in the Controller – Provides more flexibility for future improvements and scales well.
    • Support for single user Controller Portal authentication – Admins user access to the Controller Portal.
    • Support for multi-user access control and management through OAUTH based SSO integration – Multi-user access to the Portal.
    • Secure, policy managed, and encrypted VMs – Configure Security policies and encryption on the Build VMs.
    • Management of testing on real devices attached to the VMs through the cOntroller REST APIs – Enables creation of a local device cloud.
    • Certificate-based authentication between Plugins, Controller, Registry, and Nodes – Enables a secure setup.
    To make it extremely easy for users to deploy Anka Build, we now have a Mac Application package to install Controller and registry services on Mac hardware.If you are new to Anka, visit the products page and register for the Anka Build trial. 

    T2 in new Mac hardware and its impact on virtualizing and running macOS – Role of SMC

    In the last few weeks, we have received a lot of inbound questions on the existence of T2 chip in new Apple Mac hardware models and how it impacts the ability to run macOS VMs. In this blog series, we will try to share our knowledge and insights on this topic.

    The focus of this first blog is on the role SMC plays in booting macOS and how T2 chip impacts this function. Apple T2 chip can control many aspects of the macOS platform over a single unified bus. One of them is, performing additional firmware validation in a trusted execution environment before supplying it to the chipset for execution. During macOS booting, macOS is accessing the hardware SMC chip to read key validation of the Mac.

    In T2 enabled Mac hardware, T2 chip is acting as gatekeeper to SMC. Hackintosh and KVM  based projects, which use Clover, are using the saved key of the SMC key, and bypassing the SMC validation. And, tools like ESXi access to SMC in this hardware is blocked by T2, unable to boot macOS VMs (https://twitter.com/lamw/status/1120368830427959297).

    However, any solution that leverages macOS native hypervisor.framework like Anka, can boot macOS VMs without any issues. In this scenario, SMC calls are placed through the hypervisor.framework APIs and T2 is able to pass those onwards.



    T2 chip in new Mac hardware not only acts as a gatekeeper during the boot process, but also prevents unauthorized access to internal SSD and Thunderbolt ports. So, even if any tool somehow manages to boot macOS VMs(using saved SMCkeys), it will not have access to the SSD. The workaround would be to use USB or network attached storage which is slow, not scalable and unreliable. More on this in our next blog.

    Let us know if you have additional questions/comments in our slack channel.

    References – https://www.apple.com/mac/docs/Apple_T2_Security_Chip_Overview.pdf

    Cloud-based continuous integration for macOS with Concourse and Anka

    Pix4D creates professional photogrammetry software for drone mapping. The people who use our products include surveyors, farmers, and researchers, and they use the software on a range of platforms including desktop, mobile, and cloud. Pix4D’s continuous integration (CI) team is based in the headquarters in Lausanne, Switzerland, where we design and maintain the systems used by our developers to automatically build, test, and package our suite of products.

    Our CI system is designed around the paradigm of infrastructure as code. We manage the system’s resources using text-based, version-controlled configuration files and automation tools such as Terraform and Salt. All resources are located in a virtual private cloud (VPC) and hosted by Amazon Web Services (AWS). We use Concourse CI as our CI software because it is fully configurable by YAML files and scripts that may be shared, reviewed, and tracked in version control as an application code base would be managed. In addition, each task that is run on a Linux platform is executed inside a Docker container which provides the isolation that is necessary for reproducible builds.

    Recently our developers began building some of our software products for macOS. This meant that we had to find a way to incorporate macOS builds into our CI environment with Concourse.

    Integrating macOS builds into our existing CI

    The first challenge we faced in incorporating macOS builds into our CI was finding the right infrastructure. The services provided by our current CI are running as AWS Elastic Cloud Compute (EC2) instances. Unfortunately, macOS images are not available for EC2. This meant that we would need to either subscribe to a separate service or self-host our own machines. We ultimately decided that we would have the most flexibility by hosting a number of on-premises Mac Minis and integrating them into our existing cloud.

    The second (and more important) problem was determining how we could create an isolated build environment on the Mac. Concourse uses Docker containers for isolation on Linux. A number of Linux instances are registered with the Concourse task scheduler, and the scheduler sends them jobs to do in the form of Docker containers and volumes. On the Mac, however, Concourse executes builds directly in the host machine. This increases the chances for environmental drift and flakiness, ultimately reducing our ability to maintain the system and scale up our services. The solution to this problem was provided by Anka, a macOS virtualization technology for container DevOps by Veertu, Inc.

    Anka and Concourse

    Infrastructure – The primary components of our CI system are displayed in the following figure.



    Our local cluster of Mac minis is connected to the VPC through a single entry point – the reverse proxy. The individual Macs register directly with the Anka controller and registry inside the VPC, both of which are hosted on a Linux EC2 instance. The Anka Build controller spawns and destroys VM instances on the Mac minis while requests to spawn or destroy an instance are made through the controller’s REST API.

    The Anka Build Registry acts as a store of VM images. We pre-bake images with both the Concourse binary and the necessary tools and libraries to build our software, then push the images to the registry. If a Mac mini does not yet have an image saved locally, it will automatically download the image from the registry when a new VM is requested.

    A startup script registers each newly created Anka VM instance as a worker with the ATC, which is the term that Concourse uses to refer to its task scheduler. Once registered, the worker is ready to receive data and build instructions from the ATC and stream the artifacts back into the VPC where they are ultimately stored in Amazon S3 buckets.

    Creating new macOS VM templates

    We streamline the creation of new Anka VMs with Veertu’s own builder extension for Packer by Hashicorp. A single Mac mini in our cluster is reserved for building new images. The creation of a new VM image starts with the execution of a script that handles the transfer of the necessary configuration files to the Mac builder via SSH. The same script then launches the packer build process directly on the Mac builder. Packer derives each new VM image from a base VM that mimics a fresh install of macOS. The result is a new Anka VM image which is pushed to the Anka registry inside the VPC, thereby making it available to each Mac host within the cluster.

    This method of creating Anka VMs follows the infrastructure as code paradigm because the provisioning of the VM is configured with files that are parsed and executed by Salt. Another advantage is that the configuration of the VMs is stored in version control and old VM images may be stored on the registry in case we need to revert any changes to our CI.

    Dynamic provisioning of the worker VMs

    We faced another engineering challenge when it came to provisioning: how should dynamic information be provided to the newly spawned VMs? Such information includes the Concourse worker name and the cryptographic keys that are necessary to register with the ATC. We especially did not want to bake the keys into the VM images as this would pose a security risk. Fortunately, the Anka Controller REST API allows us to pass a startup shell script to each VM. A condensed version of the startup script’s Jinja template follows.

    cat << EOF | sudo tee /srv/pillar/bootstrap.sls

    {{ anka.anka_vm_pillar }}

    EOF

    sudo INSIDE_ANKA_VM="TRUE" salt-call --local state.apply


    Two important events occur in the lifetime of this template. The first event is the rendering of the template which occurs inside the VPC on the Anka controller instance. The cryptographic keys and ATC endpoint are dynamically pulled from a secure parameter store when the Anka controller is provisioned on AWS. Importantly, this data is interpolated directly into the script at the line {{ anka.anka_vm_pillar }}.

    The second event occurs immediately after a new VM is spawned. The script is passed from the controller to the new VM and executed, which directly injects the dynamic data into the VM instance’s pillar. (Pillar is the term used by the Salt provisioner for a static data store.) The Salt states that register the worker with Concourse are then applied to the VM by setting the INSIDE_ANKA_VM environment variable to TRUE and running salt-call –local state.apply.

    After a successful salt-call, the new Concourse Mac worker should be registered with the Concourse ATC and ready to work.

    Current limitations

    There are currently two limitations to the setup described above.
  • On Linux, in which each Concourse task runs in a container, data is streamed between containers within the VPC. In the case of Anka and the Mac, the workers need to receive data from and send data to the VPC. This can incur some latency and bandwidth issues due to the Mac minis’ physical separation from the AWS hardware. We were able to work-around this limitation by shaping the traffic around our Mac cluster in our office’s router. We also expect that changes in streaming behavior in the upcoming Concourse releases to further improve streaming performance.
  • Concourse currently does not support on-demand VMs for the Mac. To prevent drift of the Build environment, a Cron job on the Anka controller respawns the VMs once per day. We found that this frequency was enough to keep our build environment fresh.


  • Summary

    In summary, we use Veertu, Inc.’s Anka virtualization technology to achieve well-isolated build environments for macOS builds. The unique circumstances regarding Apple license agreements meant that we needed to design a custom solution that integrated the Macs into our current CI system with Concourse on AWS. Despite this challenge, we were able to launch Concourse Mac workers inside Anka VMs that are running on an on-site cluster of Mac minis. The VMs are connected to the Concourse ATC inside our private cloud using credentials that are dynamically provisioned by the Anka controller.

    Overall, Anka has provided us symmetry with our Linux builds in Concourse.

    iOS 12 USB pairing with Anka VM for iOS CI

    In iOS 12 the lockdown procedure (mutual authentication of USB host and iOS device on each other) was modified. This lockdown procedure affects the workflow of attaching iOS devices to Anka VMs with USB pass-through for iOS CI real device testing using Anka Build or Anka Flow.

    We have fixed this issue in Anka version 1.4.3 and also determined that a specific set of instructions are required to pair iOS 12 device with the Anka VM Template and then it can be used in a fully automated manner as part of CI.

    Step 1 – If the device in the past, was already connected to host (on which the VM is going to run), then remove host record from the device DB, i.e., make the device forget the host. Execute the following to do this.
  • Detach the device from the host
  • On the device, Select Settings -> General -> Reset -> Reset Network Settings.
  • Step 2 – Attach the device to host and “Don’t Trust” the host in the dialog box that appears on the Device.



    If a dialog box also appears on the host, don’t ‘trust’ the device on the host.



    Step 3 – In the terminal window on the host, claim the device with these commands.

    sudo anka usb list

    sudo anka sub claim -n ios12iphone iphone/location

    “-n” flag can be used to associate a name with the claimed device. You can claim multiple devices connected to the same host with the same name, thus creating a group(name).



    Step 4 – Attach the device to the VM.

    anka start -d iPhone VMNAME

    The trust dialog will appear again on the device. Click on ‘trust’ the VM.



    and allow access to the device in the VM (trust device)



    Now the VM and the device are paired and accessible in Xcode and iTunes. You could detach the device, suspend or stop the VM. Next sessions of the VM or its clones on any other host should silently connect the device without additional prompts.



    Important – To pair another iOS12 device or other VMs, repeat steps 2 – 5, and make sure to “Don’t Trust” host on which you are performing the pairing. Contact us in our slack channel with additional questions on how to use the real devices with Anka Build dynamically provisioned VMs.

    Announcing Anka Build version 1.4.3 – Supports Docker inside Anka VMs

    Anka Build and Anka Flow version 1.4.3 is now available for all users. Some key new features in version 1.4.3 of Anka include support to run Docker and other hypervisors inside Anka VMs, create groups/clusters in Anka Build macOS cloud, priority based provisioning and core-based licensing. For more details see the version 1.4.3 release notes here.

    Run hypervisors inside Anka VMs

    Starting version 1.4.3, you can run Docker and other hypervisors like Anka, Android Emulator inside an Anka VM. See documentation to enable this feature on existing or newly created Anka VMs.

    Anka VM running Android Emulator and VirtualBox hypervisors



    Create groups/clusters to run VMs on specific Anka Build nodes(Host machines)

    When your Anka Build host machines are a mix of different types, you can create groups of similar machine types and provision specific Anka VMs to execute certain iOS CI jobs on specific groups.

    Also, use groups/clusters feature to assign pre-determined capacity to different groups using the Anka Build iOS CI infrastructure.  Grouping and Priority based provisioning features are available in Anka Build Enterprise Tier.

    Groups



    Creating Groups



    Core Based Licensing

    Going forward, Anka Build licensing will be generated for total cores in Anka Build Cloud instead of total machines and machine types. This will make it easier to add/remove and manage license key across multiple machines.

    Machine/Host-based licensing will be supported until the expiration of subscription to maintain compatibility with existing user setup. Sign up at https://veertu.com/getting-started-anka-trials/ to try Anka for iOS and macOS CI. 

    Quick setup of Anka Build macOS cloud on Mac hardware

    In this blog post, we will discuss how you can very quickly set up a macOS cloud using Anka Build exclusively on Mac Hardware. Anka Build cloud management software component is packaged as Linux Docker containers, and most common implementation architecture is to run them on a Linux instance. However, if you don’t have access to a Linux instance, you can set up a test/POC environment for Anka Build all on Mac hardware. The Mac hardware can be a single Mac machine (your MBP) or a cluster of Mac machines. We recommend this setup for quick proof of concept purposes, but you can use it for small-scale iOS CI operations.

    Setup on a single Mac machine

    Anka Build runs on all Mac hardware, including the latest 6-core 2018 mac minis. Follow these steps to get started.
  • Step 1 – Download Anka Build and Anka Build Controller and Registry for Mac packages from the Anka Build download page after trial registration. Note – Make sure there is enough free space on the Mac machine (at least 60GB).
  • Step 2 – Install AnkaBuild.pkg mac application on the Mac. https://ankadoc.bitbucket.io/getting-started/#anka-package-installation-and-upgrade-on-mac-hardware
  • Step 3 – Install the anka-controller-registry-mac-X.X.XX package. This package contains the management pieces of Anka Build macOS Cloud software. The controller gets installed as a docker container and registry is installed as a Mac application, Follow the instructions under Mac installation in https://ankadoc.bitbucket.io/getting-started/#registry-and-controller-installation-and-upgrade
  • Step 4 – Join the AnkaBuild node to Controller using ankacluster join command. https://ankadoc.bitbucket.io/using-controller/. For controller-address[:port], you will use http://localhost:8090.
  • Step 5 – Create your macOS VM template on the mac machine using AnkaBuild CLI. https://ankadoc.bitbucket.io/creating-vms/#using-macos-installer-application-recommended-to-create-mojave-hisierra-sierra-vms.
  • Step 6 – Connect the mac machine to the Registry using anka registry add [OPTIONS] REG_NAME REG_URL command. https://ankadoc.bitbucket.io/using-registry/#adding-a-new-registry. For REG_URL, you will specify the IP of your Mac. It’s the same value that you specified in step 3 in anka-controller. Docker file ENV REGISTRY_ADDR variable.
  • Step 7 – Push your macOS VM template to the registry using anka registry push command. https://ankadoc.bitbucket.io/using-registry/#pushing-vms-to-registry
  • Step 8 – You will now see this VM template under templates in Controller Portal dashboard.
  • You have now completed the setup of a single node macOS cloud and have one macOS VM template that you can use to provision on-demand VMs.
  • Step 9 – Go to Instances page on the portal dashboard and test using “Create Instances”.


  • Setup on Multiple Mac machines

    In this setup, it assumed that you want to set up a multi-node Anka Build macOS cloud. You will install the AnkaBuild package on all the Mac machines.

    Install the anka-controller-registry-mac-X.X.XX package on only one Mac machine, where you want to run the cloud management services. Follow the same instructions as Step3 outlined in the earlier section. Install the anka-controller-registry-mac-X.X.XX package on only one Mac machine, where you want to run the cloud management services. Follow the same instructions as Step3 outlined in the earlier section.

    Join all the Mac machines running AnkaBuild package to the Controller to create a multi-node cloud.

    Select one of the Mac machines to create VM templates from the nodes and execute steps 5.6.7 from the earlier section.

    You should now see multiple nodes in the portal dashboard.

    For additional questions, you can join the slack channel at https://slack.veertu.com/.