Configuring iOS real device infrastructure for in-house parallel mobile test execution

Agile development is putting a lot of pressure on traditional CI and testing approaches. It demands faster build and test times so that feedback can be provided quickly to the developers. It also assumes that test system related issues shouldn’t slow down CI pipelines and that tests in isolated, sterile reproducible environments offer the most reliable feedback. For mobile projects, that can also mean performance testing on real devices as well.

Let’s look at the particular case of testing iOS applications on real devices in CI. Execution time for tests is very critical, and testing on multiple devices sequentially would take a long time. The problem gets worse with a lot of test executions and the requirement to keep pace with rapid build frequencies. If the budgets are limitless, then hosted device farms are a great option. Most often teams discover that unlike other cloud services which have gotten cheaper over time, hosted device farms prices don’t come down, but only increase. It’s because new devices get added, and the frequency of test execution keeps growing. As a result, the total cost can get pricey. Additionally hosted device test farms may not benefit the security posture of some organizations that want to secure the protection of their intellectual property.

The approach users take in this scenario is to connect as many iOS devices as possible to available macOS hardware and run test sequentially through CI. But, anyone who is doing this is familiar with the challenges of maintaining and automating test execution on such an infrastructure. They include but are not limited to–cleaning up after a test, starting a new test in a sterile setup, executing tests in parallel on multiple iOS devices (not an issue in newer Xcode version) and more.

One alternate approach is to launch a clean macOS VM which has an iOS device connected for every test execution. Each test run gets a sterile environment and environments with particular test tool settings can be consistently reproduced for very execution. When you build a macOS self-service on-premise cloud with Anka Build, you can programmatically manage and automate association of physically connected iOS devices to macOS VMs for running parallel iOS real device tests.

Anka Build exposes the ability to manage iOS device association with the dynamically provisioned macOS VMs through a command line interface and enables automation of this task. Then, it can easily integrate with the existing iOS CI process.

How does it work?

Let’s say that you have a cluster of centralized build/test Apple machines (on-premise or hosted).

Step 1 – Connect one or more iOS devices to these machines.

Step 2 – Create a group using `anka usb-group` command. The Anka USB group feature eliminates the need to pre-define/attach individual iOS devices to test environments. A dynamically provisioned test environment/VM can, at run time, get a free device from the USB group (think of the group as a pool) attached to it, execute the test and release the device for use in the subsequent test.

Step 3 – Add the iOS devices to this usb group.

anka usb-group --help

Usage: anka usb-group [OPTIONS] COMMAND [ARGS]...

manage usb device groups, and you can attach a group to a vm so the vm could pull an available usb device from that group

Options: --help Show this message and exit. [optional]

Commands:

add-device : adds a device to a usb group, if the group...

create : create a new usb group

delete : delete a usb group and release all it's...

list : show all usb groups

list-group-devices : show all devices that belongs to a group

remove-device : remove a device from a usb group


Step 4 – Attach this usb group (pool of iOS devices) to the VM using anka modify command.

anka modify VM add usb-group --help

Usage: anka modify add usb-group [OPTIONS] GROUP_NAME


Step 5 – Push the VM as a base VM to Anka registry.

Step 6 – Now, when you provision test environments with this base VM on the build host with the devices attached, the VMs will claim the first available devices from the group, execute the test and release them for the next test job execution.

This approach provides a clean test environment setup for every test run, reduces the management overhead of claiming/releasing devices and offers a programmatic interface to do this automatically.

If you would rather work with the single iOS device instead of a pool, you can use the same USB pass through Anka USB command line interface.

anka usb [OPTIONS] COMMAND [ARGS]...

Do actions on USB devices

Options: --help Show this message and exit. [optional]

Commands:

claim : make a device available for attaching to vms,...

list : list all available usb devices on this host

release : release a device back to host availability,...

anka modify add usb-device [OPTIONS] DEVICE_IDENTIFIER


For detailed information on how to setup a self-service macOS on-premise cloud, read this Anka Build blog.

Anka Beta new version 1 Build 41 update – For self service on-premise macOS VMs

Today, we are announcing the availability of Anka Beta version 1 build 41 to all the beta users. Please check your email for the download link. New users can sign up at www.veertu.com. In this update, we have added some pretty cool feature updates to Anka macOS VMs configuration, that are directly relevant to use cases for enabling DevOps to setup a macOS on-premise CI infrastructure to test their enterprise wide deployment scripts, upgrade processes etc for Mac workstations. These features were implemented based on beta user feedback.

Check out the version 1 build 41 release notes for more details.

Let’s take an example: You are an enterprise with a large deployment mac hardware(laptops etc) being used by the employees. The portfolio of Mac hardware being used can be a mix of different models. Now, you have a set of bootstrap actions that need to take place on every employee mac or a set of macs. For example, applications deployed, permissions etc. Before rolling out any new changes to your bootstrap process, you want to test it. Because once it’s rolled out to thousand of employees, the cat is out of the bag and it can be challenging to roll back the changes if they don’t work as designed.

In this case, you can take your cluster of Mac test hardware, deploy Anka Build on it to transform into a self-service macOS cloud. Now, use this infrastructure to do CI testing of your employee mac machine bootstrap process’. Spin up reproducible, sterile Anka macOS VMs on-demand and simulate testing on different models by modifying hw.product, hw.family, hw.uuid etc. parameters for the Anka VMs.

                                           

Anka Packer Builder – To create macOS Anka VMs for iOS/macOS CI

Thanks to Lachlan Donald, there is now a Packer Builder for Anka. The Anka Packer builder is able to create macOS Anka VMs starting from a macOS installer app.

The builder builds a fully functional macOS VM from scratch, provisioned with Anka PV network and block drivers for near native build and test performance. You can configure CPU cores, RAM and Disk size. Quickly setup your Anka VM and then use ‘anka run’ to configure it with your dependencies.

Download it from GitHub. Packer Builder for macOS Anka VM

Refer to Anka documentation to create macOS Anka VMs from “*.dmg” or “*.iso”.

Anka Build Terminology – Components to setup on-premise iOS CI infrastructure

Anka Build is macOS native virtualization platform from Veertu. It’s a complete platform to implement and manage macOS VMs on a large scale for iOS CI. Anka Build includes lightweight Anka client, which is the native hypervisor to run macOS VMs on Mac hardware, the Controller, which is used to on-demand provision and manage macOS VMs and the Registry, which is used to store and distribute macOS VM templates for CI.

While Anka Build manages macOS VMs on a large scale for CI, all Anka Build modules by themselves are very lightweight, easy to deploy and manage.

iOS Ci with Anka Build

Anka Client

Anka client is a very lightweight mac application. Unlike, other server virtualization solutions for mac hardware, Anka Build doesn’t strip away the core macOS. Anka Build runs as another mac application on the mac hardware and enables the creation and running of macOS VMs. Anka Build macOS VMs are very responsive and perform at near native speed. These Anka VMs are used as on-demand build and test environments for iOS/macOS CI.

Anka Controller

Anka Controller module is packaged as a docker container. The Controller provides a single interface through its REST APIs to on-demand provision macOS VMs, requested by CI jobs. Think of it as central management for your Anka Build Cloud.

Anka Registry

Anka Registry module is used to store and distribute different versions of macOS VMs. Anka Registry is also packaged in a Docker container. A iOS/macOS CI job can request X quantity of N type of macOS VM. When Controller receives this request, it pulls N macOS VM (version tag) from the registry and spins up X quantity of these VMs using Anka Client on your Anka nodes. Anka Controller also maintains a queue of all incoming requests from the CI system and manages them.

Anka Node

Anka node is one or many Mac hardware on which you want to run your iOS CI jobs. You install Anka Build client on each of your Anka Build Nodes and then connect the nodes to the Anka Controller.

Connecting Anka Build to your CI system

While Anka Build platform consists of multiple modules, you only need to integrate your CI system to Anka Build Controller in order to start using Anka Build for executing iOS CI jobs. Anka Build Controller exposes REST APIs, through which you can integrate your existing CI system.

For Jenkins based CI implementation, there is a Jenkins plugin available, which can be used to very easily set up integration with Anka Build iOS cloud infrastructure.

Performance and Scalability in virtualized iOS build and test infrastructure

Growth is good, and when development teams grow, often, they find themselves having to choose between performance and scalability factors related to their development process. Most often, iOS application development project start by consuming hosted service offerings for their build and test processes. Then, as growth happens, team size increases, the velocity of code commits increase significantly, the number of projects increase, and soon performance issues like long queue wait times and longer build and test times start to creep up.

In most scenarios, one can throw more resources at this problem to address the issue. However, in the macOS world, that’s not the case. While scalable and high-performing platforms like AWS, Google Cloud, Azure which are powered by virtualization exist for Linux and Windows, nothing similar exists for macOS. All existing approaches to virtualization of macOS suffer from a tradeoff between scalability and performance.

Veertu Anka hypervisor eliminates the need for the user to make this tradeoff. Anka Build, powered by Anka hypervisor, enables growing iOS development teams to very easily, setup and configure a scalable and high performing AWS like private cloud for iOS build and test.

There is a lot of skepticism around the performance of iOS simulator test jobs on virtualized infrastructure, especially as the test portfolio expands. Anka Build addresses this for its users with a combination of three variables. – The performance of Anka VM, Time to provision an on-demand VM and VMs in Anka registry with pre-boot iOS simulator.

Anka VMs run on top of Anka Hypervisor, which is a new lightweight virtualization technology that leverages the macOS Hypervisor.framework to run macOS VMs in the most resource efficient manner. Instead of directly requesting and managing macOS resources like CPU or memory for the VMs, Anka Hypervisor, through the macOS Hypervisor.framework, lets the operating system manage the VM resource needs in the same way it runs other applications. As a result, Anka VMs perform much faster and are more responsive. Anka VMs are also packaged with a para-virtual block and network drivers, to enable faster execution of build and test jobs.

Time to provision or the time it takes to have an environment ready for the iOS build and test job is a significant factor that impacts overall performance. Anka Build addresses this with ‘Instant Start’ feature. ‘Instant Start’ offers on-demand provisioned instances instantaneously (1-2 sec) to execute the iOS Ci jobs.

iOS simulator UI test job performance on dedicated or virtualized hardware is impacted by, the long boot time of iOS Simulator. With Anka Build, DevOps can address this problem by pre-booting the simulator in the VM for each device type and pushing the VM to Anka registry. Then, CI UI test jobs, through Anka controller can on-demand provision instances from these VMs and use the pre-booted iOS simulator at full throttle. Each test is run in a clean, sterilized environment with pre-booted simulator of a particular device type.

In our internal tests, we have recorded that iOS simulator UI tests performed with Anka Build and pre-booted simulator VMs take less time than when running on dedicated hardware.

                                         

iOS simulator test in Anka VM



               

iOS simulator test on dedicated hardware

Building a private iOS Device Cloud

Now that you have Anka Build up and running for your iOS CI infrastructure, your developers are more productive and your environment is easier to manage. But how do you test your iOS app releases on multiple iOS devices to make sure they are rendering properly?

Anka Build enables you to execute tests on real devices, in the same way as your simulator-based test workflow. Cable your physical devices to your Anka Build macOS private cloud nodes, then connect them at run time to on-demand macOS VMs and run your tests. Using Anka, you can manage a single device or a group of similar devices through a simple command line interface.

What are the benefits of this approach?
  1. One single scalable iOS infrastructure for build, simulator based tests and real device tests.
  2. Eliminates the need to separately manage mac hardware connected to real devices for CI.
  3. Management of real devices through a simple, command line interface.


Check out the following example of making a real device available on-demand for testing:

Step 1 – Create a VM to be used for real device testing

Step 2 –  Let’s say you need to run multiple tests on multiple ipad models. Create a USB-group called ipadgrp:

`anka usb-group create ipadgrp`

Step 3 – Add all of your ipad models to this group:

`anka usb-group add-device <device identifier1> ipadgrp`

`anka usb-group add-device <device identifier 2> ipadgrp`

Step 4 – Attach this usb-group to the previously created VM:

`anka modify VM add sub-group ipadgrp`

Step 5 – Configure your specific CI job to use this VM for real device testing.

Step 6 – Use Anka Controller to spin up on-demand instances of this VM. Each VM instance at run time will claim an available ipad device from the `ipadgrp` and make it available for testing.

Additional documentation on this feature is available at using real devices with Anka Build.

Building on-premise iOS/macOS build and test infrastructure in four steps

Anka Build controller is the management engine for your macOS private cloud. It presents a single entry point, through its REST APIs, for external systems to provision macOS VMs on a cluster of mac hardware (Anka Build nodes) from a specific version of a VM template pulled from Anka registry, for each run.

It eliminates the need to implement custom modules to address issues like reverting a VM to the snapshot, automated assignment of IPs, multiple request management through queuing, cleanup and other management tasks.

Pulling a VM template from the Anka registry for on-demand provisioning also eliminates the need to bootstrap the VM at runtime and enables instantaneous execution of tests. You might still need to do some incremental bootstrapping to pull latest build etc, but the initiation time is significantly reduced.





Watch Anka Build in action in this brief video – Demo Video

Let’s take a look at how you will set it up.

Step 1 – Deploy Anka Registry in a docker container. Installing Registry section in https://ankadoc.bitbucket.io/getting-started/

Step 2 – Deploy Anka controller in a docker container and configure it. Installing Controller section in https://ankadoc.bitbucket.io/getting-started/

Step 3 – Deploy Anka Build application package on all the mac machines where you want to run your VMs. Then, join them to the controller build cluster. These machines are now Anka build nodes. https://ankadoc.bitbucket.io/using-controller/

Step 4 – Create the Anka VM templates that will serve as your iOS or macOS build or test environments. Load all your dependencies in this VMs. Then, push it to Anka registry. If your dependencies change, you can pull this VM from the registry, update it and then push it back with a different version/tag. https://ankadoc.bitbucket.io/using-registry/

Your private macOS cloud infrastructure is now ready.

Use the Anka Controller REST APIs to provision environments or if you use Jenkins, deploy the Jenkins plugin.

REST APIs – https://ankadoc.bitbucket.io/using-controller/#controller-rest-apis

Jenkins Plugin – https://ankadoc.bitbucket.io/using-jenkins/

Developing iOS applications in consistent local environments

Is iOS developer productivity suffering in your organization? Let’s take a scenario of a company with more than one iOS developer working on different iOS projects. In such a setting developers tend to have issues running builds on CI, only to have it pass in their local environment. These types of failures during integrated build and test significantly impact developer productivity. When you dig into these issues, it’s very often because of incorrect global dependency linking, different library versions, incorrect or missing environment variables, or simply that their system is not set up with consistency.

While in theory, it’s possible to sanitize and standardize a developer’s machine, in practice, it’s almost impossible. Developers will always download new libraries and update versions of other tools to accelerate development, sometimes creating conflict with your build tools. Similarly, when iOS developers pull the latest source for development and build it on their systems, they face local build and test failures due to incompatibilities in the configuration of their dev environment. This problem is even bigger when iOS developers are simultaneously working on multiple projects.

Anka Run technology is built to address this issue. It brings the container-based development workflow to the iOS and macOS development world. Until now, it was not possible to package an iOS development environment in a container-like entity and distribute it to your team. Anka Run enables you to do this.

There are two modules in Anka Run. Anka Run for macOS and Anka Registry. Anka Run for macOS is a very lightweight engine with a footprint of 15MB, to create and run macOS VMs. Anka Registry is a repository where these iOS or macOS dev environment VMs are managed and can be pulled to each developer machine, creating a consistent developer experience everyone can use.

First, Anka Run is deployed on all iOS developer machines, and Anka Registry is set up in a docker container. After initial installation is complete, devOps engineers and/or developers can create fully self-contained iOS development environments with all dependencies in an “Anka VM”. This Anka VM is then versioned and pushed to Anka registry.

Next, developers continue to work in their native Xcode IDE or another environment. They pull the respective Anka VM corresponding to multiple projects and configurations on their machine and use them to execute local build and unit tests. With the ‘anka run’ command developers work from their local command line interface to execute builds and tests inside the Anka VMs.

The end result is that iOS developers are always building and testing against a common set of dependencies and linkages, without the need to set these on their machines. Simply put, Anka enables an agile and developer-centric approach.

Watch this video to see all Anka Run components in action.

Setup macOS 10.13 beta High Sierra environment in an isolated VM

This week Apple announced developer beta availability of the new macOS version 10.13, High Sierra and, we started investigating support for this in Anka. We are excited to share that now you can create isolated dev/test environment with this version in an Anka VM.

Our developers have discovered that this macOS beta version prevents auto install/load of third party signed drivers. These drivers need to be explicitly enabled by the users. Hence, just using Anka auto install method to create VM is not enough. After creating the VM, you will need to do additional setup to enable Anka PV drivers.

Following are the setup steps

Step 1 – Download and install Anka Beta from www.veertu.com

Step 2 – Download ‘Install macOS 10.13 Beta.app’ from your developer account.

Step 3 – Use Anka auto install method to create High Sierra VM. Type following in command line.
anka create-disk --size 40G --app ~/Desktop/Install\ macOS\ 10.13\ Beta.app/


Step 4 – After successful execution of step 4, execute ‘anka create’.
disk created successfully at: diskid.ank
 
anka create --image-id diskid --ram-size 2G --cpu-count 2 hisierra


Step 5 – Type ‘anka view hisierra’ and complete the remaining macOS setup steps.



Step 6 – Go to system preferences and then go to Security and Privacy and allow extension by Veertu Inc.



Step 7 – Stop and start the VM.

Step 8 – Run the following from a terminal inside the VM.
cd /Library/Filesystems/osxfuse.fs/Contents/Extensions
 
sudo ln -ls 10.9 10.13
Step 9 – Stop and start the VM.

Step 10 – Go to the VM window
anka view hisierra
Step 11 – Go to system preferences and then go to Security and Privacy and allow osxfuse extension by ‘Benjamin Fleischer’. We use third party osxfuse for shared FS feature.



Step 12 – Stop and restart the VM.

Step 13 – Verify the setup. From your mac/host, execute the following command.
anka run hisierra ls -l
If it displays the contents of your current mac folder, then your VM is fully configured with all Anka PV drivers, guest add-ons and ready to use.

Anka Public Beta

As the WWDC conference gets underway with exciting new updates for macOS, iOS developers, we are thrilled to announce the availability of the beta version of Anka. For the past few months, we have been working with private beta users of Anka and are extremely grateful for all the feedback we received, which helped us shape Anka.

Anka is now available as ‘Anka Run’ and ‘Anka Build’ beta products. Anka Run can be used by iOS and macOS developers to create and consume reproducible development environments on their dev machines. Anka Run will enable developers working on the same project, to build and test their code changes against the same set of external libraries, configuration, toolchain and environment variables. This will significantly increase developer productivity. Check out technology page for additional technical details.

Anka Build can be used by infrastructure admins to very quickly build a private cloud on Apple hardware and provision on-demand instances of their environments from preconfigured VM templates. Anka Build is an all-in-one solution to virtualize macOS, manage macOS VM templates and provision them on-demand for CI jobs. It removes the setup complexities associated with existing approaches while providing a fast-performing and responsive macOS private cloud. Check out technology page for additional details. Anka Run and Anka Build, when used together can significantly accelerate the velocity of macOS and iOS development.

How to Get Started with Anka

UPDATE – High Sierra macOS 10.13 beta is out and Anka supports it. Download developer beta version ‘.app’ and work with it inside an Anka VM. Here are the setup instructions.

There are multiple ways to get started with Anka. Start with first downloading and installing Anka Run beta. Download and install the Anka Run application package on a few developer machines. Setup Anka registry in a docker container using the dockerfile included in the Anka-registry package. Then, create macOS Anka VM on one of the mac machines running Anka Run application and configure that VM with your project dependencies. Push that VM to Anka registry. Then, ask your developers to pull it on their machines(running Anka Run) and use them for local build and tests.

Then, move on to configuring a private macOS cloud for your CI. Download and install the Anka Build application package on your macOS CI machines. Setup Anka Controller in a docker container using the dockerfile included in the Anka-controller package. Register Anka Build nodes to Anka Controller cloud, configure the controller to talk to the previously installed Anka registry and you have a private macOS build and test cloud!

Additionally, if you use Jenkins for your CI, then install the Anka Jenkins plugin on your Jenkins master server to run your jobs on Anka cloud. Send us your feedback and questions at ankabeta@veertu.com, share insights on your setup with others at @veertu_labs, and help us spread the word.