Posts

Build cache for faster iOS builds with Jenkins CI and on-demand macOS VM slaves

Most of the existing iOS CI environments are configured to run the build and tests jobs directly on the Mac hardware. In such installations with large projects, some data like XCode derived data, and code repository get cached on the hardware and subsequent builds use these caches. In this blog, we will describe how you can set up your Anka Build macOS cloud in a Jenkins CI environment to build cache inside the VM and use them for on-demand provisioned Jenkins iOS CI slaves. This will provide the advantages of isolated, reproducible environments and build caches for faster build times.

We have implemented a Jenkins plugin called “Anka Slave template Prepare” plugin. This plugin enables devops to load and update build cache on existing jobing Anka macOS VM templates in an automated manner. Devops can then use the Anka Build Cloud Jenkins plugin and setup their primary iOS CI Jenkins jobs to always run using the VM templates pre-loaded with caches.

Visit https://ankadoc.bitbucket.io/using-jenkins/#using-the-anka-slave-template-prepare-plugin for detailed documentation to configure the plugin.

Anka Slave Template Prepare Jenkins Plugin



Post Build Action Configuration

iOS CI infrastructure – macOS cloud on Mac Pros with Anka Build

In our previous blog post, we discussed setting up macOS cloud for iOS CI on a mac mini cluster. Let’s now look at creating macOS cloud on mac pros for scalable and agile iOS CI. While the CPU on mac minis are much faster(as proven in various benchmarks) than existing Mac Pro models and result in a much high performing macOS cloud for iOS CI, there is still a great deal of interest in setting up iOS CI infrastructure on Mac Pros. Let’s look at how you can use Anka Build technology to very quickly set up and efficiently manage a scalable and agile macOS cloud on Mac Pros. One of the cool things about Anka Build virtualization technology is that it’s built on top of macOS Hypervisor framework and supported on all the Mac hardware. You can configure your iOS CI infrastructure on Mac minis, Mac Pro, iMac Pro, MBP or a combination of these. Download 30 Day trial at www.veertu.com.

Installing and configuring macOS cloud on Mac Pro cluster

  • Anka Build.pkg – This is a mac application package and installed on each of the Mac Pros.
  • Anka Build Controller and Anka Build Registry – This is a docker container package and installed in a VM or an AWS instance. If you set up your Mac Pros in Macstadium, you can set up and run controller and registry modules in a VM in Macstadium.


  • Hosted Anka Build Cloud can be secured entirely with VPN based connectivity between on-premise CI and hosted Anka Build controller. If you are using Macstadium, chat with their sales support to get more details on this.

    Step 1 – Install AnkaBuild.pkg on the Mac Pros. Check out Getting Started here.

    Step 2 – Install Anka Controller.

    Step 3 – Install and configure Anka Registry.

    Step 4 – Create macOS cloud by joining each Mac pro(running Anka Build) to the controller. Use ‘ankacluster join’ command.

    Step 5 – View your macOS cloud from controller management portal (this module is not available in the current version). We are providing a sneak peek into this controller portal through this blog. It will release in Anka version 1.2 upcoming release.

    Controller Anka Build macOS cloud dashboard showing overall health



    2 node Mac Pro macOS cluster



    Step 6 – SSH or VNC into one of the Mac Pros(running Anka Build). Let’s call this the builder mac. Create your macOS VM iOS build and test iOS environment (Anka VM Templates). Full command line support is available (‘anka create’) and you can fully automate this step (‘anka create’ & ‘anka run’). In this demo, we are creating a VM with 4vCPU, 8GB RAM and all dependencies to build and test the Kickstarter iOS project. You can change VM settings (CPU, RAM, etc.) with ‘anka modify’ command.

    macOS Hisierra Anka VM created on macOS Hisierra Mac Pro



    Step 7 – Connect the builder Mac Pro to Anka Registry and push the prepared VM template to this central private macOS cloud registry. You can tag (version) and use it as a base VM to create new environments(with different tags) and store and manage them in the registry.

    Latest tagged version of Hisierraga VM template in the Registry



    Step 8 – If you use Jenkins for your CI, install the Anka Build Jenkins plugin. For any other CI system, use the controller REST APIs to integrate. If you use hosted CI services like Buildkite or Gitlab CI, contact us to get additional details on integration. Check out details on Jenkins plugin here.

    Step 9 – Check the health of all Anka Build components in the controller portal.

    Step 10 – Let’s do a test run. I will launch multiple macOS VMs on my 2 Mac Pro node macOS Anka Build cloud.

    Manually creating on-demand VMs



    On-demand launched Anka macOS Vms on Anka Build cloud



    Step 11 – Now, let’s create a real job queue in Jenkins to run on this multi-node Mac Pro Anka Build cloud. Jenkins Anka Build plugin instantly launches clean VMs on the Anka Build cloud, and when the build job finishes, the VM dies. Next job starts a fresh VM.

    Kickstarter iOS project jobs in Jenkins



    Automatically launched clean Anka VMs from Jenkins



    Jenkins job running in Anka VMs



    On-demand clean Anka VMs launched automatically for Kickstarter Jenkins Jobs



    Every iOS CI build or test job request instantly gets a clean, consistent macOS VM environment. Test new versions of Xcode, iOS SDKs by creating new tagged Anka VM from a base template, manage them in the registry and use them for your CI jobs. Scale up the capacity by easily adding additional Mac Pros to the existing Anka Build cloud cluster.

    Launching build/test environment for iOS CI jobs at AWS instance launch speeds

    In this blog, we will describe how teams can configure a private (on-prem or hosted) cloud and instantaneously launch on-demand, isolated build/test instances to execute their iOS CI jobs, while still meeting the tough demands of the speed of overall execution of the CI jobs.

    It’s no secret that speed of CI job execution directly impacts developer productivity and you see developer productivity teams making this one of their top priorities. There are multiple variables that impact the speed of execution of the CI jobs – provisioning time for the environment, seeding the environment for the CI job (config mgmt etc), performance in the environment and getting build/test artifacts in and out of the environment.

    If you look outside of macOS dependent development, most of the CI process is happening either in private or public clouds. In these setups, (taking the example of AWS) launching an instance on-demand is super easy, completely extensible and fast. Just start one with an AMI and you have an isolated Linux instance available. Containers make it even easier and more replicable. However, the story is quite different for macOS dev environments for CI.

    Existing virtualization approach to creating a macOS dev cloud on Mac hardware is based on the same solution available to virtualize Linux and Windows. However, since this technology is not optimized for a macOS cloud use case, it suffers from challenges of scalability and manageability and only supports specific mac hardware.

    In the Anka Build solution, our goal was to make sure that these challenges are addressed and that it provides the same level of agility and scalability as one would get from using AWS. So, Anka Build packages a virtualization software that can run on any mac hardware, is built on top of macOS native Hypervisor Framework, exposes an extensible CLI.

    Let’s look at the method of provisioning or launching macOS VMs on-demand in Anka Build.

    Step 1 – Create your macOS Anka VM (this will be your template) and seed it with all your build/test dependencies using Anka CLI. This will ensure that no time is spent on preparing the dependencies in the environment for when a job runs.

    Step 2 – Put this Anka VM (template) in an ‘Instant Start’ State. It’s called `suspend` state in Anka CLI. This will ensure that when your iOS CI job requests an environment, the macOS Anka VM will be available instantly for execution, without any boot time issues. It will also ensure that you always get the same, consistent instance for repeated execution of the same job.

    Demo Video for Instant starting Anka macOS Vm from a suspended state.

    Step 3 – Push the suspended Anka VM to Anka registry.  If you need different flavors of build/test instances for different types of CI jobs, then, create and push different versions of same Anka VM template to the registry and use them by referring to a specific tag in your jobs. You can also create additional Anka VM templates(for when you need different macOS version) and push them to the registry and use them for your CI jobs.

    Step 4 – Install Anka Build mac application on all your build and test mac hardware and join them to Anka Build controller.

    Step 5 – Integrate your CI system to Anka Build controller through one of our existing plugins(Jenkins) or through REST APIs. Anka Build controller will now launch one or multiple macOS VMs on-demand across your build and test mac hardware, based on your CI job requests and your jobs will execute instantaneously.

    Step 6 – Trash the macOS Anka instances or keep them for troubleshooting. Get consistent, isolated instances for the next job execution.

    Step 1 and Step 2 address the tough demands of developer productivity teams to provide faster provisioning time of the macOS build/test instances in a macOS cloud for CI and enables them to keep build and test queue moving along with increased speed, efficiency, and determinism.

    Anka Registry available in a Mac Application Package – Store and manage Anka VMs in macOS cloud for CI

    Anka Registry enables management and distribution of macOS development environments (macOS VMs) across developers and to macOS cloud for CI, for consistent and repeatable build and test operations. With Anka Registry, you can create macOS image and then in an incremental manner provision Xcode, final dependencies and push to the registry, from where it is distributed on-demand to create build/test instances to execute CI jobs.

    Furthermore, you can pull a specific version of your build/test image, make changes, clone it to a new image and start managing it again with Anka Registry.

    When you install Anka Build to set up a macOS build/test cloud, you also set up Anka Registry, which runs as Ubuntu service in a docker container or in a Linux VM. With today’s release of Anka Registry Mac application package, we are making it easy for DevOps and iOS developers to quickly set it up on their Mac machines for testing purposes.



    You can very quickly get started with Anka, by installing Anka Flow or Anka Build package on your Mac and then install the Anka Registry. Use `anka create` command line to create your macOS dev image and push it to the Anka Registry and start managing and distributing it amongst team members.

    Let us know your thoughts on how we can improve it further.

    Configuring Jenkins to scale iOS CI on macOS cloud with on-demand slaves

    This blog is not a comparison of CI tools or services available. It assumes that Jenkins is your CI tool of choice for iOS CI/CD. There have been a lot of blogs written on how to configure Jenkins for iOS CI – starting with plugin installation, using Fastlane and other tools etc to further streamline the process. The last step of this workflow is the slaves on which the Jenkins iOS jobs run. In all the setups I have read about, the last step executes on macOS hardware configured as static slaves/nodes. Some of the common challenges with this static slave setup, that we have discovered through our conversation with the users are as follows.

    Lack of consistent and isolated slave environments for all the jobs

    Almost all iOS build/test jobs irrespective of differences in application, install multiple dependencies(internal or from public repositories). Multiple runs of these jobs pretty much require re-install or fresh install of these dependencies on the Jenkins slave mac node. Often times, the public dependencies which are directly git cloned from public repositories change between different runs of the same job, resulting in build/test failures. Multiple installs, uninstall of the dependencies can leave the Jenkins slave node in a polluted state with leakage between different job runs.

    Difficulty with reproducible slave environments

    When CI build/test fail due to environment-related issues (for example wrong version of a specific library) the CI queue starts to build up rapidly and troubleshooting options become a great challenge. Devops/Developers can’t spend too much time troubleshooting in slave environments as it impacts the overall available slave capacity. The focus then is to quickly patch/fix and get the CI queue moving again. There is no easy way to have reproducible environments to assist in these scenarios.

    No flexibility to iterate changes in the environment

    In a static slave node Jenkins CI setup, most often, when you have to test the CI workflow on new releases of macOS and Xcode, you have to shard the available static node capacity.

    These challenges can be addressed if it was possible to spin up/down macOS slaves on-demand on a pool of hardware, similar to the way you can spin up/down instances on AWS from the AMIs. However, nothing like this exists when your job needs to run on macOS and not on Linux or Windows.

    At Veertu, we built a Jenkins plugin for our Anka macOS cloud technology. The way it works is first, you configure macOS cloud on a cluster of Mac hardware (hosted or on-premise) with Anka Build and create your iOS build/test VM templates. Then, install Anka cloud Jenkins plugin from the Jenkins plugin center. Once the plugin is installed, you can define your slave template definitions and associate them to build/test iOS CI pipelines or jobs. Now, when your jobs execute, they will on-demand spin up a required instances of specific macOS VMs(as defined by the label associated with the job) environments on the Mac hardware cluster to act as your Jenkins slave instances.

    Below steps describe this in more detail.

    Step 1

    Install Jenkins – Anka plugin (current version is 1.6). This plugin code is also available on GitHub at https://github.com/veertuinc/jenkins-anka-ci.

    Anka Jenkins plugin

    Anka Jenkins plugin



    Step 2

    Point the plugin to connect with the Anka Build macOS cloud. You will need IP address of the Anka Build controller https://veertu.com/anka-technology/. Think of Anka Build controller as a central management server for your macOS cloud built on top of Anka. It exposes a rich set of REST APIs for integration with other third party CI systems https://ankadoc.bitbucket.io/using-controller/#controller-rest-apis.

    Anka jenkins plugin

    Anka jenkins plugin setup 2



    Step 3

    Define one or many slave template definitions for your iOS build/test jobs.

    Anka Jenkins plugin setup 3

    Anka Jenkins plugin setup 3



    Template field in the plugin shows all the macOS VM templates that you have created. Think of these templates as AWS AMIs. These templates are stored in a central Anka Registry storage component. You can pull these VM templates on your machine, make changes to them(install new dependencies) and push them back to the Anka Registry with a new tag/version.

    Anka Jenkins plugin setup 4

    Anka Jenkins plugin setup 4



    All the templates stored in the Anka Registry and their versions are available for use as slave templates in the Jenkins plugin. If you don’t select a version, the plugin will use the latest version.
    Anka Jenkins plugin setup 4.1

    Anka Jenkins plugin setup 4.1



    You can now store and manage reproducible environments for iOS CI jobs.

    Step 4

    Assign slave template label to your jobs/pipelines.

    Anka Jenkins Plug Step 4

    Anka Jenkins Plug Step 4



    Step 5

    Now, execute one or multiple concurrent jobs. Jenkins Anka plugin will start the required number of Jenkins slave macOS VMs of the type defined in the slave template definition. Every job executes in a consistent and completely isolated environment.

    The central management server aka Anka Controller manages the request queue for slave instances and provisions instances on available mac hardware nodes in your Anka Build cluster. So, you can run concurrent jobs in Sierra, HiSierra and also different versions of Xcode on the same pool of hardware.

    Anka Jenkins Plugin step 5

    Anka Jenkins Plugin step 5



    Let us know your thoughts and ideas on how we can further enhance the Jenkins-Anka plugin. You can also try it out with Anka Build cloud for iOS CI with a 30 day trial of Anka Build.

    Anka Beta new version 1 Build 41 update – For self service on-premise macOS VMs

    Today, we are announcing the availability of Anka Beta version 1 build 41 to all the beta users. Please check your email for the download link. New users can sign up at www.veertu.com. In this update, we have added some pretty cool feature updates to Anka macOS VMs configuration, that are directly relevant to use cases for enabling DevOps to setup a macOS on-premise CI infrastructure to test their enterprise wide deployment scripts, upgrade processes etc for Mac workstations. These features were implemented based on beta user feedback.

    Check out the version 1 build 41 release notes for more details.

    Let’s take an example: You are an enterprise with a large deployment mac hardware(laptops etc) being used by the employees. The portfolio of Mac hardware being used can be a mix of different models. Now, you have a set of bootstrap actions that need to take place on every employee mac or a set of macs. For example, applications deployed, permissions etc. Before rolling out any new changes to your bootstrap process, you want to test it. Because once it’s rolled out to thousand of employees, the cat is out of the bag and it can be challenging to roll back the changes if they don’t work as designed.

    In this case, you can take your cluster of Mac test hardware, deploy Anka Build on it to transform into a self-service macOS cloud. Now, use this infrastructure to do CI testing of your employee mac machine bootstrap process’. Spin up reproducible, sterile Anka macOS VMs on-demand and simulate testing on different models by modifying hw.product, hw.family, hw.uuid etc. parameters for the Anka VMs.

                          

    Anka Build Terminology – Components to setup on-premise iOS CI infrastructure

    Anka Build is macOS native virtualization platform from Veertu. It’s a complete platform to implement and manage macOS VMs on a large scale for iOS CI. Anka Build includes lightweight Anka client, which is the native hypervisor to run macOS VMs on Mac hardware, the Controller, which is used to on-demand provision and manage macOS VMs and the Registry, which is used to store and distribute macOS VM templates for CI.

    While Anka Build manages macOS VMs on a large scale for CI, all Anka Build modules by themselves are very lightweight, easy to deploy and manage.

    iOS Ci with Anka Build

    Anka Client

    Anka client is a very lightweight mac application. Unlike, other server virtualization solutions for mac hardware, Anka Build doesn’t strip away the core macOS. Anka Build runs as another mac application on the mac hardware and enables the creation and running of macOS VMs. Anka Build macOS VMs are very responsive and perform at near native speed. These Anka VMs are used as on-demand build and test environments for iOS/macOS CI.

    Anka Controller

    Anka Controller module is packaged as a docker container. The Controller provides a single interface through its REST APIs to on-demand provision macOS VMs, requested by CI jobs. Think of it as central management for your Anka Build Cloud.

    Anka Registry

    Anka Registry module is used to store and distribute different versions of macOS VMs. Anka Registry is also packaged in a Docker container. A iOS/macOS CI job can request X quantity of N type of macOS VM. When Controller receives this request, it pulls N macOS VM (version tag) from the registry and spins up X quantity of these VMs using Anka Client on your Anka nodes. Anka Controller also maintains a queue of all incoming requests from the CI system and manages them.

    Anka Node

    Anka node is one or many Mac hardware on which you want to run your iOS CI jobs. You install Anka Build client on each of your Anka Build Nodes and then connect the nodes to the Anka Controller.

    Connecting Anka Build to your CI system

    While Anka Build platform consists of multiple modules, you only need to integrate your CI system to Anka Build Controller in order to start using Anka Build for executing iOS CI jobs. Anka Build Controller exposes REST APIs, through which you can integrate your existing CI system.

    For Jenkins based CI implementation, there is a Jenkins plugin available, which can be used to very easily set up integration with Anka Build iOS cloud infrastructure.

    Performance and Scalability in virtualized iOS build and test infrastructure

    Growth is good, and when development teams grow, often, they find themselves having to choose between performance and scalability factors related to their development process. Most often, iOS application development project start by consuming hosted service offerings for their build and test processes. Then, as growth happens, team size increases, the velocity of code commits increase significantly, the number of projects increase, and soon performance issues like long queue wait times and longer build and test times start to creep up.

    In most scenarios, one can throw more resources at this problem to address the issue. However, in the macOS world, that’s not the case. While scalable and high-performing platforms like AWS, Google Cloud, Azure which are powered by virtualization exist for Linux and Windows, nothing similar exists for macOS. All existing approaches to virtualization of macOS suffer from a tradeoff between scalability and performance.

    Veertu Anka hypervisor eliminates the need for the user to make this tradeoff. Anka Build, powered by Anka hypervisor, enables growing iOS development teams to very easily, setup and configure a scalable and high performing AWS like private cloud for iOS build and test.

    There is a lot of skepticism around the performance of iOS simulator test jobs on virtualized infrastructure, especially as the test portfolio expands. Anka Build addresses this for its users with a combination of three variables. – The performance of Anka VM, Time to provision an on-demand VM and VMs in Anka registry with pre-boot iOS simulator.

    Anka VMs run on top of Anka Hypervisor, which is a new lightweight virtualization technology that leverages the macOS Hypervisor.framework to run macOS VMs in the most resource efficient manner. Instead of directly requesting and managing macOS resources like CPU or memory for the VMs, Anka Hypervisor, through the macOS Hypervisor.framework, lets the operating system manage the VM resource needs in the same way it runs other applications. As a result, Anka VMs perform much faster and are more responsive. Anka VMs are also packaged with a para-virtual block and network drivers, to enable faster execution of build and test jobs.

    Time to provision or the time it takes to have an environment ready for the iOS build and test job is a significant factor that impacts overall performance. Anka Build addresses this with ‘Instant Start’ feature. ‘Instant Start’ offers on-demand provisioned instances instantaneously (1-2 sec) to execute the iOS Ci jobs.

    iOS simulator UI test job performance on dedicated or virtualized hardware is impacted by, the long boot time of iOS Simulator. With Anka Build, DevOps can address this problem by pre-booting the simulator in the VM for each device type and pushing the VM to Anka registry. Then, CI UI test jobs, through Anka controller can on-demand provision instances from these VMs and use the pre-booted iOS simulator at full throttle. Each test is run in a clean, sterilized environment with pre-booted simulator of a particular device type.

    In our internal tests, we have recorded that iOS simulator UI tests performed with Anka Build and pre-booted simulator VMs take less time than when running on dedicated hardware.

                         

    iOS simulator test in Anka VM



            

    iOS simulator test on dedicated hardware

    Building a private iOS Device Cloud

    Now that you have Anka Build up and running for your iOS CI infrastructure, your developers are more productive and your environment is easier to manage. But how do you test your iOS app releases on multiple iOS devices to make sure they are rendering properly?

    Anka Build enables you to execute tests on real devices, in the same way as your simulator-based test workflow. Cable your physical devices to your Anka Build macOS private cloud nodes, then connect them at run time to on-demand macOS VMs and run your tests. Using Anka, you can manage a single device or a group of similar devices through a simple command line interface.

    What are the benefits of this approach?
    1. One single scalable iOS infrastructure for build, simulator based tests and real device tests.
    2. Eliminates the need to separately manage mac hardware connected to real devices for CI.
    3. Management of real devices through a simple, command line interface.


    Check out the following example of making a real device available on-demand for testing:

    Step 1 – Create a VM to be used for real device testing

    Step 2 –  Let’s say you need to run multiple tests on multiple ipad models. Create a USB-group called ipadgrp:

    `anka usb-group create ipadgrp`

    Step 3 – Add all of your ipad models to this group:

    `anka usb-group add-device <device identifier1> ipadgrp`

    `anka usb-group add-device <device identifier 2> ipadgrp`

    Step 4 – Attach this usb-group to the previously created VM:

    `anka modify VM add sub-group ipadgrp`

    Step 5 – Configure your specific CI job to use this VM for real device testing.

    Step 6 – Use Anka Controller to spin up on-demand instances of this VM. Each VM instance at run time will claim an available ipad device from the `ipadgrp` and make it available for testing.

    Additional documentation on this feature is available at using real devices with Anka Build.

    Anka Public Beta

    As the WWDC conference gets underway with exciting new updates for macOS, iOS developers, we are thrilled to announce the availability of the beta version of Anka. For the past few months, we have been working with private beta users of Anka and are extremely grateful for all the feedback we received, which helped us shape Anka.

    Anka is now available as ‘Anka Run’ and ‘Anka Build’ beta products. Anka Run can be used by iOS and macOS developers to create and consume reproducible development environments on their dev machines. Anka Run will enable developers working on the same project, to build and test their code changes against the same set of external libraries, configuration, toolchain and environment variables. This will significantly increase developer productivity. Check out technology page for additional technical details.

    Anka Build can be used by infrastructure admins to very quickly build a private cloud on Apple hardware and provision on-demand instances of their environments from preconfigured VM templates. Anka Build is an all-in-one solution to virtualize macOS, manage macOS VM templates and provision them on-demand for CI jobs. It removes the setup complexities associated with existing approaches while providing a fast-performing and responsive macOS private cloud. Check out technology page for additional details. Anka Run and Anka Build, when used together can significantly accelerate the velocity of macOS and iOS development.

    How to Get Started with Anka

    UPDATE – High Sierra macOS 10.13 beta is out and Anka supports it. Download developer beta version ‘.app’ and work with it inside an Anka VM. Here are the setup instructions.

    There are multiple ways to get started with Anka. Start with first downloading and installing Anka Run beta. Download and install the Anka Run application package on a few developer machines. Setup Anka registry in a docker container using the dockerfile included in the Anka-registry package. Then, create macOS Anka VM on one of the mac machines running Anka Run application and configure that VM with your project dependencies. Push that VM to Anka registry. Then, ask your developers to pull it on their machines(running Anka Run) and use them for local build and tests.

    Then, move on to configuring a private macOS cloud for your CI. Download and install the Anka Build application package on your macOS CI machines. Setup Anka Controller in a docker container using the dockerfile included in the Anka-controller package. Register Anka Build nodes to Anka Controller cloud, configure the controller to talk to the previously installed Anka registry and you have a private macOS build and test cloud!

    Additionally, if you use Jenkins for your CI, then install the Anka Jenkins plugin on your Jenkins master server to run your jobs on Anka cloud. Send us your feedback and questions at ankabeta@veertu.com, share insights on your setup with others at @veertu_labs, and help us spread the word.