Uncovering OS X Hypervisor.Framework – Part 1

We mentioned in our previous blog that we had been working with hypervisors and most recently with Apple Hypervisor.Framework since it’s release last year. We have used Hypervisor.Framework to develop light weight, high performance and highly reliable virtualization for OS X. With the recent attention and focus on this framework, we thought it would be good to share more detailed information on this with our users. First, a little bit of history on virtualization for Mac. While the hardware of a Mac is produced solely by Apple, it does use certain standard components like Intel x86-64 processors. x86 processor architecture uses the concept of privilege levels (or privilege rings) for machine instructions. The most privileged operations, which are reserved for the host operating system, have a privilege level of 0. A virtual system running on top of the host can’t access the 0 privilege level directly. So, Virtualization technology(VT) extension to CPU, memory and I/O hardware were developed as a technique to overcome this challenge. There are also software based approaches to address this – binary translation and paravirtualization. But, they have some challenges around ring aliasing, address-space compression, and non-faulting access to privileged processor state. Intel’s VT-x virtualization extensions, VT-i extensions and AMD’s AMD-V technology were released to enable hardware assist virtualization. Now, in order to use the VT extensions, hardware-assisted virtualization engines (hypervisor) were developed as kernel extensions(KEXTs) on the Mac. In this model, the hypervisor has complete control over hardware, memory and operates in VMX root mode. It prepares a special Virtual Machine Control Structure (VMCS), which sets the state of the guest. The VMCS stores copies of guest system registers and also includes control bits that specify conditions under which the host will trap (exit) into the hypervisor. Once the VMCS is ready, the hypervisor executes VMLAUNCH and lets the OS take control. The OS runs in VMX non-root mode (guest mode). When any of the exit conditions (privileged instructions) that were specified in the VMCS are met, the processor VM Exits into the hypervisor mode. After execution of the privileged instruction, the hypervisor executes VM Entry instructions, returning to the VMX non-root mode. Because, these hardware assist hypervisors are implemented as kernel extensions for Mac, there are some known issues identified with this approach. Any bug in KEXTs (kernel extensions) can have severe consequence than bugs in user-level code and could cause a kernel-panic, crashing the operating system. In some implementations, KEXTs close interrupts, so, which can have a negative impact on the responsiveness of the system. Kernel resources are wired(permanently resident in memory) and thus are more costly to use than resources in user space task of equivalent functionality. There are also no safeguards in place inside the kernel. For example, a memory leak inside a kernel will result in loss of this memory until the next reboot of the system. With OS X v10.10, Apple introduced the Hypervisor.Framework, which allows virtualization to be implemented on top of OS X, without needing to deploy third-party kernel extensions (KEXTs). It exposes Intel-VT extensions through APIs, to be run in the user space and eliminates the need to run third party developed virtualization code directly in the OS X kernel. At the command execution level it looks something like this. Hypervisor.Framework guest execution: Userspace:
  1. Getting VMCS ready – Veertu code
  2. Execute hv_vcpu_run() – Veertu code
Kernel space:
  1. Hypervisor framework validates VMCS – Apple code
  2. Hypervisor framework execute VMLAUNCH – Apple code
  3. Hypervisor framework wait for VMEXIT – Apple code
  1. Analyze VMEXIT and get back to step 1 – Veertu code
Now, this approach addresses most of the KEXT based implementations issues. A large portion of OS X virtualization code (dealing with VMEXIT etc.) executes in the user space. If there is a bug, it only impacts this application and the application can be closed without impacting other apps or the OS. Any memory leak type situations are handled by the OS X kernel in a safe manner. The decision to close system interrupts is done natively by Apple as per the OS X scheduling policy. What we have shared with you in this blog are additional details based on our experience working with this Hypervisor.Framework. Veertu implementation has lot more details and is much more extensive with guest add-ons etc. In the upcoming months, we plan to share additional details with you on our implementation. Do share your feedback with us.

The March towards adoption of OS X Hypervisor.Framework for Native Virtualization capabilities

Today, Docker announced beta availability of Docker for Mac, built on top of xhyve, one of the derivatives of OS X Hypervisor.Framework. This demonstrates two things – OS X recently released Hypervisor.Framework is a better architecture for developing virtualization on top of OS X, as opposed to installing layers of virtualization products. As a quick reminder, the docker machine was based on VirtualBox and required users to set VirtualBox on their Macs to be able to use the Docker ToolBox. A quick google search will reveal that this approach had a lot of stability and reliability issues. The second point, this announcement reveals is the growing importance of mac as the development desktop for developers. We, at Veertu, have been at work using the OS X Hypervisor.Framework for more than nine months. From the very beginning, we saw the advantages of using this to provide a native virtualization experience on mac. It’s great to see others follow the same path, and all put together will lead to more rapid awareness and adoption of native virtualization for OS X as well as Apple adding more useful features like bridge networking and USB support to the Hypervisor.Framework.

Developers, their desktop and dev environments in 2016

I recently came across this very interesting survey (http://stackoverflow.com/research/developer-survey-2016) done by stack overflow( a site, I absolutely love for researching) on their developer/programmer community. While the survey has insights on trends on dev languages, technology stack etc., one of the highlighted trend was the increasing adoption of Mac OS X as the desktop OS of choice for developers. The survey indicated that Mac OS was overtaking linux as the number 2 operating system among developers. And, that the windows share of desktop OS was decreasing.
This got me thinking, because all around me, Linux seems to the platform of choice for development amongst developers. So, what’s really happening behind this trend? There are some factors which are driving the developers to adopt Mac OS X as their desktop OS, however it seems like, they are most likely still developing on Linux on their Macs. A long time windows desktop user, I recently made the switch to a Mac and I am extremely happy about it. What got me was the simplicity and the ease of use – it just feels natural and light. Of course, these are all user experience descriptions, but, If I think more on it, it’s really about the way the apps that I use on Mac behave. They feel like they are part of the OS, well integrated into it. They feel Native to the Mac.
And, that’s what got me thinking more about the user feedback we have been getting about Veertu. Up until now, we have been looking through a different set of lenses at Veertu – the kind that compels you to think in terms of features. So, I recently sat down with one of founders and had him draw me the most simple, architecture representation for Veertu virtualization on OS X. 
Veertu operates just like any other Mac OS X app. It’s not a heavy virtualization platform which runs on top of Mac OS X and changes the desktop experience for the developers. Instead, it’s an app, which developers can use to run linux, windows dev environments on their Mac. So, like other Mac apps, it lets the OS X do what it does best – Scheduling of resources and power management, without interfering or over riding it. It’s completely sandboxed. Why? Because, like other native Mac Apps, Veertu executes at a higher user privilege with no access to kernel level functions. This has some very unique advantages – The dev VM environments running on top of Veertu are completely sandboxed, operating in a capsule. They don’t inadvertently expose security holes in your Mac. And, recently, I heard from one of the users, how he is able to take advantage of this to run multiple VMs for malware testing. 
Like other apps running on top of your Mac, Veertu is executing in user space in it’s won capsule,  so if it crashes (Why not…it’s software and every software has bugs!), it doesn’t impact other running apps or processes.
So, It was starting to become clear to me that with these new set of lenses, I was looking at native virtualization for Mac OS X. That, Veertu is an app for developers who use Mac OS X as their desktop, using which, they can run dev environments of all flavors. Aas we continue to make Veertu more rich in features for our users, ‘ Native Virtualization’ will always be the overriding theme for us and we will continue to stay true to it. Do share your views and thoughts on this.