Hyper-V Shared Nothing Live Migration In Windows Server 2012– VM Mobility Rules

I see and hear some people shrug at the idea of Shared Nothing Live Migration, dismissing it as marginally useful. Some do state they’ll have it as well but that it’s not that valuable. Well I disagree totally. A lot of the time these remarks are due to a lack of understanding about how several technologies in the Microsoft stack work together. Combine this with tunnel vision and the fear of some vendors and you get a lot of FUD.

I advise you to look beyond the virtualization stack, to the issues that people who are building infrastructure for dynamic, flexible and * cloud  data centers are dealing with.

Look, as “architects” we have to design & build for failure. We all know that it’s just a matter of time before things go BOINK.  So we build in redundancy, some of this within a silo, some of this is between silos. The two approaches compliment each other. What this gives you is options and everybody who knows me, especially those who work  with me has heard my mantras: “Assumptions are the mother of all F* Ups” and “Options, options, options”. Make sure you design & build in options. This way you can maneuver your self out of a bad situation. Don’t ever assume you’re out of options, especially not when you put some in the design on purpose Winking smile. It’s also very useful beyond that because a lot of you might agree with me that silos and fork lift, down time inducing upgrades, migrations, transitions or replacements are expensive and bad. This is where Share Nothing Live Migrations comes into play. You gain mobility over silos. That silo might be a server, a cluster, storage or mixtures of them all.

With Shared Nothing Live Migration we can migrate virtual machines between those silos with nothing more than a network cable.This is huge people. You are no longer trapped in that silo. In this context it provides you with all the options & flexibility mobility gives you. even it the technology itself is not about high availability.

Some very useful scenarios

Migrate virtual machines from an old cluster to a new cluster with out any down time

  1. Migrate virtual machines from stand alone hyper-V hosts to a fail over cluster with out any down time
  2. Migrate virtual machines from one stand alone host to another one for maintenance, again, without any down time
  3. Choose different types if storage & Hyper-V deployment depending in IOPS, redundancy, availability, manageability needs. With Shared Nothing Live Migration you can be confident  that  you can move your virtual machine from one environment to the other when needs change. This is breaking the storage silo boundaries open people! This is huge … think about it.

How it works

The details are for another post but basically is made possible by the combination of Live Storage Migration and Live Migration.

First the Storage is Live Migrated


After the Live Storage Migration is done the state of the virtual machines is copied and synchronized.


This Is Mobility

I hear the competition shrug.  It isn’t high availability. Well indeed no one who understands the feature ever said it was. It’s virtual machine mobility. Look at the scenarios above and you’ll see that this ability could very well be game changer in how we look at storage & design solutions.

Speed & Performance

What did we hear on this front: “it will be too slow to be really useful”. Really? Well let’s see:

  1. The world is converging to 10Gbps and after that 40Gbps and up will come
  2. NIC Teaming in box With Windows 2012 which can provide more bandwidth.
  3. SMB 3.0 Multichannel. This provides multiple channels per connection spreading the load over multiple CPUs
  4. SMB Direct, have you seen the speeds this achieves?

Before you state that this doesn’t work on Live Migration … as confirmed at TechEd 2012 Europe with Jose Baretto this does work when both the source AND the target is an SMB 3.0 share. This means yet another reason to use SMB 3.0 share for your Hyper-V storage needs! So unlike what Tad at vLimited keeps saying, unhindered by any knowledge, it is a very valuable feature and it can be extremely fast given the right connectivity and storage that can handle the IOPS. And no, the fact that it’s unbuffered doesn’t impact this to much. Test this by using xcopy/robocopy /J with a VHD over your infrastructure.


Even if you’re on a budget and cannot go for the RDMA NICs & SMB 3.0 you have several options to get very decent virtual machine mobility and not be stuck in a silo. And for those who want to leverage this feature to create and agile & mobile virtual environment you have some very nice technologies available to optimize to your needs & budgets.


Virtual Machine mobility and storage mobility are very interesting features that provide for a previously unknown flexibility. Windows Server 2012 makes us rethink our storage approaches (I sure am) and I’m very interested in seeing how this will evolve.

Very Educational Microsoft TechEd 2012

Hello from TechEd 2012 Europe at the RAI in Amsterdam. I’ve been extremely busy attending sessions, talking to Microsoft employees and vendor engineers. We’ve had some very interesting discussions and I learned a lot and clarified even more. TechEd has once more proven to be an excellent investment of time and I have been able to get a lot of face time with the right people. To me this is important because that helps me tremendously when designing solutions. Sorry for the low quality pictures.


Bob Combs on stage at the TechEd Europe 2012  educating us on NIC Teaming

You know my mantra “options, options options” as this is what gets you a way out of a pinch. However a lot of options also mean you need to make decisions, and not just when dealing with issues but also at design time. Knowledge and understanding is what help to make the correct of the best decision fast. Attending this conference with its tremendous networking opportunities provides for a very nice and effective setting for passionate discussions and deep dives into scenarios. Challenging vendors, interacting with peers, throwing ideas out there and deep diving into the possibilities and drawbacks with each other is great and helps a lot to understand technologies better. You have to thrown what you have learned out there and discuss it to test your understanding of the subject. Don’t be afraid to do so. We all don’t know things, get stuff wrong, etc. Don’t let  fear stop you from interacting with your peers.


Ben Armstrong in action on Live Storage Migration

It is also great to meet up with my community buddies from all over the word again and I feel privileged to have the opportunity to attend these conferences. For me personally these are priceless and the value to my employers/clients is considerable. There is a tsunami of new technology in the Windows Server 2012 stack and learning to put these into context is both fun and useful. These are very interesting times in the Microsoft Infrastructure ecosystem so life is good!

Answer to Brad at TechEd Europe 2012 Keynote: Pessimists & Tad Don’t like Windows Server 2012

Brad is on stage for the opening keynote asking if the glass is half full or half empty. Well it depends on where you are in the ecosystem. For us the glass is half full and filling up fast.

Some people nag me about the fact that Windows Server 2012 is so different and that it’s wrong to turn the world upside down. Yes, it is different and new in many ways.  There are also many improvements to features that already exist. There is a lot to learn and understand. Why are some people so pessimistic?

Ever since I got my hands on the BUILD Developer Preview bits I have personally invested a lot of my time in Windows Server 2012. With the beta that only increased. Why? Well, that’s the way forward, because that’s where the improvements are. We can’t do tomorrows jobs and meet tomorrows demands with yesterdays technology.


The picture above is basically the pessimists view of the world. Enjoy your cupper but I’m not joining you. Windows Server 2012 rocks and it’s going to do a whole lot for our industry and businesses. But wait a minute, I do understand why Tad is so pessimistic. But that’s about the future of vLimited and being stuck in the past. Listen Tad, you’d better empty that cup because this is where vLimited becomes history rather than write it.

Does that mean I’ll be throwing away Windows 2008 R2? Nope. I expect to deal a lot with it in the next few years but I’m not going to build future infrastructure on the previous version. I will introduce Windows Server 2012 where and when we benefit from it. For me that is from day one the bits RTM. The benefits are so overwhelming we’d hurt ourselves by not doing it. Your mileage may vary. But don’t get stuck in the past  Here’s a link to your escape pod: Microsoft Virtual Machine Converter Solution Accelerator I’m happy it’s here. That’s what people are asking me more and more about, how to move to Hyper-V.

But what’s with the negativism of some? Sure people are still running Windows Server 2000/2003. Sometimes for good reasons, often for (very) bad ones. Are some going to go through all this again with people clinging to Windows2008 R2? No doubt. Been there, seen it. Very predictable. Is Windows Server 2012 going to fail?  No way.  And what I’m seeing in Windows Server 2012 is great technology. Will it be perfect? No. I already have feature requests for vNext Smile. But this is pushing the ball forward, this is ambitious in the best sense of that word.  There will be bugs, there will be challenges and hiccups. That’s part of the business and the realities of life.  But look at all what’s available in there. Don’t just read some industry press articles. Did you test it your self already? Did you do any clustering? Tested all the new functionality in Hyper-V? The innovations in Live Migration options and networking? Looked at the amount of PowerShell support in there? Notice the improvements in Active Directory, DHCP and other core infrastructure services? Have you used Windows Server 2012 at all yet? You didn’t look at SMB 3.0 and all the storage improvements in there did you? Go talk to Jeff Woolsey, he’s passionate about it and for good reasons. Put in some effort, live a little, get out of your comfort zone and you’ll be going places. Don’t be a pessimist. Think positive or you’ll end up like Tad who was the joke of the party at MMS2012


Moving Clustered Virtual Machines to Windows Server 2012 with the Cluster Migration Wizard

As you might remember I did a blog post on transitioning from a Windows Server 2008 R2 Hyper-V cluster to Windows Server 2012 (well I was using the beta at the time, not the RC yet):

  1. Part 1 Upgrading Hyper-V Cluster Nodes to Windows Server 2012 (Beta) – Part 1
  2. Part 2 Upgrading Hyper-V Cluster Nodes to Windows Server 2012 (Beta) – Part 2
  3. Part 3 Upgrading Hyper-V Cluster Nodes to Windows Server 2012 (Beta) – Part 3

Microsoft has now blogged about the process themselves and they use the migration wizard in Failover Cluster Manager to get the job done where I did this using the Import, “register only” functionality.

This is the first step by step that describes the official way. You can read about the process here:

How to Move Highly Available (Clustered) VMs to Windows Server 2012 with the Cluster Migration Wizard

Multi Site SAN Storage & Windows Server 2012 Hyper-V Efforts Under Way

First some stats: 36 pallets of hardware handled over a period of 10 days. 29 of those over a period of 3 days. Most of it didn’t even exist at the beginning of the month, it was just an order. But DELL is a logistical force to be reckoned with. “Easy as DELL” is a reality, the speed at which they respond to request and orders is amazing. For quality/price balance, service, logistics, speed and support, it’s hard to beat them

A lot of people are used to dealing with slower processes and think SANs take at least 2 to 3 months to de delivered after ordering. This means they are caught of guard by this. I’m happy to say I’m not otherwise the data center would have been blocked by a tsunami of packaging material and hardware.

We’ve been busy unloading, unpacking, racking and partially cabling the new hardware coming in for a multi site SAN project. And let’s not forget the labeling. While we are far from finished, this good news. We’re finally busy working on the installation after the long time consuming process of procuring the equipment. That’s never an easy process, let alone a fast one. But I digress.

What are we working with?

  • Dell Compellent SANs (intra and inter site data protection / redundancy)
  • PowerVault MD3600 & MD1200 storage units for disk to disk backup capacity

Now to go from this


to this and beyond  …



Takes quite a while as you can imagine and we still have a ton of stuff to do Smile. I’ll be sharing my experiences and findings via this blog when I can.

My high level design  focuses on scale out to achieve both performance, flexibility and resiliency. We’ll build a modular scale up and scale out solution using commodity hardware and not in a mega redundant, ultra scalable single and very expensive storage solution. You can read more on my views about this subject here Some Thoughts Buying State Of The Art Storage Solutions Anno 2012.For the backup we are following the same approach. We cannot afford to pay the amounts of money that seems to be needed to buy high end backup appliances. We have plans to leverage Windows 2012 to help us achieve this but these are subjects for some other blog posts later.

Windows Server 2012 Hyper-V Supports IPsec Task Offloading

IPsec has been around for a while now. In an ever more security conscious & regulated world you want and/or are required to protect your network communication by
authenticating and encrypting the contents of at least some of your network traffic. Think about SOX and HIPPA and you’ll see that trade or government security requirements are not going anywhere but up for us all. This is not just restricted to military of intelligence organizations.

We’ve seen the ability to offload IPsec traffic to the NIC for a while now. This is great as the IPsec processing is a very CPU intensive workload. Unfortunately it didn’t work for virtual machines . Until now IPsec offloads was only available to host/parent workloads in using Windows Server 2008 R2. The virtualization of high volume network traffic workloads that require encryption means a serious hit on the resources on the host. If you’re willing to pay you might get by by throwing extra host & CPU power at the issue. But what if the load means a single virtual machine with 4 vCPUs can’t hack it? Game over. Sure Windows Server 2012 Hyper-V allows for 32 vCPUs now,  but that is very costly, so this is not a very cost effective solution. So in some cases this lead to those workloads being marked as “unsuited for virtualization”.

But with Windows Server 2012 Hyper-V we get a very welcome improvement, that is the fact that a virtual machine can now also offload the IPsec processing to the physical NIC on the host. That frees up a lot of CPU cycles to perform more application-level work, resulting in better virtualization densities, which means less costs etc.

Let’s take a look where you can set this in the Hyper-V GUI where you’ll find it under the network adaptor /Hardware Acceleration.


IPsec offload is also managed by the Hyper-V switch, this controls whether the offloading will be active or not. This is to prevent that the IPsec offload stopping the services if insufficient resources are available. Please do note that IPsec when required in the guest will be done anyway creating an extra CPU burden. So this does not disable IPsec, just the offloading of it. On top of this and in the gravest extreme you can guarantee that IPsec servers can get the resources they need by sacrificing less important guest if needed. by using virtual machine prioritization. The fact that you can configure the number of security associations helps balancing the needs of multiple virtual machines requiring IPsec offload.

To conclude, this wouldn’t be Windows Server 2012 if you couldn’t do all this with PowerShell. Take a look at  Set-VMNetworkAdapter and notice the following parameter:


This specifies the maximum number of security associations that can be offloaded to the physical network adapter that is bound to the virtual switch and that supports IPSec Task Offload. The thing to notice here is that specify a zero value is used to disable the IPsec Offload feature.


Microsoft Private Cloud Computing–The Book

I’d like to mention the launch today of a new book Microsoft Private Cloud Computing written by a group of experts in this subject.


Take a look at the authors below. You probably know them from the community efforts they put in and might have read previous materials from them or heard them present.They bring a varied background and a mix of experiences and knowledge to the table needed to tackle any aspect of the private cloud and to a write complete book on the subject: hardware (servers, networking, storage), operating systems, hypervisors and the software stack.

Knowing these people and how busy they are this quite an achievement requiring a serious effort.  The technical reviewing was done by Kristian Nese (Cloud and Datacenter Management MVP,@KristianNese), who’s has been working on private clouds from day one.

Well done guys and thank you!

Windows Server 2012 Cluster Reset Recent Events Feature

There are various small improvements in Windows Server 2012 Failover Clustering that make live a little easier. When playing in the lab one of the things I like to do is break stuff. You know, like pull out the power plug  of a host during a live migration or remove a network cable  for one or more of the networks, flip the power of the switch off and on again, crash the vmms.exe process and other really bad things …Smile Just getting a feel for what happens and how Windows 2012 & Hyper-V responds.

As you can imagine this fills up the cluster event logs real fast. It also informs you in that you’ve had issues in the past 24 hours. Those recent cluster events could not be cleared or set to “acknowledged” up to Windows 2008 R2 except by deleting the log files. Now this has to be done on all nodes and is something you should not do in production and is probably even prohibited. There are environments where this is indeed a “resume generating” action. But it’s annoying that you can leave a client with a healthy looking environment after you have fixed an issue.


For the lab or environments where event log auditing is a no issue I used to run a little script that would clear the event logs of the lab cluster nodes not to be dealing with to much noise between tests or to leave a GUI that represents the healthy state of the cluster for the customer.

This has become a lot easier and better in Windows Server 2012 we now have a feature for this build in to the Failover Cluster Manager GUI. Just right click the cluster events and select “Reset Recent Events”.



The good thing is this ignores the recent events before “now” but it does not clear the event log. You can configure the query to show older events again. This is nice during testing in the lab. Even in a production environment where this is a big no-no, you can’t do this you can now get rid of noise from previous issues,focus on the problem you working on or leave the scene with a clean state after fixing an issue without upsetting any auditors.


Configuring Jumbo Frames with PowerShell in Windows Server 2012

During lab and test time with Windows Server 2012 Hyper-V some experimenting with PowerShell is needed to try and automate actions and settings. One of the thing we have been playing around with was how to enable and configure jumbo frames.

Many advanced features like Large Send Offload have commandlets of their own (Enable-NetAdapterLso etc.), but not all them and jumbo frames is one of the latter. For those advanced features you can use the NetAdapterAdvancedProperty commandlets (Network Adapter Cmdlets in Windows PowerShell). You can than set/enable those features via the registry keywords & values. Let’s say we want to enable jumbo frames on a virtual  adapter named “ISCSI” in a VM.


To know what values to use you can run:

Get-NetAdapterAdvancedProperty -Name ISCSI


As you can see Jumbo Packet has a RegistryValue of 1514 and a DisplayValue  of “Disabled”. You can also see that the RegistryKeyword to use to enable and configure jumbo frames is “*JumboPacket”. So to enable jumbo frames you run the following command:

Set-NetAdapterAdvancedProperty -Name “ISCSI” -RegistryKeyword “*JumboPacket” -Registryvalue 9014


The RegistryValue is set to 9014 and the DisplayValue is set to “9014 Bytes”, i.e. it’s enabled.

If you type in an disallowed value it will list the accepted values. Please note also that these can differ from NIC to NIC depending on what is supported. Some will only show 1514, 4088, some will show 1514, 4088, 9014.


Now to disable jumbo frames you just need to reset the RegistryValue back to 1514

Set-NetAdapterAdvancedProperty -Name “ISCSI” -RegistryKeyword “*JumboPacket” -Registryvalue 1514

The result of this command can be seen in the picture below. DisplayName Jumbo Packet has a DisplayValue of “Disabled” again.


Let’s say you want to enable jumbo frames on all network adapters in a host you can run this:

Get-NetAdapterAdvancedProperty -DisplayName “Jumbo Packet” | Set-NetAdapterAdvancedProperty –RegistryValue “9014

Or run

Set-NetAdapterAdvancedProperty -Name * -RegistryKeyword “*JumboPacket” -Registryvalue 9014

I didn’t notice much difference in speed testing this with measure-command.

If you mess things up to much and you want to return all DisplayName settings to a well known status, i.e. the defaults you can run:

Reset-NetAdapterAdvancedProperty –Name SCSCI –DisplayName *

If you’ve just messed around with the jumbo frame settings run

Reset-NetAdapterAdvancedProperty -Name ISCSI –DisplayName “Jumbo Packet”

Or you can do the same for all network adapters:

Reset-NetAdapterAdvancedProperty –Name * –DisplayName “Jumbo Packet”

There you go, you’re well on your way doing the more advanced configurations of your network setup. Enjoy!

Transition a Windows Server 8 to Windows 2012 Release Candidate Hyper-V Cluster

For those of you interested in moving their lab from Windows Server 8 beta to Windows Server 2012 Release Candidate I can refer you to my 3 part blog series on Upgrading Hyper-V Cluster Nodes to Windows Server 2012 (Beta).

  1. Part 1 Upgrading Hyper-V Cluster Nodes to Windows Server 2012 (Beta) – Part 1
  2. Part 2 Upgrading Hyper-V Cluster Nodes to Windows Server 2012 (Beta) – Part 2
  3. Part 3 Upgrading Hyper-V Cluster Nodes to Windows 8 (Beta) – Part 3

So the entire process is very similar but for the fact that to go from Windows Server 8 Beta you have to do a clean install on every node you evict during the process. An upgrade is not supported and not possible. I even tried the old trick of editing the cversion.ini file in the sources folder to lower the supported minimum version for an upgrade, but no joy.imageimage

You probably remember this trick to enable an upgrade form the beta/RC  to RTM with Windows Server 2008 R2/Windows 7  But that doesn’t work and even if it did it would not be supported.

But just follow the 3 part series and do an fresh install instead of an upgrade of the cluster nodes and you’ll be just fine.