Some Insights Into How Windows 2012 R2 Hyper-V Backups Work


How Windows Server 2012 R2 backups differ from Windows Server 2012 and earlier

You’ll remember our previous blog about an error when backing up a virtual machine on Windows Server 2012 R2, throwing this error:

Dealing With Event ID 10103 “The virtual machine ‘VM001’ cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID DCFE14D3-7E08-845F-9CEE-21E0605817DC)” In Windows Server 2012 R2

The fix was easy enough, adding a virtual SCSI controller to the virtual machine. But why does it need that now?

Well, this all has to do with the changed way Windows Server 2012 R2 backups work. Before Windows Server 20012 R2 the VSS provider created a VSS snapshot inside the guest virtual machine. That snapshot was exposed to the host, to create a volume snapshot for backup purposes. Right after the volume snapshot has been taken this VSS snapshot inside the guest virtual machine needed to be reverted. The backups then run against that volume snapshot and is consistent thanks to both host & guest VSS capabilities.

For an overview of VSS based backup process in general take a peak at Overview of Processing a Backup Under VSS

Now it is the “Hyper-V Integration Services Shadow Copy Provider” that is being used. When the the host initiates a volume snapshot (Microsoft or hardware VSS provider) the host VSS writer goes in to freeze. This process leverages the Hyper-V Integration Services Shadow Copy Provider  to create the virtual machine checkpoint. After that the volume/LUN/CSV snapshot is taken. When that is done the host VSS writes goes into thaw and the virtual machine checkpoint is deleted. After that the backup runs against the Volume snapshot and at the end that is also deleted. You can follow this process quite nicely in the GUI of your Hyper-V host, you SAN (if you use a Hardware VSS provider).

Dear storage vendors: a great, reliable, fast VSS Hardware Provider is paramount to success in a Microsoft environment. You need to get this absolutely right and out of the door before spending any more time and money on achieving yet more IOPS. Keep scalability in mind when doing this.

Dear backup software vendors: think about the scalability when designing your products. If we have 200 or 500 or a thousand VMs … can we leverage CSV based backups to protect every VM on the LUN or do we need to snap the LUN for every VM backed up? Choice there is good for both data protection schemes and scalability.

At this stage the hardware VSS snapshot is being taken …

image_thumb3

Contrary to common belief this means that the backup will indeed application consistent to the time of the checkpoint as the CSV snapshot being taken is of a consistent checkpoint. It’s the delta in the active avhdx that is only crash consistent, like any running VM by the way. Now pay attention to the screenshot below. The two red arrows are indicating to ntfs source events, two volumes seem to be exposed to the next free drive letters. E: and F: here as C: is the virtual machine OS and D: the DVD.

image_thumb5

Look at the detail. Indeed two. Well it the previous screenshot we only saw one in the CSV path but there are two avhdx files indeed.

image_thumb[1]

Exposing a snapshot on the SAN to a server actually shows us this much better … look here at the avhdx with the GUID and one with “AutoRecovery” in the name. So that makes for two nfts events … and as the backup needs to do this life it requires a vSCSI controller to be present in the virtual machine … and vIDE controller can’t do this.

image_thumb[3]

Anyway, enough under the hood detective work for now, In VEEAM that stage looks like this:

image_thumb7

And on the Compellent it looks like this. The screenshots are from different backups at different times so don’t get confused about the time stamps here. It’s just as illustration of what you can expect to see.

image_thumb12

Now when the CSV snapshot has been taken the virtual machine checkpoint is removed. At that time the backup runs against the CSV snapshot. In our case (hardware VSS provider) this is a snapshot on the SAN that gets exposed in a view and mapped to the off host backup proxy VEEAM server. On the DELL Compellent it looks like this.

image_thumb16

This takes a while to o…but after a while the backup will kick off. Do not that the checkpoint has merged and is no longer visible at this time.

image_thumb18

Once the backup is complete, the mapping is removed, the view deleted and the snapshot expired. So your SAN is left as the backup found it.

There you go. I hope this helped clarify certain things on how Hyper-V guest backups work in Windows 2012 R2. So your backups are still application consistent, just not when you’re running Linux or DOS or NT4.0 as there is no support / VSS for that. However they are based on a  consistent virtual machine snapshot which explains why Hyper-V backups can protect Linux guests very adequately!

Advertisements

14 thoughts on “Some Insights Into How Windows 2012 R2 Hyper-V Backups Work

    • They are only used to create a LUN snapshot needed to bakup the VM and are not applied or even kept around to be applied long term so no. Restoring DCs, whatever version, requires you to follow any rules that apply

  1. Pingback: Markus Bäker » DPM 2012 R2: Abbrüche bei Sicherung auf Hyper-V Hosts

  2. Awesome Info….We have a similar setup, but using DPM 2012. I have question regarding how you place your VM’s in the cluster, For example if i am starting a backup job for 10VM’s that are in CSV1, do all the 10 VM’s need to be on one cluster node during the backup ? at the moment unless they are on the same node, we are finding it hard to get successful backups…..when you have hundreds of VM’s it gets bit harder to do that
    Thank you 🙂

  3. Great article and the inclusion of the VM snapshot in the VSS process is a great enhancement. I do however have a question for you:
    – If you wanted to mount the array snapshot and bring the VM online in an application consistent state, do you have to roll the autorecovery.avhdx into the parent vhdx to achieve this?
    – If so, what data is inside the autorecovery.avhdx? Is it system state data from within the VM etc?

    • I have a blog post in the queue with some more background on the whole autorecovery which in VSS backups isn’t that new. The way that Hyper-V uses checkpoints & leverages autorecovery is. The backup app should handle everything for you but if you go directly to the snaps on a SAN you might notice things are not application consistent depending on the version of the guest OS. If you create a new VM form a VHD(X) You should be ok. With the avhdx not so much, that’s crash consistent and booting from it should show you that. If you want to merge it will be manually and then you have to create a VM from the resulting VHD(X). Note that you should not have to mess with the autorecovery.avhx. It’s a huge topic in itself and I haven’t tested all possible OS versions & permutations either, there’s other work to be done 🙂

  4. Hi we have an Equalogic and now with the Hardware Provider the Backup Process locks like the first Picture. With 2012 Hyper-V it makes Hardware Snapshot on the Equalogic equal the Dell Compelled Picture. What is the right way now? So the Process is completely different from 2012 Hype-V Backup? Is this true?

    Thanks for your answer in advantage

    René

    • The way Windows Hyper-V does backups has indeed changed in Windows 2012 R2. This is seen reflected in the files in a snapshot. There is no wrong or right way, it just works differently.

  5. Good Day,

    We are a small business that is just getting into the virtual environment due to most of our servers running 2003. We are running into all kinds of headaches with going virtual. We (the boss) have chose to go with Hyper-V as our first server migrations and we have a good server with two VM’s running but now I am getting in the management console the Best Practices saying I need to backup the VM’s weekly.
    Have been trying to find a solution that is built in or at lease from Microsoft since it is there product and they are suggesting to do the backups. But mostly what I am finding is 3rd party applications.
    Looked at the Powershell code but I am not that familiar with Powershell and the almost 600 likes of coding, that would have to be modified, is a little intimidating.
    So I have come to the conclusion that we will have to get a third party application but by boss if very unhappy about it. So I am asking is there any easy way to do backups that Microsoft recommends with Server 2012 or other utility?
    Thank you for your time.

    Mark M.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s