Why I’m No Fan Of Virtual Tape Libraries

After implementing a couple of SAN’s with backup solutions I have come to dislike Virtual Tape Libraries. This is definitely technology that, for us, has never delivered the promised benefits. To add insult to injury it is overly expensive and only good to practice hardware babysitting. When discussing this I’ve been told that I want things to cheap and that I should have more FTE to handle all the work. That’s swell but the business and the people with the budgets are telling me exactly the opposite. That explains why in the brochures it’s all about reduced cost with empty usage of acronyms like CAPEX and OPEX. But when that doesn’t really materialize the message to the IT Pros is to get more personnel and cash. In the best case (compared to calling you a whining kid) they‘ll offer to replace the current solution with the latest of the greatest that has, wonder oh wonder, reduced CAPEX & OPEX. Fool me once shame on you, fool me twice, shame on me.

So one thing I’m not planning to buy, ever again, is a Virtual Tape Library (VLS). Those things have a shelf life of about 2 years max. After that they turn into auto disintegrating pieces of crap you’ll spend babysitting for the rest of the serviceable live. This means regularly upgrading the firmware to get your LUN(s) back, if you get them back that is. This means convincing tech support to come up with a better solution than restarting from scratch when they acknowledge that their OS never cleans up its own log files and thus one day just kicks the bucket. Luckily they did go the extra mile on that one after we insisted and got a workaround without losing al backups. Babysitting also means that replacing the battery kits of all shelves becomes a new hobby. You become so good at it that you have better and faster way of doing it than the junior engineers they send who happily exclaim “so this is what it looks like”. The latter is not a confidence builder. The disks fail at a rate of 1 to 2 per week until you replaced almost all of them. Those things need to be brought down to fix just about anything. That means shutting down the disk shelf’s as well and cutting the power, not just a reboot so yes you need to be in the data center.

There is no RAID 6, no hot spares (global or otherwise). The disks cost an arm and a leg and have specialized hardware to make sure all runs fine and well. But in they are plain7.200 rpm cheap 500 GB SATA disks that cost way too much. The need for special firmware I can understand but the high cost I cannot. The amount of money you pay in support costs and in licensing the storage volume is more than enough to make a decent profit. Swapping disks and battery kits isn’t hard and we do it ourselves as waiting for a support engineer takes more time. We have spares at hand. We buy those when we by the solution. We’ve grown wise that way. We buy a couple units of all failure prone items at the outset of any storage project. Having only RAID 5 means that one disk failure puts you virtual tapes at a very high risk so you need to replace it as soon as possible. Once they shipped us a wrong disk, our VTL went down the drain due to incorrect firmware on disk. They demanded to know how it got in there. Well Mr. Vendor, you put it in yourself a as a replacement for a failed disk. In the first year it often happened they didn’t have more than 1 spare disk to ship. If anyone else has a VLS in your area you’re bound to hit that limit have to wait longer for parts. They must have upped the spare parts budget to have some more on hand just for us as we now get a steady supply in Confused smile.

When you look at the complete picture the cost of storage per GB on a VLS is a much as on 1st tier SAN storage. That one doesn’t fly well. At least the SAN has decent redundancy and is luckily a 100 fold more robust and reliable. Why buy a VLS when you can have a premier tier SAN for the same cost, the VLS functionality? No sir that also never lived up to its promises. It has come to the point that the VTL, due to the underlying issues with the device, are more error prone than our Physical Tape Library. That’s just sad. Anyway, we never got the benefits we expected form those VTL. For disk based backup I don’t want a Virtual Tape Library System anymore. It just isn’t worth the cost and hassle.

Look you can buy 100 TB of SATA storage, put it in couple of super micro disk bays, add a 10Gbps Ethernet to your backup network and you’re good to go. Hey that even gives you RAID 6, the ability to add hot spares etc. You buy some spare disks, controllers, NICS, and perhaps even just build two of these setups. That would give you redundant backup to two times 100 TB for under 60.000 €. A VLS with 100TB, support and licensing will put you back the 5 fold of that. Extending that capacity costs an arm and a leg and you’re babysitting it anyway so why bleed money while doing that?

Does this sound crazy? Is this blasphemy? The dirty little secret they don’t like you to know is that’s how cloud players are doing it. First tier storage is always top notch, but if you talk about backing up several hundreds of terabytes of data, the backup solutions by the big vendors are prohibitively expensive. This industry looks a lot like a mafia racketeering business. Well if you don’t buy it you’ll get into trouble, you’ll lose your data. You won’t’ be able to handle it otherwise. Accidents do happen. The guys selling it even dress like mobsters in their suits. But won’t you miss out on cool things like deduplication? If your backup software supports it you can still have that. The licensing cost for this isn’t that bad a deal when compared to VLS storage costs. And do realize instead of 2*100 TB you could make due to 2* 25 TB Open-mouthed smile Hey that price even dropped even more.

When it comes to provisioning storage our strategy is to buy as much of your storage needs during the acquisition phase. That’s the only time deals can be made. The amount of discounts you’ll get will make you wonder what the markup in this market actually. It must be huge. And storage can’t cost as much as you think to build as they would like us to believe. Last time the storage sales guy even told us they were not making any money on the deal. Amazing how companies giving away their products have very good profit margins, highly paid employees with sales bonuses and 40.000 € cars If one vendor or reseller ever tells you that things will get cheaper with time, they are lying. They are actually saying their profit margins will increase during the life cycle of you storage solution. Look, all 1st tier storage is going to be expensive. The best you can hope for is to get a good quality product that performs as promised and doesn’t let you down. We’ve been fortunate in that respect to our SAN solutions but when it comes to backup solutions I’m not pleased with the state of that industry. Backups are extremely important live savers, but for some reason the technology and products remain very buggy, error prone, labor intensive and become very expensive when the data volumes and backup requirements rise.

Keeping The Same Server Name & IP Address When Moving A Domain Controller, DNS, DHCP & WINS Server To New Hardware

Introduction

At one place I provide Infrastructure consulting  & services the supportable life of some servers running Active Directory, DNS, WINS and DHCP has come and gone (+5 years old DELL PE1850) so I was asked to renew them. These are in fact the servers that where upgraded to Windows 2008 R2 some 16 months ago right after Windows 2008 R2 went RTM. See this blog post about that: http://workinghardinit.wordpress.com/2010/01/22/dell-pe1850-domain-controller-upgrade-to-windows-2008-r2/.

For several reasons not all domain controllers or core infrastructure servers should be virtualized so this was not just a matter or doing a P2V migration and be done with it. That would have made it very easy as not only are they already running on the latest Windows version but they had some extra requirements. They wanted to preserve the server names and IP addresses for these servers. Normally DHCP takes care of the DNS servers for clients. Servers that have static IP addresses without using DHCP reservations can be dealt with via scripting. But al lots of devices that need a DNS Server are a pain to deal with. There are devices that do not accept FQDNs, can’t be DCHP clients. Often a management solution for settings the IP configurations on hardware devices is not in place even when does tools exists (i.e. for network switches). Even if such software is in place more likely than not it doesn’t cover all devices (printers, copiers, scanners, load balancers, …) You’re lucky if they know and have a list of all devices that need their DNS entries changed, sometimes you don’t even have that. So I understand the reasons behind their request. In a perfect world this might not be needed but I’ve seldom see that in reality.

So the names and IP addresses needed to be preserved. That is can be done with some preparation and planning.  There is the approach of restoring the server to new hardware using a Windows backup but that’s not without its own issues. You could use one of the (paying) P2P migration tools that comes with various backup solutions or dedicated disaster recovery tools. To do it for free and in the cleanest possible manner I chose to go with the approach you’ll find below. There are multiple domain controllers with DNS, WISN and DHCP set up for redundancy. This means we can take these out of production one by one to recuperate the names and IP addresses. If you didn’t have that you could even introduce a temporary one (virtualized) to give you that option.

Imagine you have to servers named bert.big.corp (192.168.2.10/24) and ernie.big.corp (192.168.2.20) with Active Directory, DNS, WINS and DHCP running on them. These will be replaced with new hardware whilst maintaining the name and IP address. Basically you stage the new servers with Windows 2008 R2 and get everything installed what you need (Server management software from the hardware vendor, the UPS Software) and that doesn’t depend on computer name, IP address and domain membership. That way they are ready to go. You’ll need to wait to rename the servers and add them to the domain one by one until you’ve demoted the current servers one by one that use that name and IP address. So first you replace Bert and only after that has been completed successfully and has been verified you can replace server Ernie. That way you can recuperate the name and IP address without losing those services at any given time. The end result is two new servers with the name Bert (192.168.2.10) and Ernie (1692.168.2.20) and none of your network devices will need any DNS FQDNs or IP addresses changed. Do note that some devices only allow for one IP address or FQDN so they might lose DNS name resolution for the period in which the old server is taken out of service and the new server is brought into service. This is not the only way to do it. I do not like to rename domain controllers and I have other roles to migrate such as WINS and DHCP. So the DNS & Active Directory migration I do is not like the one described in http://technet.microsoft.com/cs-cz/library/dd379526(WS.10).aspx. Perhaps Microsoft can comment on the best practice of this all, but this is how I approached it. This is the first time I did this for Windows 2008 R2 in production as that hasn’t been out in the wild that long yet.

We’ll run through the process step by step for Bert. This needs to be done for every server that is being replaced. Check your Active Directory, DNS and replication is healthy. If not, fix this first. Also check out DHCP and WINS health. As always you have a good backup of the server and it roles before you attempt any of this.

Mind you can play the order of things to do according to you needs & environment but when using some option in the Server Migration Tools like –Users and –Groups you should pay attention unless you want to wind up with a truckload of users/groups from a DC created as local users and groups on a member server Confused smile Oh yeah, I’ve done that. When that happens you’ll love the fact that you know some scripting Winking smile. So if you do the DHCP migration, do both export and import from a domain controller or from a member server (i.e. after demoting the domain controller). In the former case you’ll just get a bunch of warnings during the import that the users/groups already exist and are therefore skipped.

Decommissioning The Old Server

Demote the domain controller

Move all FSMO roles to another domain controller.  Make sure you also move the time server to the domain controller with the PDC role.

  • If you’ve manually altered the Active Directory replication topology, make sure you take care of that as well. If you let the KCC handle this, you’re good to go as it will do this for you. But once you intervene manually it’s your responsibility.
  • We demote the domain controller
  • Reboot and check all went well.
  • We wait for AD replication or we induce it ourselves.
  • Check AD, DNS & replication, if all is well continue.
DHCP Decommission On The Old Server

There is a great tool to do this. Check out TechNet http://technet.microsoft.com/nl-be/library/dd379545(WS.10).aspx. The benefit of using the Server Migration Tools  is that we can also migrate the IP configuration from the old server to the new one in one step! Pretty cool Open-mouthed smile! The details differ depending what version of Windows you’re running. I’m at Windows 2008 R2 so I can install these tools via an elevated PowerShell prompt using

  • Import-Module ServerManager
  • Add-WindowsFeature Migration (to install the Server Migration Tools)
  • We then take a DHCP backup (GUI or via an elevated command prompt using  Netsh DHCP server backup "c:\DHCP\Backup”). Make sure the folder structure exists and is empty. Copy the folder structure of to a file share for safe keeping.  Remember you want to give yourself options.
  • Export the DCHP configuration and content using an elevated powershell prompt:
    • Get-Service DHCPServer (running)
    • Stop-Service DHCPserver
    • Get-Service DHCPServer (stopped)
    • Add-PSSnapin Microsoft.Windows.ServerManager.Migration (load the Server Migration commandlets)
    • Export-SmigServerSetting -featureID DHCP -Group -IPConfig -path C:\DHCP\servermig  -Verbose (not we are working on a DC so we have no local users and hence no use for the –user switch, see TechNet http://technet.microsoft.com/cs-cz/library/dd379483(WS.10).aspx) and copy the folder structure to a file share for safe keeping. You’ll be asked to provide a password for the exported files. You’ll get feedback due to the –verbose switch about the results. Review these and act accordingly. The below screenshot is an illustration of the output.

clip_image004

  • Unauthorize the DHCP Server using an elevated command prompt: Netsh DHCP delete server bert.big.corp  192.168.2.10
  • If you want you can also uninstall the DHCP server role.
WINS Decommission On The Old Server

We won’t be migrating WINS. We’ll just replicate from the other WINS server(s). But we’re a careful bunch and you have to give yourself multiple options to get out of bad situations, as the saying goes: “one is none, two is one”. Do make sure that you adjust the WINS replication topology if you have more than 2 servers so that all of them get the required registration changes. And just in case you ask, yes I know about the Windows 2008 R2 option to dump WINS and use DNS, but they did not need/want this.

  • Take a WINS backup (GUI) and copy the folder structure of to a file share for safe keeping.
  • Stop the WINS service, disable it and copy the WINS database (mdb) over to a file share for safe keeping as well, just in case.
  • Remove the server as WINS replication partner where it was added. It remains in there when you uninstall WINS (old technology). You can actually leave it in there in this case, as a server with that name will be back.
  • If you want you can already uninstall the WINS feature. This requires a reboot.

That’s it we have now successfully stripped the old server Bert from all its previous roles. We can now change the IP address of the old server to 192.168.2.110 and rename the server to BertOld. I give it a restart and check to see if the DNS record is updated. if not a swift restart of the netlogon service might help. It is now a plain member server in the domain that can be taken out of service at later. This means we are ready to bring the new server Bert to life.

Commissioning The New Server

Commissioning DHCP on the new server
  • Rename the new server to Bert.
  • Restart the server.
  • Make Bert a member of the domain.
  • Restart the server.
  • We now have a plain member server in the domain, named Bert on which we also install the Windows Migration Tools using an elevated PowerShell prompt:
    • Import-Module ServerManager
    • Add-WindowsFeature Migration
  • We copy the folder structure from the file share where we earlier copied the exported older DHCP settings and data, back to server Bert, i.e. C:\DHCP\servermig
  • We import the IP configuration plus the DCHP configuration & database using the following commandlet: Import-SmigServerSetting -featureid DHCP -Group -IPConfig NIC -SourcePhysicalAddress 00-14-72-7D-B3-C3 -TargetPhysicalAddress 44-2C-2B-AF-67-D2 -Force -path C:\DHCP\servermig –Verbose
  • Start-Service DHCPServer
  • Via an elevated command prompt we authorize Bert as a DHCP server using “netsh DHCP add server bert.big.corp 192.168.2.10”

Voila, we have Bert running as a DHCP server already!

Commissioning Wins on the new server

We Install WINS and just add the desired replication partners and configure the WINS replication schedule and option to your liking. This is the “cleanest” way. A backup restore is also an option but it could transplant old issues.

Manual trigger Replication and check if you are getting the entries you need (netstat –RR, or via the WINS GUI)

Please note that the browser service is not running by default on Windows 2008(R2) so if you need this for you clients you should set the service to automatic and start it.

Commissioning Active Directory and DNS on the new server
  • Make sure you can contact an existing domain controller before promote Bert to become a domain controller, this can be assured by having the first DNS server in your IP Configuration point to an operational DNS Server in the domain.
  • Promote Bert to a domain controller. By default the DNS option is enabled. Take a look at http://technet.microsoft.com/cs-cz/library/dd379526(WS.10).aspx that will walk you through the wizard.
  • Check the health of you Active Directory, replication and name resolution once more and keep a close eye on it the following days. You should be good to go.

Conclusion

Voila, a new Bert server is born. Not too bad and a happy customer. Rinse and repeat for every domain controller you want to replace.  Do make sure you check out the documentation on the Server Migration Tool and its options. Also, if you are running Windows 2008(R2) and got there by upgrading from Windows 2000/2003 consider moving from FRS to DFRS for SYSVOL replication. As mentioned is a previous blog post of mine (http://workinghardinit.wordpress.com/2011/01/06/dcdiag-exe-problem-on-windows-2008r2-verifyenterprisereferences-indicates-problem-missing-expected-value-points-to-knowledge-base-article-q312862/) there is an absolute brilliant step by step guide to get the move from FRS to DFRS completed without a problem in a series by the storage team at Microsoft . You can find the first of a 5 part blog series over here http://blogs.technet.com/b/filecab/archive/2008/02/08/sysvol-migration-series-part-1-introduction-to-the-sysvol-migration-process.aspx. I prefer to have this done before I introduce the new domain controllers. This helps me with my neurotic urge to have a SYSVOL folder named SYVOL and not SYSVOL-DFRS clip_image002[1]

Windows 2008 R2: The system image restore failed. Error details: The parameter is incorrect. 0×80070057

Note to self: read your own blogs on Windows 2008 R2 Native Backup :-) . Yes people, Windows 2008 R2 Bare Metal restore to dissimilar hardware does work as long as you follow the rules and guidelines. Those are not super evidently documented but still, if I can find ‘m you can too! But today we lost some time because we didn’t head one of the rules that trip people up frequently. That rule is that the disk layout on the restore server can’t differ from the original one. I literally wrote “Pay close attention to the disk layout/ boot order as well, the restore doesn’t allow for variation from the original layout” in http://workinghardinit.wordpress.com/2010/01/27/using-windows-2008-r2-backups-to-go-virtual-2/. That means you need to simulate the same disk layout on the new hardware. If the new server has an extra disk, disable that one for the restore, if it has one less, add one. Another situation where the disk layout comes into play is when you boot from an USB stick with W2K8R2. If you leave it plugged in there during the restore the recovery will fail. Because if that extra attached disk isn’t the one containing the backup image you’ll get a very harsh error:

"The system image restore failed. Error details: The parameter is incorrect. 0×80070057"

image

Not very helpful in explaining but that generally means you’ve got a disk layout issue. In this case because you have the bootable USB stick attached. Once you’ve booted to the “Repair your computer” functionality, selected “Select a system image backup” and found your image to restore you should remove the bootable USB stick from the server if you’re not going to be doing an install. Beware of this! Typically when you boot from DVD or PXE you wouldn’t even notice but when using a bootable USB device with W2K8R2 you might forget that this changes the disk layout. So again, always pull the bootable USB stick from the server before you restore and you’ll be fine. Yes the recovery will work a soon as you’ve booted, you don’t need the media anymore so you can unplug it safely. You can even attach another USB disk in its place containing the backups if you only have one USB port available. That will work because the disk with the backup itself is never taken into consideration and won’t cause any issues with the restore.

So we’ll never forget to head our own warnings again (I hope). The good thing is we had some refresh training on restoring today and it’s all refreshed in our minds :-)

Calling x64 CLI Tools in x86 Scripting Tools and Processes

Every now and then I get the same question from people who only recently decided to make the switch to x64 bit Windows operating systems. I’ve been running on x64 since Vista RTM and I’m very happy with it. When those people start scripting with their tools, which are 32 bit, calling some CLI tool in %windir%\System32 they can run into an annoying issue that express itself in the correct yet somewhat misleading “WshShell.Exec: The system cannot find the file specified.”. But you know it’s there in %windir%\System32, you checked and double checked!

When your scripting tool is 32 bit and you run your script it usually launches an 32 bit version of the CLI tool you’re calling. This behavior is a result of file redirection. This is a transparent process that’s part of the Windows-on-Windows 64-bit (WOW64) subsystem that is used to run 32 bit apps. When a 32 bit applications calls a CLI tool in the %windir%\system32 directory it silently redirects this to the %windir%\SysWOW64 where 32 bit apps can happily run without a worry on an x64 bit operating system. Yes, indeed %windir%\system32 is for x64 code only and %windir%\SysWOW64 is for 32 bit code.

What’s in a name :-) Some people argue they should have use system32 for 32 bit and system64 for x64 bit but I’m sure they had their reasons for what they did (i.e. it would have been hell for some reason I guess). Other suggestions have also been made by people who are far better qualified than I am. For example by Mark Russinovich, a hard core systems developer, in http://blogs.technet.com/b/markrussinovich/archive/2005/05/07/running-everyday-on-64-bit-windows.aspx.

Now all this can happen transparently for the user if the tools used have both an x64 and a x86 version. Cmd.exe and ping.exe are fine examples. If you run some VBScript in my favorite scripting tool for example (Sapiens PrimalScript) which is 32 bit it will launch a 32 bit cmd.exe, that launches the cscript.exe 32 bit version and which will launch ping.exe (using WScript.Shell) in %windir%\SysWOW64 by silently redirecting your %windir%\system32 path. No worries, you don’t know any better and the result is the same. So it’s usually not a problem if there is both a x64 and a x86 version to the CLI tool as you have seen in the ping.exe example. When a 32 bit process calls a tool in %windir%\system32 it’s redirected to %windir%\SysWOW64 and uses the 32 bit version. No harm done.

The proverbial shit hits the fan when you call a CLI tool that only has a x64 bit version. As the scripting tool is x86 it’s call is redirected to the WOW64 and the script fails miserably as the CLI tool can’t be found. This can be pretty annoying when writing and testing scripts. The CLI backup tool of Windows Backup is a prime example. It does not have a 32 bit version. Consider this little script for example:

Option Explicit

Dim oShell
Dim oExecShell
Dim sBackupCommandString
Dim sText

Set oShell = CreateObject("WScript.Shell")
'sBackupCommandString = "%windir%\sysnative\wbadmin get disks"
sBackupCommandString = "%windir%\system32\wbadmin get disks"

Set  oExecShell = oShell.Exec(sBackupCommandString)

Do While oExecShell.Status = 0
    Do While Not oExecShell.StdOut.AtEndOfStream
        sText = oExecShell.StdOut.ReadLine()
        Wscript.Echo sText 
    Loop    
Loop

Set oShell = Nothing
Set oExecShell = Nothing

There is a lot of File Redirection going on here to %windir%\SysWOW64 when running this code in the 32 bit scripting tool. That tool launches the 32 bit cmd.exe and thus the 32 bit cscript.exe which then launches a 32 bit shell and tries to run "%windir%\system32\wbadmin get disks" which is also redirected to %windir%\SysWOW64 where wbadmin cannot be found throwing the error: “WshShell.Exec: The system cannot find the file specified.”. If you don’t have a 32 bit code editor just launch the script manually from an 32 bit command prompt to see the error.

The solution as demonstrated here is to use as in “%windir%\Sysnative\wbadmin.exe get disks”. Uncomment that line and put the line with sBackupCommandString = "%windir%\system32\wbadmin get disks" in comment. Do the same test again and voila. It runs. So there you have it, you can easily test your script now. Just make sure that when the time comes to put it out in the wild you replace it with the real path if the calling process is x64 bit, which for example wscript.exe and cscript.exe are when you launch the form a x64 bit shell (explorer.exe or cmd.exe), which is the default on a x64 operating system. The x86 version runs when you launch them from a x86 shell. But remember the default on x64 bit operating systems is x64 bit and sysnative only functions when called from a 32 bit process (it’s a virtual directory that doesn’t really exists).

Sysnative was introduced in Vista/Windows2008 x64 bit. Not only 32 bit script editor users a affected by this, all 32 bit processes launching tools in "%windir%\system32 are. See more on MSDN via this link http://msdn.microsoft.com/en-us/library/aa384187(VS.85).aspx.  For the folks running XP or Windows 2003 x64 bit it is perhaps time you consider upgrading to Windows 2008 R2 or v7 x64 bit? If you can’t, no need to worry, you’re in luck. Microsoft did create a hot fix for you (http://support.microsoft.com/?scid=kb;en-us;942589) that introduces sysnative on those platforms. So welcome to the x64 bit universe, beware of file redirection in WOW64 and happy scripting :-)

Using Windows 2008 (R2) Backups to Go Virtual Part III

Part III

Sanitizing the Disk Configuration & Layout


This series describes a physical to virtual migration of a Windows Server 2008 SP2 Standard x64 on a Dell Power Edge one unit rack server with SATA disks (PE750). In fact that was the first production Windows 2008 server ever at that company, they really wanted to use the RTW FTP over SSL functionality. For this exercise we had no use of commercial P2V tools and the original server had to be kept running during the virtualization exercise. The aim was to achieve this with the free / built in tools as much as possible. The hands on approach also provides for an excellent learning opportunity. We have reconstructed our experiences for this article and will discuss the subject for both Windows 2008 and Windows 2008 R2.

 Part I: http://workinghardinit.wordpress.com/2010/01/27/using-windows-2008-r2-backups-to-go-virtual-2/
Part II:  http://workinghardinit.wordpress.com/2010/02/01/using-windows-2008-r2-backups-to-go-virtual-ii/

ParrtI:

Breaking the Software Mirror


We don’t need the software mirror anymore for redundancy. On the Hyper-V host the disks are LUNs presented from SAN. The SAN provides the redundancy. So the first thing we do is break up the software mirror. You can use the Disk Management GUI to break the data volume(s). Select the member of the mirror you are not booted from, right click and select break mirror. Confirm that you want to do this at the warning message and you’re done. We opt to break the mirror as this gives us to working copies. This comes in handy in case we mess something up. Removing the mirror is also an option but then you’ll end up with only one copy.

Part III – Figure 1: Breaking the data partition mirror.

Now when you try this on the boot/system volumes this will not work at all. You get the following error: “The specified plex is at the current system or boot plex”.

The solution to this is using diskpart to break the mirror as explained in following KB article: Software mirrored boot volumes in Windows Server 2008 cannot be broken from Disk Manage
http://support.microsoft.com/kb/969749/EN-US I guess somewhere Windows GUI admins where deemed not savvy enough to know what they where doing?

Part III – Figure 2: You have to use Diskpart to break the boot partition mirrors.

Sanitizing Disk configuration and layout


The task at hand is to go from dynamically expanding VHD files to fixed vhd files and shrinking them so they don’t waste so much space. While dynamically expanding vhd’s can be compacted to take up less physical storage, the total capacity of the vhd disk itself does not change.

One option would be to convert them but then the issue with the available tools Hyper-v from Microsoft is that they become 1TB fixed vhd’s. We would then still need to shrink them. The Microsoft Hyper-V tools however only allow for extending disks, not shrink them. Another consideration when using the Hyper-V Disk Edit tool to convert to a fixed disk is that you need sufficient disk space to do it. We are dealing with + 1TB disks here and we just could get enough free disk space to do this.

There is a free tool called VhdResizer (http://vmtoolkit.com/files/folders/converters/entry87.aspx, you need to register for the download) that can shrink vhd files. However, it needs free, non-partitioned space to do this. And it cannot handle Windows dynamic disks. This brings us to another major concern and that is that many disk tools can’t handle dynamic disks and only work well with basic disks. To be clear, I’m not talking about dynamically expanding vhd’s here but about dynamic disks in windows. Try working with a vhd (dynamically expanding or fixed) that contains a Windows dynamic disk and you’ll see this error:


Part III – Figure 3: VhdResizer can’t handle Windows dynamic disks.

The Data Partition


Let’s first focus on getting the data from its partition on the boot disk with the operating system to its own disk. That disk has to a fixed vhd disk with a windows basic disk. There are several ways to get there. Not all are fully supported by Microsoft. We list three approached that work.

Copying or Restoring With NTFS Permissions Intact.


Xcopy, robocopy the data from the original data partition or restore the data directly from backup to a newly created fixed vhd of the desired size and which is a windows basic disk in the virtual machine. This method can retain NFTS permissions and is fully supported. Afterwards we can delete the original partition.

Cloning


We could use a disk clone tool that supports dynamic windows disks and clone the partition to a new fixed vhd of the desired size and which is a windows basic disk in the virtual machine. Ghost.exe/Ghost32.exe can do this perfectly. The cloning is supported by Norton in this case. Afterwards we can delete the original partition. Or use Gparted to achieve the same thing al be it a bit more hands on as you’ll see when we start playing with the boot/system disk.

Using VhdResizer/Gparted free tools


Another way which requires also no paid for tool and uses VhdResizer is the following:

Take copy of the vhd from the backup folder that corresponds with the data partition.

Rename it to something sensible if you want.

Add it to the virtual machine.

Start the virtual machine. You’ll see that is a windows basic disk.

You need to create free space on the disk resize the disk with VhdResizer. So shrink the partition to its desired size or a small as you can with Disk Management. This might still leave considerable space (about half of the original partition size) due to unmovable files.

If that’s too much like in this case (we could not shrink the partition in windows beyond 500 GB), gparted live CD comes to the rescue! The disk is already windows basic disk and that is a format this tool can handle. So if you need to shrink more that windows will allow you to do. Turn to gparted live CD. If the disk you’re working with is larger than 127 GB like in this case mount the disk you’ll have to test various versions … for example gparted-live-0.4.1-2.iso can handle this, but the more recent gparted-live-0.4.6-1.iso can’t. Such in live in systems engineering J


Part III – Figure 4: resizing with gparted.


Part III – Figure 5: reading the data


Part III – Figure 6: Copying the data, after this step it’s done! Just exit gparted & shutdown the VM.

Stop the virtual machine

Remove the vhd from the virtual machine

The vhd is a dynamically expanding one but you can convert it to a fixed vhd and resize it (make it smaller in our case) with vhd resizer. This takes a long time and the GUI doesn’t give any decent feedback. Give it time and be patient.


Part III – Figure 7: Using VhdResizer to shrink the vhd and convert it to a fixed one.

Attach the resized, fixed vhd to the virtual machine, start it, make sure it gets the correct drive letter and you’re good to go!

Now VhdResizer.exe is not supported by a vendor, it’s freeware. Gparted Live CD has no warranty but you can get decent information/help on line. Be aware of the fact that between versions (like the example above) certain functionality is sometimes broken. Such is life. The good thing is that Gparted had drivers for keyboard and mouse that work in Hyper-V from the start. A lot of other tools didn’t or don’t. In the end make up your own mind. I have used all methods and tools on several occasions with success.

The Boot/System Disk


Here we want to get rid of the large data partition on the disk. If the above actions where successful we can just delete that partition. We already have the data covered. Now we still want to get to a fixed, decently sized vhd that contains a Windows basic disk. Again we have several options but beware one of them is very risky and not supported by Microsoft. We list three approached that work.

Cloning


Agian we could use a disk clone tool that supports dynamic windows disks and clone the partition to a new fixed vhd of the desired size and which is a windows basic disk in the virtual machine. Ghost.exe/Ghost32.exe can do this perfectly. The cloning is supported by Norton in this case. Since we didn’t provide any details of this above we’ll dig a bit deeper here. Gparted can’t be used directly as it doesn’t handle windows dynamic disks.

Warnings concerning the techniques discussed below:

When you mount the same dynamic disk as a second drive on a virtual server that boots form the original copy of that dynamic drive you’ll get errors in diskpart/disk manager about the second drive being invalid. Trying to manipulate it (bringing it on or off line) makes the system unresponsive and even corrupts that drive. The trick is to mount the bootable dynamic disk (on the dynamically expanding vhd we want to clone) in another virtual OS that boots form a basic disk.

Now that makes you appreciate Windows 2008 R2 where you can just attach a VHD and work with it, right? Or so you would think. Wrong!
THIS DOES NOT WORK WITH WINDOWS DYNAMIC DISKS in a vhd, whether it contains a bootable partition or not. They will also pop up with “Invalid”. WARNING: If you convert it to a basic disk, you’ll lose all data. The good news is that vhdmount from Virtual Server R2 SP1 does the job.

OK now you have been warned lets walk through the cloning process. Create a new fixed VHD file of new required size. Mount both the new and the old vhd file so we can clone them. You can mount these vhd in several ways depending on what you like best or is more suited for your environment.

You can mount the disks in a host using either the native vhd mount capabilities of Windows 2008 R2 if they are NOT Windows dynamic disks (See warnings above). In our case we cannot us this as the original disk is a dynamic disk. If you haven’t got Windows 2008 R2 by using the Virtual Server 2005 R2 SP1 VHD Mount utility which is a free download? Follow the instructions at http://technet.microsoft.com/en-us/library/cc708295(WS.10).aspx to get vhdmount operational. This does work with dynamic disks! You can also attach them to another virtual machine (Do not use the original one, see warnings above) as extra disks and run ghost/ghost32 in that virtual machine. There are lots of options here; you can figure it out it’s not hard.

I’ll just post the commands for vhdmount I used here:

vhdmount /p /f “E:\Disks\DynamicallyExpandingDiskWeWantToResize.vhd”

vhdmount /p /f “E:\Disks\FixedEmpty.vhd”

Note: Mounting the VHD files is possible only if the VHD files are not currently in use by any virtual machine or other program. You should receive a “The Virtual Hard Disk is successfully plugged in as a virtual disk device.” response, followed by a series of “New hardware found” messages.

Initialize the disk to write a MBR to it to avoid possible booting issues you’ll need to fix.

Run GHOST32 utility, which can be found in the directory where you’ve installed Symantec Ghost. Using the GHOST32 utility, you can “resize while copy” any of the local drives (Local → Disk → To Disk) including virtual ones now mounted. Make sure you choose the destination disk drive that represents the new VHD file, as you can easily overwrite any other local disk drive. You can always use GDISK32 utility to determine first which one is the new drive if uncertain.

Once transfer is complete you can safely choose not to restart the computer and just exit the GHOST32.

Don’t forget to detach the vhd’s! When using vhdmount use following commands:

vhdmount /u “E:\Disks\DynamicallyExpandingDiskWeWantToResize.vhd”

vhdmount /u “E:\Disks\FixedEmpty.vhd”

Hxd Hex Editor


This is risky business but worth a try just to give us an option with a free tool. We can convert the dynamic disk to a basic disk with Hxd editor http://mh-nexus.de/en/hxd/. This is totally unsupported but I’ve had great success with it. It doesn’t work for all conversions under all conditions but it’s worth a shot. Make sure to make a backup copy of your virtual machine before truing this! It’s the only way I know to convert a dynamic disk to a basic disk without a commercial tool like Ghost that is one of the few that supports dynamic disks. You can find a good write up of how to use Hxd to convert a dynamic disk to a basic disk here: http://strangelyperfect.tv/6415/how-to-convert-a-dynamic-disk-to-basic-disk-in-windows-7/ & here http://www.dynamic-disk.com/convert-dynamic-disk-to-basic.html

I’ll just report what I did here.

Install the tool in the virtual machine, launch it and select the disk you want to edit. In our case it’s disk 1. In the partition table live in the area between the two arrows. Every 42 you see starting at 1C0 up to must be changed into 07. You have to them all (one for all primary partitions the disk sees) or you’ll mess up the disk.


Part III – Figure 8: The select disk to edit. Note the 42 values in the partition table.


Part III Figure 9: Warning when you save the changes to disk in Hxd.

Save your changes to the disk, these are the values marked in red. Acknowledge the warning, close Hxd and restart your virtual machine. When the servers reboots successfully you can have a look in Disk Management. Sometimes you’ll have to do some disk cleaning afterwards as in the figure below, but the important thing is that the dynamic disk has become a basic disk. Just delete the volumes on the dynamic disk that’s reported as missing and that’s it.


Part III Figure 10: Clean up the dynamic disk that is now reported as missing because it was converted to a basic disk.

Again, I haven’t always been able to get this tool to work in every situation. But it’s worth a try on a COPY of your virtual machine if you have no other options.

You can optimize your disk partition size if you want using Windows Disk Management or Gparted. And finally we can delete the data partition and follow the same methods we described to create data partition to resize the disk to the desired size, using VhdResizer … or not?

Nope no such luck! When I tried to use VhdResizer.exe we got that error again “Invalid VHD file. Please select another”. Now what? Well cloning the disk to a squeaky clean fixed vhd directly and after that see if VhdResizer.exe can handle it. Of cause if you have a disk cloning tool that can handle cloning from a larger disk to a smaller disk you do no not need + 1TB in space and you can simply skip the process of having to reduce its size with VhdResizer.exe. Ghost can handle this but is a paying tool. Gparted is also very handy in achieving this, but more hands on. I tried both. Ghost worked well and without issues. I used ghost/ghost32.exe (11.5). Again we can boot into a virtual machine to do the cloning or mount the vhd’s using VhdMount.exe or directly in the Disk Management GUI (W2K8R2 only) since the disk is already a basic disk! Now to prevent any issues, the ones we warned about above, during the cloning we like to mount the source disk and the target disk in an operating system that is another (not just a copied virtual machine) than the original. That will save you a lot of head ache.

But we really wanted to succeed trying this with a free tool so we took to Gparted once more with the source (dynamically expanding vhd) and target disk (fixed vhd) mounted in a virtual machine.We succeeded to clone the partitions one by one to a new fixed vhd disk. We also marked the correct partition as bootable as judged form looking at the original. But booting from that new disk failed with error 0xc000000e, the boot selection failed because a required device is inaccessible. After rebooting into the Windows 2008 (R2) DVD and opting to repair an existing Windows 2008 installation that it found (0 MB!) I launched the command prompt and ran following commands from there:

bootrec /fixmbr

bootrec /fixboot

boorrec /rebuildbcd


Part III Figure 11: Fixing 0xc000000e the boot selection failed because a required device is inaccessible

For more information see “How to use the Bootrec.exe tool in the Windows Recovery Environment to troubleshoot and repair startup issues in Windows” http://support.microsoft.com/kb/927392.

So than I restarted to try and boot from disk, I was told it needed to run chkdsk which is normal when you used Gparted. After chkdsk ran it booted into the operating system from a fully functional fixed vhd disk that was resized to the correct dimensions. It was even accepted by VhdResizer.exe again as a valid vhd file when I checked.

In the end all was well. Now for the record, this was a demonstration of what we did in one case. It worked for us. The virtualized server is running happily on a live migration Hyper-V cluster as you this is published. Not bad J Just don’t expect Microsoft to support you if you have issues, you’re on your own when you venture into this area. Search and find your own creative solutions, you’re working on VM’s so don’t worry, take copies and you have a lot of changes to try again and again …

The Final Result achieved in production


As a reminder: this is where we came from on the production physical server.

Part III Figure 12: The disk configuration on the original physical machine (W2K8). The bad quality is due to it being taken over a KVM over IP session.

And here we have our final result. This is the layout of our production virtualized server.

Part III Figure 13: Success, the desired disk configuration in the achieved production virtual machine (W2K8).

So we have what we set out to achieve , a virtual machine running on Hyper-V, with 40 GB system drive on a basic disk in a fixed VHD and a 150 GB data drive on a basic disk in a fixed VHD. We succeeded in separating the C: and the D: partitions to their own disk (vhd). The C: and D: disks were also converted from dynamic disks to basic disks. They were also shrunk to a more acceptable size. Finally we converted the disks from dynamically expanding vhd disks to a fixed vhd disks. Mission accomplished.

Conclusions


Some understanding and hands on experience with Windows 2008 (R2) Backup and Complete PC Restore are very valuable to have. Learn this in labs and during testing. Make sure you don’t have to your first restore for real on a critical server.

There is so much to try and experiment with regarding the disk conversions and resizing that it’s almost impossible to discuss them all here. Every situation is different & what I did here might not work for you. So be careful. The good thing is that, once you have virtualized your server, you have the ability to make copies and backups very easily so you can afford to experiment. I suggest you do exactly that and learn from the experience. We all have situations, environments where things are not ideal and where we need to use our creativity to achieve the desired outcome. Whether all of this is worth the time and a good idea in a particular situation is not for me to decide. I’ll just try and get the job done when needed. Advice and consultancy on how to prevent issues, how all their stuff should be in the cloud and such is all very well but people tend to get annoyed with that when they are bleeding from open wounds. So fix their wounds first. And that’s what we did here. Their entire old SATA disk based 1U rack server infrastructure has been virtualized or replaced and is now running happily on a high available Hyper-V cluster.

Well, that’s it. I hope you find some useful tips, pointers and inspiration for both Windows 2008 (R2) Backup and P2V migrations in these notes. The build in tools are good and work well. But you do need to learn about their abilities and limitations. They are free, support and you don’t need agents to backup/restore. Are they the best fit in any situation? No. Are they sometimes the best value for money? Absolutely!

Using Windows 2008 (R2) Backups to Go Virtual II

Part II

Restoring the Backup – P2V

This series describes a physical to virtual migration of a Windows Server 2008 SP2 Standard x64 on a Dell Power Edge one unit rack server with SATA disks (PE750). In fact that was the first production Windows 2008 server ever at that company, they really wanted to use the RTW FTP over SSL functionality. For this exercise we had no use of commercial P2V tools and the original server had to be kept running during the virtualization exercise. The aim was to achieve this with the free / built in tools as much as possible. The hands on approach also provides for an excellent learning opportunity. We have reconstructed our experiences for this article and will discuss the subject for both Windows 2008 and Windows 2008 R2.

Part I: http://workinghardinit.wordpress.com/2010/01/27/using-windows-2008-r2-backups-to-go-virtual-2/
Parrt III: http://workinghardinit.wordpress.com/2010/02/02/using-windows-2008-r2-backups-to-go-virtual-part-iii/

This will probably be the most “boring and redundant” part of the series. A very good post about this process for Windows Server 2008 can be found on the internet @ http://blogs.technet.com/askcore/archive/2009/02/04/windows-server-backup-2008-restore-from-network-location.aspx. They also discuss the command line (wbadmin.exe) way of doing it. So check it out if you need to do that. I did a restore of a Windows Server 2008 R2 machine for this demonstration for completeness. You’ll see that some of the redundant messages during a restore have been fixed and the work flow has been stream lined.

If you did the preparations for the virtual machine mentioned in Part I correctly this part will hopefully be an uneventful activity. Apart from the fact that you do a bare metal recovery from the backup of a physical server to a virtual server which is pretty cool, the result of this restore is the virtualized server with mirrored dynamic disks we’ll have some fun with in Part III. But in the end Part II is a GUI walk through the restore process for illustration purposes as this series is part of a workshop/documentation paper on Windows 2008 (R2) backup and restore for internal/personal use.

Walk Through

We boot the virtual machine we prepared in part I from the Windows 2008 R2 DVD ISO image that we the mounted via the setting for our virtual machine in Hyper-V.

We choose the desired language/keyboard settings and click “Next”.

We’re not going to install Windows but do a restore so we need to select “Repair your computer”.

We’re using a blank virtual machine, so the fact we do not see any installed operating system is no reason, to worry. Select to “Restore your computer using a system image that you created earlier.” and click Next.

At the “Select a system image backup” Click “Next” and your system will be scanned for system image disks

You will get a warning that “A valid backup location could not be found”. Just ignore this and click “Cancel”. It’s perfectly OK.

No latest available and automatically found images appear. This is logical as “a valid backup location could not be found (we have our backups done to a network share) and this is bare metal recovery. It would be spooky if something was found. Select “Restore a different backup” and click “Next”.

Click Advanced. You do not need to install a network driver if you prepared the virtual machine with a legacy network adapter, that driver is included in the Windows Server 2008/R2 DVD so that it just works. The synthetic driver is not included in Windows Server 2008 DVD but it is in Windows Server 2008 R2 DVD.

It will ask “Are you sure you want to connect to the network”, which is what we want and then specify the path of your backup. Remember the remarks if you do not have name resolution you can use the IP address. Also make share you have the credentials needed to access the share, you really need those.

Fill out the UNC path to where you store your backups. That is the path that points to the root where the “WindowsImageBackup” folder resides.

When asked for credentials please provide ‘m.

Select the backup listed and then click “Next”.

Select the desired system image and then click “Next”.

You are presented with the restore options.

We don’t want to exclude disks, we don’t need to install drivers and the default setting under the Advanced button work fine for us, as we need to restart the computer to finish the restore and there is nothing else left to do. Checking for and updating error information is a good thing.

Click Finish to confirm the settings. As we selected “Format and repartition disks” they need you to confirm that this is what you really want. Click “Yes” to start the restore.

You then see a number of dialogs reporting the progress of the restore operation.

The virtual machine restarts and boots into the restored operating system. After verifying all is well with the restored server, making sure we have the integration tools up and running we have accomplished our physical to virtual migration.

The final thing to do is open up a Command Prompt Windows (run as administrator), enter set devmgr_show_nonpresent_devices=1 and then start devmgmt.msc from that same command prompt window. After selecting show hidden devices on the Device Manager View menu you can now see devices that are no longer present in the system, we’ll find our missing hardware NIC’s and delete them. That way we can assign the configuration to the new NIC. See http://support.microsoft.com/kb/241257 for more information. It’s about Windows 2000 but it still applies. That’s it. In the next part (Part III) we’ll deal with the disk sanitization & reconfiguration.

Using Windows 2008 (R2) Backups to Go Virtual

Part I

The Challenge

This series describes a physical to virtual migration of a Windows Server 2008 SP2 Standard x64 on a Dell Power Edge one unit rack server with SATA disks (PE750). In fact that was the first production Windows 2008 server ever at that company, they really wanted to use the RTW FTP over SSL functionality. For this exercise we had no use of commercial P2V tools and the original server had to be kept running during the virtualization exercise. The aim was to achieve this with the free / built in tools as much as possible. The hands on approach also provides for an excellent learning opportunity. We have reconstructed our experiences for this article and will discuss the subject for both Windows 2008 and Windows 2008 R2.

Physical Machine

Disk Layout

The physical server has no raid controller. For redundancy reasons dynamic disks are used for software mirroring. The original 160 GB SATA disks were going to run out of space "very soon" during its lifespan they claimed, so they were swapped out for bigger ones. This was done one by one, with mirror rebuilding in between, and as such we "leapfrogged" to two 1TB disks. We then made a bigger data partition by concatenating the disks. That combined with some leading free disk space where OEM partition used to be on the original disks resulted in an interesting disk layout. The installation was healthy but the disk layout was not optimal and in fact, the server had way to much disk space. The owners indicated they would need vast amounts of disk space but this proved exaggerated and actual data needs are around 110 GB with some margin for growth. The disk layout is illustrated in below in Figure 1 and shows the disk configuration on the physical machine.

Windows Backup

Backups of this server are made to a file share using a script that automates Windows 2008 Backup (wbadmin.exe). The script does not only backup the server, it also creates backup archives. This is need as Windows 2008 Backup to a UNC path overwrites the last backup. So to keep a couple of weeks/months worth of backups we need to move them out to a save place. The script also collects logs about what’s happening for auditing and mails these to the system administrators indicating success or failure. Since the script also needs to work on Windows Server Core it’s VBScript (it was written late 2007) as .NET support and thus Power Shell is only available in Windows 2008 R2. We will use a backup of the physical machine to restore the server to a virtual machine.

Figure 1: The disk configuration on the physical machine. The bad quality is due to it being taken over a KVM over IP session from the original server!

Goal of the Migration

During the migration we wanted to sanitize the disk layout if possible. The task at hand was to migrate this server to a virtual machine using built in or free tools. Only if we could not find another way we could resort to commercial third party tools but only if we had them available at no extra cost (so a working trial or evaluation version is allowed). We could not use SCVMM 2008 as this server is member of a different domain (in the DMZ). The option of making it a member of our production domain is a no go due to security concerns and installing SCVMM2008 in the DMZ domain was not allowed. The original server also had to be online during the virtualization. We therefore opted to restore the backup of the physical server to a virtual server. During this process our goal was to:

  • resize the disks (no use wasting disk space),
  • separate the system drive from the data drive,
  • convert theWindows dynamic disks to basic disks (no need for dynamic disks and better support with a lot of hard disk tools ),
  • use fixed vhd files for better performance (note: dynamically expanding disks are not that noticeably slower I must say, in low IO situations with little growth. In W2K8R2 the difference is reported to be negligible)

The picture below, Figure 2, illustrates the desired (and achieved) outcome.

Figure 2: The desired disk configuration in a virtual machine.

Preparing a Virtual Machine to Restore the Backup

The entire process described below describes most of the problems you can encounter with restoring Windows 2008 Backups in general. Pay attention to the solutions to these problems and you’ll do just fine. I read a lot of postings on the internet about Windows 2008 backups not working properly but it does. You just need to learn about its limits and abilities. And yes, there are some "quirks" that you need to be aware of. Once you learn this the build in tool will serve you well. I think some of these quirks Microsoft should have documented better and some things are indeed missing functionality (most have been fixed in R2), which is unfortunate but not a show stopper. Remember the definition of engineering: building good working solutions with the means available.

Since we will restore the backup of the physical server to a virtual server, we need to prepare one. We use the Hyper-V to achieve this.

Create the Disks

We start by creating two disks. We need two disks because the original backup was of a server with two physical disks that were members of a software mirror, which also means they were dynamic disks. If you don’t have to disk to restore to you’ll get a cryptic error that the system image restore failed with some error details "Element not found. (0×80070490)". This is the same error you can when the disk layout/boot order is different on the restore machine than the original one. Pay attention to this.

Figure 3: Mind the disk layout and order or you’ll get into trouble.

Windows backup can only restore to disks of the same size or larger (unless you start doing partition restores manually and get handy with BCDEDIT). Since our physical are 1TB and we didn’t have the disk space for such large VHD files we needed to be a bit creative. We came up with the workaround of restoring to dynamic vhd files of 1048 GB (making sure it was indeed a bit larger than the original ones in the backup). The actual size of the data is not higher that 160 GB anyway so the dynamic VHD will not grow beyond that. The restore will see that the disk are large enough and function as planned. We’ll deal with the resizing of the disks, making them basic disks in windows and converting them to fixed vhd after the Servers has been restored to a virtual machine.

Figure 4: The details of the dynamically expanding vhd’s we create to restore the backup to.

Do make sure that you restart the virtual machine once with the new hard disks attach. This is to make sure the disk preparation is 100% finished. IF not a restore might fail with following error: "The system image restore failed." The error details state: "The computer needs to be restarted to finish preparing a hard disk for restore. To continue, restart your computer and run the restore again. (0×80042403)."

Figure 5: Error trying to restore a backup to new not yet prepared disk.

Allocate CPU & Memory

We configure the VM with CPU like the original one. As the original operating system is Windows 2008 x86 we won’t have many issues with virtualization due to the wrong HAL and such … things on that level have definitely become a lot easier and better than they used to be. We allocate enough memory to get the job done on my host.

Networking

We also configure the machine with a temporary legacy NIC. We use a legacy NIC for the restore when we restore a Windows 2008 server because the synthetic NIC aren’t included in the DVD image (you could build one using the WAIK) and we need a functional NIC to get to our backup on the file share. The Windows 2008 R2 image does contain the drivers you need for a synthetic NIC. To make live easy we have a DHCP server had out IP addresses. That saved us the trouble of configuring the TCP/IP settings manually during the restore process. No other NIC’s where configured so the restored VM wouldn’t conflict with the physical one still running in production.

Figure 6: The settings of the prepared VM to which we are going to restore the Windows Backup of the physical machine.

Recapture of Windows Backup in 2008 (R2) Restore Quirks

Mirrored Disks & Disk Layout

When restoring from a software mirrored dynamic disks make sure you have two disks to restore to or you’ll get an error indicating very helpfully that an "element was not found". Pay close attention to the disk layout/ boot order as well, the restore doesn’t allow for variation from the original layout.

Disk Size

Make sure the disks are as large as or larger than the original ones. Another possible other work around is using wbadmin and restoring each partition individually to where you want it and manually editing the disk with BCDEDIT to get it to boot properly would also work it’s a bit tedious and we would still need to shrink the data partition.

Wrong path to backups

Make sure you specify the correct path to restore from \\MyServer\MyBackupShare where it will than find the WindowsImageBackup folder by itself with the backup(s) of one or more servers in there. You can even copy backups from other locations to that folder; they will be found and listed.

Hyper-V Legacy NIC required for UNC path restores

Make sure your virtual machine in Hyper-V has a legacy nic when restoring a Windows 2008 server or the restore from the unc path will fail because network connectivity can’t be established otherwise and you’ll receive this error: "An internal error occurred. The following information might help you resolve the error: The network path was not found. (0×80070035)". See Figure 7.

Figure 7: Use a NIC that the Windows DVD has drivers for.

This is due to the fact that there are no synthetic NIC drivers in Windows 2008 DVD image. They are available in the Windows 2008 R2 DVDV image however.

I hope you realize you’ll need the correct share/NTFS permissions for the account you’ll use to access the backup share!

So we’re all set. In the next part we’ll discuss the actual restore.

Stories of Windows Backup, P2V, V2V & V2P

In the period 2008-2009 I did quite a lot of work on windows backup for Windows 2008 (R2) involving both physical and virtual (VMware Server & Hyper-V) servers. Apart from backing up certain servers we used it as a free Physical to Virtual (P2V) solution. We also did some Virtual to Virtual (V2V) migrations from VMware Server guests to Hyper-V. All this was in smaller business units or in SME’s where using the functionality inside Windows and free tools was important. Also some servers need nothing beyond what windows backup can deliver.

Part I: http://workinghardinit.wordpress.com/2010/01/27/using-windows-2008-r2-backups-to-go-virtual-2/
Part II: http://workinghardinit.wordpress.com/2010/02/01/using-windows-2008-r2-backups-to-go-virtual-ii/
Part III: http://workinghardinit.wordpress.com/2010/02/02/using-windows-2008-r2-backups-to-go-virtual-part-iii/

The benefit of such work is that one gets to learn a lot about the quirks and hiccups of a product. One also becomes aware of its capabilities. Windows Backup can do more than most people think. For that reason I’m quite happy (more than a lot of people) with the product, especially in Windows 2008 R2.

Just like for ntbackup.exe before, I scripted an entire backup solution around wbadmin.exe with logging, archiving, mail alerts etc. Once you have such an adaptable script you can nicely make due with Windows Backups in various environments. The next step is to write the script in PowerShell. That hasn’t been done yet because the script also needed to run on Windows Core which until R2 had no support for .NET and thus PowerShell.

Al lot has happened in the last quarter of 2009. On October 8th 2009 Sysinternals released disk2vhd tool. The current version now also has CLI support. See http://technet.microsoft.com/en-us/sysinternals/ee656415.aspx for more information. Recently some commercial vendors have also released free tools you can use for this. VizionCore made there vConverter available for free (http://www.vizioncore.com/free/vConverter/index.php). These tools greatly simplify the P2V or V2V scenarios. Still the “hands on approach” is a good way to learn about technology so I will publish my experiences in a series about windows backup, virtualization (P2V and V2V) and such.

As stated above I know people who dismiss Windows Backup as a valid solution. This is not always justified. But if you haven’t really figured out the tool, how would one know? Sure I use Backup Exec, CommVault and the like for larger environments. Sometimes I work in places with dozens to a hundred terabytes of data and multiple SAN’s. But that doesn’t mean one needs to dismiss all other approaches. Use what is needed where it is needed. I have saved several severs and data using the native backup tools in windows over my career. As recently as 10 days ago I did a bare metal recovery of a W2K8R2 that was messed up by user error. Just know what, where, how and when to use it. Isn’t that the same for any product or solution?

In this series we’ll describe a rather challenging disk configuration on the physical machine we need to virtualize and describe the desired end state of the virtual machine. We’ll discuss setting up the virtual machine to recover the backup. After that we’ll discuss the restore process. Finally we’ll turn our attention to sanitizing the disk layout & configuration. I hope you’ll find it both interesting and useful.