Microsoft Really Listens & Enhances NVSPBIND


Microsoft’s Hyper-V team really did amaze me today. They provided enhanced functionality in NVSPBIND http://blogs.technet.com/jhoward/archive/2010/01/25/announcing-nvspbind.aspx#comments based on passionate feedback on John Howard’s Blog.  Great going  Microsoft.

Crab the bits here: http://code.msdn.microsoft.com/nvspbind

Using Windows 2008 (R2) Backups to Go Virtual


Part I

The Challenge

This series describes a physical to virtual migration of a Windows Server 2008 SP2 Standard x64 on a Dell Power Edge one unit rack server with SATA disks (PE750). In fact that was the first production Windows 2008 server ever at that company, they really wanted to use the RTW FTP over SSL functionality. For this exercise we had no use of commercial P2V tools and the original server had to be kept running during the virtualization exercise. The aim was to achieve this with the free / built in tools as much as possible. The hands on approach also provides for an excellent learning opportunity. We have reconstructed our experiences for this article and will discuss the subject for both Windows 2008 and Windows 2008 R2.

Physical Machine

Disk Layout

The physical server has no raid controller. For redundancy reasons dynamic disks are used for software mirroring. The original 160 GB SATA disks were going to run out of space "very soon" during its lifespan they claimed, so they were swapped out for bigger ones. This was done one by one, with mirror rebuilding in between, and as such we "leapfrogged" to two 1TB disks. We then made a bigger data partition by concatenating the disks. That combined with some leading free disk space where OEM partition used to be on the original disks resulted in an interesting disk layout. The installation was healthy but the disk layout was not optimal and in fact, the server had way to much disk space. The owners indicated they would need vast amounts of disk space but this proved exaggerated and actual data needs are around 110 GB with some margin for growth. The disk layout is illustrated in below in Figure 1 and shows the disk configuration on the physical machine.

Windows Backup

Backups of this server are made to a file share using a script that automates Windows 2008 Backup (wbadmin.exe). The script does not only backup the server, it also creates backup archives. This is need as Windows 2008 Backup to a UNC path overwrites the last backup. So to keep a couple of weeks/months worth of backups we need to move them out to a save place. The script also collects logs about what’s happening for auditing and mails these to the system administrators indicating success or failure. Since the script also needs to work on Windows Server Core it’s VBScript (it was written late 2007) as .NET support and thus Power Shell is only available in Windows 2008 R2. We will use a backup of the physical machine to restore the server to a virtual machine.

Figure 1: The disk configuration on the physical machine. The bad quality is due to it being taken over a KVM over IP session from the original server!

Goal of the Migration

During the migration we wanted to sanitize the disk layout if possible. The task at hand was to migrate this server to a virtual machine using built in or free tools. Only if we could not find another way we could resort to commercial third party tools but only if we had them available at no extra cost (so a working trial or evaluation version is allowed). We could not use SCVMM 2008 as this server is member of a different domain (in the DMZ). The option of making it a member of our production domain is a no go due to security concerns and installing SCVMM2008 in the DMZ domain was not allowed. The original server also had to be online during the virtualization. We therefore opted to restore the backup of the physical server to a virtual server. During this process our goal was to:

  • resize the disks (no use wasting disk space),
  • separate the system drive from the data drive,
  • convert theWindows dynamic disks to basic disks (no need for dynamic disks and better support with a lot of hard disk tools ),
  • use fixed vhd files for better performance (note: dynamically expanding disks are not that noticeably slower I must say, in low IO situations with little growth. In W2K8R2 the difference is reported to be negligible)

The picture below, Figure 2, illustrates the desired (and achieved) outcome.

Figure 2: The desired disk configuration in a virtual machine.

Preparing a Virtual Machine to Restore the Backup

The entire process described below describes most of the problems you can encounter with restoring Windows 2008 Backups in general. Pay attention to the solutions to these problems and you’ll do just fine. I read a lot of postings on the internet about Windows 2008 backups not working properly but it does. You just need to learn about its limits and abilities. And yes, there are some "quirks" that you need to be aware of. Once you learn this the build in tool will serve you well. I think some of these quirks Microsoft should have documented better and some things are indeed missing functionality (most have been fixed in R2), which is unfortunate but not a show stopper. Remember the definition of engineering: building good working solutions with the means available.

Since we will restore the backup of the physical server to a virtual server, we need to prepare one. We use the Hyper-V to achieve this.

Create the Disks

We start by creating two disks. We need two disks because the original backup was of a server with two physical disks that were members of a software mirror, which also means they were dynamic disks. If you don’t have to disk to restore to you’ll get a cryptic error that the system image restore failed with some error details "Element not found. (0x80070490)". This is the same error you can when the disk layout/boot order is different on the restore machine than the original one. Pay attention to this.

Figure 3: Mind the disk layout and order or you’ll get into trouble.

Windows backup can only restore to disks of the same size or larger (unless you start doing partition restores manually and get handy with BCDEDIT). Since our physical are 1TB and we didn’t have the disk space for such large VHD files we needed to be a bit creative. We came up with the workaround of restoring to dynamic vhd files of 1048 GB (making sure it was indeed a bit larger than the original ones in the backup). The actual size of the data is not higher that 160 GB anyway so the dynamic VHD will not grow beyond that. The restore will see that the disk are large enough and function as planned. We’ll deal with the resizing of the disks, making them basic disks in windows and converting them to fixed vhd after the Servers has been restored to a virtual machine.

Figure 4: The details of the dynamically expanding vhd’s we create to restore the backup to.

Do make sure that you restart the virtual machine once with the new hard disks attach. This is to make sure the disk preparation is 100% finished. IF not a restore might fail with following error: "The system image restore failed." The error details state: "The computer needs to be restarted to finish preparing a hard disk for restore. To continue, restart your computer and run the restore again. (0x80042403)."

Figure 5: Error trying to restore a backup to new not yet prepared disk.

Allocate CPU & Memory

We configure the VM with CPU like the original one. As the original operating system is Windows 2008 x86 we won’t have many issues with virtualization due to the wrong HAL and such … things on that level have definitely become a lot easier and better than they used to be. We allocate enough memory to get the job done on my host.

Networking

We also configure the machine with a temporary legacy NIC. We use a legacy NIC for the restore when we restore a Windows 2008 server because the synthetic NIC aren’t included in the DVD image (you could build one using the WAIK) and we need a functional NIC to get to our backup on the file share. The Windows 2008 R2 image does contain the drivers you need for a synthetic NIC. To make live easy we have a DHCP server had out IP addresses. That saved us the trouble of configuring the TCP/IP settings manually during the restore process. No other NIC’s where configured so the restored VM wouldn’t conflict with the physical one still running in production.

Figure 6: The settings of the prepared VM to which we are going to restore the Windows Backup of the physical machine.

Recapture of Windows Backup in 2008 (R2) Restore Quirks

Mirrored Disks & Disk Layout

When restoring from a software mirrored dynamic disks make sure you have two disks to restore to or you’ll get an error indicating very helpfully that an "element was not found". Pay close attention to the disk layout/ boot order as well, the restore doesn’t allow for variation from the original layout.

Disk Size

Make sure the disks are as large as or larger than the original ones. Another possible other work around is using wbadmin and restoring each partition individually to where you want it and manually editing the disk with BCDEDIT to get it to boot properly would also work it’s a bit tedious and we would still need to shrink the data partition.

Wrong path to backups

Make sure you specify the correct path to restore from \\MyServer\MyBackupShare where it will than find the WindowsImageBackup folder by itself with the backup(s) of one or more servers in there. You can even copy backups from other locations to that folder; they will be found and listed.

Hyper-V Legacy NIC required for UNC path restores

Make sure your virtual machine in Hyper-V has a legacy nic when restoring a Windows 2008 server or the restore from the unc path will fail because network connectivity can’t be established otherwise and you’ll receive this error: "An internal error occurred. The following information might help you resolve the error: The network path was not found. (0x80070035)". See Figure 7.

Figure 7: Use a NIC that the Windows DVD has drivers for.

This is due to the fact that there are no synthetic NIC drivers in Windows 2008 DVD image. They are available in the Windows 2008 R2 DVDV image however.

I hope you realize you’ll need the correct share/NTFS permissions for the account you’ll use to access the backup share!

So we’re all set. In the next part we’ll discuss the actual restore.

NVSPBind released today!


Good news on long overdue support for NIC bindings on core was annouced today on  John Howards blog http://blogs.technet.com/jhoward/archive/2010/01/25/announcing-nvspbind.aspx with the release of NVSPBind. Messing arround in the registry to get this done when you have a lot of NIC is a recipe for disaster. Now let’s how we also get NIC/Connection binding support for Windows Core, as that is also one of the reasons that stopped Core deployments in my little part of the world.

Stories of Windows Backup, P2V, V2V & V2P


In the period 2008-2009 I did quite a lot of work on windows backup for Windows 2008 (R2) involving both physical and virtual (VMware Server & Hyper-V) servers. Apart from backing up certain servers we used it as a free Physical to Virtual (P2V) solution. We also did some Virtual to Virtual (V2V) migrations from VMware Server guests to Hyper-V. All this was in smaller business units or in SME’s where using the functionality inside Windows and free tools was important. Also some servers need nothing beyond what windows backup can deliver.

Part I: http://workinghardinit.wordpress.com/2010/01/27/using-windows-2008-r2-backups-to-go-virtual-2/
Part II: http://workinghardinit.wordpress.com/2010/02/01/using-windows-2008-r2-backups-to-go-virtual-ii/
Part III: http://workinghardinit.wordpress.com/2010/02/02/using-windows-2008-r2-backups-to-go-virtual-part-iii/

The benefit of such work is that one gets to learn a lot about the quirks and hiccups of a product. One also becomes aware of its capabilities. Windows Backup can do more than most people think. For that reason I’m quite happy (more than a lot of people) with the product, especially in Windows 2008 R2.

Just like for ntbackup.exe before, I scripted an entire backup solution around wbadmin.exe with logging, archiving, mail alerts etc. Once you have such an adaptable script you can nicely make due with Windows Backups in various environments. The next step is to write the script in PowerShell. That hasn’t been done yet because the script also needed to run on Windows Core which until R2 had no support for .NET and thus PowerShell.

Al lot has happened in the last quarter of 2009. On October 8th 2009 Sysinternals released disk2vhd tool. The current version now also has CLI support. See http://technet.microsoft.com/en-us/sysinternals/ee656415.aspx for more information. Recently some commercial vendors have also released free tools you can use for this. VizionCore made there vConverter available for free (http://www.vizioncore.com/free/vConverter/index.php). These tools greatly simplify the P2V or V2V scenarios. Still the “hands on approach” is a good way to learn about technology so I will publish my experiences in a series about windows backup, virtualization (P2V and V2V) and such.

As stated above I know people who dismiss Windows Backup as a valid solution. This is not always justified. But if you haven’t really figured out the tool, how would one know? Sure I use Backup Exec, CommVault and the like for larger environments. Sometimes I work in places with dozens to a hundred terabytes of data and multiple SAN’s. But that doesn’t mean one needs to dismiss all other approaches. Use what is needed where it is needed. I have saved several severs and data using the native backup tools in windows over my career. As recently as 10 days ago I did a bare metal recovery of a W2K8R2 that was messed up by user error. Just know what, where, how and when to use it. Isn’t that the same for any product or solution?

In this series we’ll describe a rather challenging disk configuration on the physical machine we need to virtualize and describe the desired end state of the virtual machine. We’ll discuss setting up the virtual machine to recover the backup. After that we’ll discuss the restore process. Finally we’ll turn our attention to sanitizing the disk layout & configuration. I hope you’ll find it both interesting and useful.

DELL PE1850 Domain Controller Upgrade to Windows 2008 R2


UPDATE May 20th 2010: DELL changed the location of the pdf’s I updated the links to them on this blog post as I see lost of people clicking on them. Hope this helps.

Just in case someone needs to do something similar, I’m posting some of issues I needed to resolve when I did an Active Directory upgrade for a partner’s IT department (September/October 2009). There was a need for Windows 2008 R2 DNS (for Direct Access), the Active Directory Recycle Bin as well as the desire to have as much servers as possible running the same OS to keep management easy. This is not an extensive manual of any tool used but it will give you some pointers.

The hardware in use was DELL. Seven Power Edge 1850’s servers spread around the country in corporate HQ and large branch offices. Those servers where 3 years old at the time and had 2 years support remaining on the contract. A hardware replacement or virtualization was not an option. So we needed to find out if the upgrade was possible. The good news was we could get our hands on a spare PE1850 for testing and if need be a "swing type" migration. But to reduce the work an in-place upgrade was preferred. The original installation of these domain controllers where x64 bit, otherwise that would have been a no go, W2K8R2 is x64 bit only. It was quite a smart and forward looking chap who did the original project. OK, I admit it was me, so this is blatant self-promotion. The anti-virus had W2K8R2 support; we got an updated agent for the UPS from the vendor, etc. It all looked pretty good.

I checked the following support documents on the dell website:

Microsoft® Windows Server® 2008 R2 for Dell™ PowerEdge™ Systems

Important Information Guide

https://support.dell.com/support/edocs/software/win2008/WS08_R2/en/IIG/WS08R2IG.pdf

Microsoft® Windows Server® 2008 R2 for Dell™ PowerEdge™ Systems

Installing Microsoft Windows Server 2008 R2

http://support.dell.com/support/edocs/software/win2008/WS08_R2/en/ING/WS08R2In.pdf

There found that all drivers and firm ware updates needed to support W2K8R2 on a PE1850 where available except for one and that was the driver for the raid controller, a PERC4e/si. That was a potential show stopper, but a driver was coming. So I kept a close eye on the DELL FTP site and around 3 September it showed up. Using the SUU 6.1 DVD or manually download installation packets I upgraded the firmware of the servers (BIOS, DRAC).

I also ran the upgrade advisor and found that we need to remove the Dell Open Manage Server Assistant version as the aging Dell® OpenManage Diagnostic Service used an unsigned driver (C:\Program Files (x86)\Dell\SysMgt\oldiags\packages\PORTACCESSOR64.sys). All potential problematic software, unneeded tools and drivers like video, anti-virus, UPS … where removed as well as this makes any upgrade process less risky.

No Native RAID Driver

As the DELL PERC 4e/Si is not natively supported by Windows 2008 R2 we need to use the DELL driver (R227150.exe)
from the FTP site. You could put the drivers in a subfolder $WinPEDriver$ on the root of a volume that Windows can find during the upgrade (hard disk, usb thumb drive…). Now to make absolutely sure we didn’t have any issues with the raid controller we decided to inject the driver in to WIM files to build a custom ISO. That might be redundant but we wanted to have an ISO with all needed drivers for disaster recovery purposes anyway. The drivers need to be injected into the boot.wim and thee install.wim files using DISM (Deployment Image Servicing and Management from the WAIK for Windows 7 en W2K8R2) see The Windows® Automated Installation Kit (AIK) for Windows® 7 @ http://www.microsoft.com/downloads/details.aspx?familyID=f1bae135-4190-4d7c-b193-19123141edaa&displaylang=en.

We used the x64 bit versions of the tools as our WIM files are x64 bit

The following commands inject the driver into the boot wim file’s two indexes:

DISM /MOUNT-WIM /WIMFILE:D:\InjectDriver\boot.wim /INDEX:1 /MOUNTDIR:D:\MOUNTHERE

DISM /IMAGE:D:\MOUNTHERE /ADD-DRIVER /DRIVER:D:\DRIVERS\DELL\R227150

DISM /UNMOUNT-WIM /MOUNTDIR:D:\MOUNTHERE /COMMIT

DISM /MOUNT-WIM /WIMFILE:D:\InjectDriver\boot.wim /INDEX:2 /MOUNTDIR:D:\MOUNTHERE

DISM /IMAGE:D:\MOUNTHERE /ADD-DRIVER /DRIVER:D:\DRIVERS\DELL\R227150

DISM /UNMOUNT-WIM /MOUNTDIR:D:\MOUNTHERE /COMMIT

Index 1 is the Microsoft Windows Preinstallation Environment (WinPE) and Index 2 is the actual Windows Setup that you can run when booted into WinPE. DISM has a command to find out more info about the image files: DISM.exe /Get-WimInfo. The documentation in the WAIK is quite good. Read it!

The following commands inject the driver into the install wim file. You need to do that for any index you want or need (Web, Standard, Enterprise, core and full install …) Just paste everything you need in a cmd file and you’re good to go.

DISM /MOUNT-WIM /WIMFILE:D:\InjectDriver\install.wim /INDEX:1 /MOUNTDIR:D:\MOUNTHERE

DISM /IMAGE:D:\MOUNTHERE /ADD-DRIVER /DRIVER:D:\DRIVERS\DELL\R227150

DISM /UNMOUNT-WIM /MOUNTDIR:D:\MOUNTHERE /COMMIT

Video Driver Injection Hiccup

As we wanted to have a good screen resolution for the sys admins we also embedded the video drivers. The screen resolution with the native driver wasn’t very good so we looked around to find one that would work. We found the Radeon 7000M driver (ATI_Radeon-7000M_A00_R177829.exe) on the DELL website and also injected them into the boot.wim and install.wim image files using DISM. That way we didn’t need to update the video drivers after installation. Cool. Most video drivers are packed twice. The trick is that you still need to expand the drivers after you extracted them from the installer using WinZip, 7Zip, WinRAR or whatever it is your use or prefer. Otherwise you’ll get an error like this after dism has found the inf:

Searching for driver packages to install…Found 2 driver package(s) to install.
Installing 1 of 2 – D:\DRIVERS\DELL\R177829\Driver\XP6A_INF\CA_58688.inf: Error – An error occurred. The driver package could not be installed.For more information, check for log files in the <windir>\inf folder of the target image.
Installing 2 of 2 – D:\DRIVERS\DELL\R177829\Driver\XP_INF\CX_58688.inf: Error – An error occurred. The driver package could not be installed.For more information, check for log files in the <windir>\inf folder of the target image.

For more information, check for log files in the <windir>\inf folder of the target image.  

Error 30

The command completed with errors. For more information, refer to the log file.

The DISM log file can be found at C:\Windows\Logs\DISM\dism.log

If you look in the dism.log you’ll find an error code like 0x8007001E as the cause of the error. This is rather cryptic. Any way you can prevent this by extracting them:

Expand D:\DRIVERS\DELL\R177829\Driver\XP6A_INF\B_58469\*.* D:\DriversExpanded

Copy the expanded files into a copy of the original folder structure to replace the original files. Make sure that you repeat this exercise for any subfolder as well if needed or you’ll only expand none or only a portion of the files. When you’ve done that you can add the drivers to the WIM files. Beware that adding those large video drivers can take rather long.

Now that we have added all drivers to the wim files we write the customized installation to an ISO file (oscdimg.exe, WAIK). We can burn this to a CD to add to the disaster recovery kit or mount the ISO using the DRAC media.

"C:\Program Files\Windows AIK\Tools\amd64\oscdimg.exe" -n -m -bD:\W2K8R2WithDellPerc4esiDriver\W2K8R2\boot\etfsboot.com "D:\W2K8R2WithDellPerc4esiDriver\W2K8R2" "D:\W2K8R2WithDellPerc4eDiAndRadeon7000M.iso"

Using the custom install we were able to upgrade the domain controllers fast and without any issues. All what was left to do after the upgrade was check if all was well with the DC. After that we had to clean up the post upgrade artifacts, install anti-virus, UPS software, Dell Management tools and provide for and schedule the backups.

So in in all we did 7 in place upgrades and now they have been running in native Windows 2008 Domain & forest functional level for over 4 months without any issues. Nice job. That’s the good thing about that partner, they always have some interesting jobs to do and helping out is always appreciated.

Setting Dates on Folders With PowerShell


A friend of mine with a Business Intelligence company asked me a favor. They have a lot of data (files & folders) that have to be copied around in the lab, at clients etc. This often leaves the date modified on the folders not reflecting the last modified date of the most recent modification in that folder’s sub structure.  This causes a lot of confusion in their processes, communication and testing.

The needed a script to correct that. Now they wanted a script, not an application (no installations, editable code). Good news they had a Windows machine (XP or higher) to run the code on and file sharing on Linux was using SAMBA so we could use PowerShell. VBScript/Java Script can only change dates on files using the Shell.Application object but NOT of folders. They also can’t directly call Windows API’s. First of all that’s “unmanaged code to the extreme” and using a  COM dll to get access to the Windows API violates the condition set out from the start.  But luckily PowerShell came to the rescue!

To accomplish the request we sort of needed to walk the tree backwards from all it’s branches back to the root. I’m no math guru so writing that sort of a reverse incursive algorithm wasn’t really an option. I decided to use plain good old recursion and count the depth of the folder structure to know how many times I needed to recursively parse through to get the correct modified date to “walk up” the folder structure. Here a snippet as a demo:


# Demo snippet

$root = "E:\TestRoot\TestDataStructure" # The folder structure to parse
$DeepestLevel = 0 # A counter to persist the deepest level found up to that moment
$File
$LevelCheck
$Return

#Loop through the folder structure recursively to determine the deepest level.
foreach ($folder in Get-ChildItem $root -recurse | Where-Object {$_.PsIsContainer -eq "True"})

    {
        $search = $folder.FullName
        Write-Host "Folder: $search"
        #Sort the returned objects by modified date and select the most recent (last) one
  $Return = Get-ChildItem $search | Sort-Object LastWriteTime | Select-Object -last 1
  Write-Host "Childe File/Subfolder most recently modified: $Return"
        #Check how deep is the current level
  $LevelCheck = $Return.FullName.split("\").Count -1
  # Compare above with deepest level foudn so far and set to new value if needed.
  if ($LevelCheck -gt $DeepestLevel) {$DeepestLevel = $LevelCheck}
  Write-Host "LevelCheck: $LevelCheck"
  Write-Host "DeepestLevel: $DeepestLevel"

    }
# Now actually recurively walk the folder structure x times where x = Deepestlevel
do {
  foreach ($folder in Get-ChildItem $root -recurse | Where-Object {$_.PsIsContainer -eq "True"})
        {
            $search = $folder.FullName
             #Sort the returned objects by modified date and select the most recent (last) one
  $Return = Get-ChildItem $search | Sort-Object LastWriteTime | Select-Object -last 1
            Write-Host "Child File or Folder most recently modified: " $Return.Fullname
            #Set the modified date on the parent folder to the one of most recent modified child object
  if ($Return -ne $null) {$folder.LastWriteTime = $Return.LastWriteTime}
            Write-Host "Parent folder " $search " last modified date set to " $Return.LastWriteTime
        }
  ; $DeepestLevel-- } #Counter -1
until ($DeepestLevel -eq 0)

Going through the folder structure to0 often is OK, going through it to0 few times is bad as it doesn’t accomplish the goal. So the logical bug in the code that loops once to much due to “\\” in the UNC path isn’t an issue. Not really elegant but very effective. The speed is also acceptable. It ran through 30,000 files, 20 GB in all in about a minute. Quick & Dirty does the trick sometimes.

The code will work with PowerShell 1.0/2.0 against a local and a UNC path as long as you have the correct permissions.

This is just a code snippet, not the production code with error handling, so please test it in a lab & understand what it does before letting it rip through your folder structures.

Cheers

SCVMM 2008 R2 Phantom VM guests after Blue Screen


UPDATE: Microsoft posted an SQL Clean Up script to deal with this issue. Not exactly a fix and let’s hope it gets integrated into SCVMM vNext :-) Look at the script here http://blogs.technet.com/b/m2/archive/2010/04/16/removing-missing-vms-from-the-vmm-administrator-console.aspx. There is a link to this and another related blog post in the newsgroup link at the bottom of this article as well.

I’ve seen an annoying hick up in SCVMM 2008 R2 (November 2009) in combination with Hyper-V R2 Live migration two times now. In both cases a Blue Screen (due to the “Nehalem” bug http://support.microsoft.com/kb/975530) was the cause of this. Basically when a node in the Hyper-V cluster blue screens you can end up with some (never seen all) VM’s on that node being is a failed/missing state. The VM’s however did fail over to another node and are actually running happily. They will even fail back to the original node without an issue. So, as a matter of fact, all things are up and running. Basically you have a running VM and a phantom one. There are just multiple entries in different states for the same VM. Refreshing SCVMM doesn’t help and a repair of the VM is not working.

While it isn’t a show stopper, it is very annoying and confusing to see VM guest in a missing state, especially since it the VM is actually up and running. You’re just seeing a phantom entry. However be careful when deleting the phantom VM as you’ll throw away the running VM as well they point to the same files. 

Removing the failed/Orphaned VM in SCVMM is a no go when you use shared storage like for example CSV as it points to the same files as the running one and it is visible to both the good VM node and the phantom one. Meaning it will ruin your good VM as well.

Snooping around in the SCVMM database tables revealed multiple VM’s with the same name but with separate GUIDS. In production it’s really a NO GO to mess around with the records. Not even as a last resort because we don’t know enough about the database scheme and dependencies. So I have found two workarounds that do work (used ‘m both).

  1. Export the good VM for save keeping, delete the missing/orphaned VM entry in SCVMM (one taking the good one with it if you didn’t export it) and import the exported VM again. This means down time for the VM guest. 
  2. Remove the Hyper-V cluster from VMM and re add it. This has the benefit that it creates no down time for the good VM and that the bad/orphaned one is gone. 

Searching the net didn’t reveal much info but I did find this thread that discusses the issue as well http://social.technet.microsoft.com/Forums/en-US/virtualmachinemanager/thread/1ea739ec-306c-4036-9a5d-ecce22a7ab85 and this one http://social.technet.microsoft.com/Forums/en/virtualmachinemgrclustering/thread/a3b7a8d0-28dd-406a-8ccb-cf0cd613f666

I’ve also contacted some Hyper-V people about this but it’s a rare and not well-known issue. I’ll post more on this when I find out.