Fixing Event ID 2002 “The policy and configuration settings could not be imported to the RD Gateway server "%1" because they are associated with local computer groups on another RD Gateway server”


Introduction

I was working on a little project for a company that was running TS Gateway on 32bit Windows 2008. The reason they did not go for x64 at the time was that they used Virtual Server as their virtualization platform for some years and not Hyper-V. One of the drawbacks was that they could not use x64 guest VMs. Since then they have move to Hyper-V and now also run Window Server 2012. So after more than 5 years of service and to make sure they did not keep relying on aging technology it is time to move to Windows Server 2012 RD Gateway and reap the benefits of the latest OS.

All in all the Microsoft documentation is not to bad, all be it that the information is a bit distributed as you need to use various tools to complete the process. Basically, depending on the original setup of the source server you’ll need to use the TS/RD Gateway Export & Import functionality, Web Deploy (we’re at version 3.0 at the time of writing) and the Windows Server Migration Tools that were introduced with Windows 2008 R2 and are also available in Windows Server 2012.

In a number of posts I’ll be discussing some of the steps we took. You are reading Part 3.

  1. x86 Windows Server 2008 TS Gateway Migration To x64 Windows Server 2012 RD Gateway
  2. Installing & using the Windows Server Migration Tools To Migrate Local Users & Groups
  3. TS/RD Gateway Export & Import policy and configuration settings a.k.a  “Fixing “The policy and configuration settings could not be imported to the RD Gateway server "TARGETSERVER" because they are associated with local computer groups on another RD Gateway server”

The Migration

Their is no in place upgrade from a x86 to an x64 OS. So this has to be a migration. No worries this is supported. With some insight, creativity and experience you can make this happen. The process reasonably well documented on TechNet, but not perfectly, and your starting point is right here RD Gateway Migration: Migrating the RD Gateway Role Service. These docs are for Windows Server 2008 R2 but still work for Windows Server 2012. Another challenge was we needed to also migrate their custom website used for the employees to check whether their PC is still on and if not wake it up or start it up remotely.

As you read in the previous part we had to migrate local users and groups that are also used by the TS Gateway x86 Windows 2008 Server as we still need those in the Windows Server 2012 RD Gateway. The Active Directory users and groups used in Connection Authorization Policies (CAP) and Resource Authorization Policies (RAP) require no further work.

TS/RD Gateway Export & Import

I’m not going to write on how to install  a brand new RD Gateway. That’s been done just fine by Microsoft and many other. I’ll just discuss the import and export functionality in the TS/RD Gateway manager and help you with a potential issue.

Export

This is easy. On the source TS/RD Gateways server you just right click the server in TS/RD Gateway Manager and select Export policy and configuration settings. In our case this is a Windows Server 2008 TS Gateway, X86, so 32 bit. But that doesn’t matter here.

image

Give the export file a name and chose a location.

image

You’ll get a notification of a successful import.

image

Import

Ordinarily you’ll launch the RD Gateway Manager Import policy and configuration settings feature and follow the wizard.image

Select a export file (from the old TS Gateway server) to import

 image

image

image

But instead of getting a success message you get an error.

image

If you are moving the TS/RDGateway to a new server and will not recuperate the name you’ll have to deal with the following issue: The policy and configuration settings could not be imported to the RD Gateway server "TARGETSERVER" because they are associated with local computer groups on another RD Gateway server.

This also manifests itself as an error in the TerminalServices-Gateway Admin log with Event 2002

image

“The policy and server configuration settings for the TS Gateway server "%1" could not be imported. This problem might occur if the settings have become corrupted.”

What? Corrupt? The Export went fine!? Now if you start researching this error you’ll end up here http://technet.microsoft.com/en-us/library/cc727351(v=ws.10).aspx which will tell you what to do if you get this error duse to a bad export but basically tells you you’re stuck otherwise. Not so! The solution to this is very easy, you just have to know it works. I found out by testing & verifying this. All you have to do is edit the source TS/RD Gateway export XML file.

Open op the XML file in notepad. Select Edit/Replace from the menu and do a Find "SOURCESERVER" with Replace All "TARGETSERVER" and use that XML File. Save the file and use that for the import.

image

So now start the import again with your edited file and after a while you’ll see that you have been successful this time.

image

If you are recuperating the name you will not have this issue as the name in the export file will match the host name. However as this server is domain joined to the same domain as the original one you’ll have to respect the order of taking down the original one, resetting it’s AD computer account and reusing it for then new RD gateway server. This is more risky as you take down the service before you switch over. With a new server and a DNS alias you can just swap between the old and the new one by simply updating the DNS record(s) or even recuperating the old IP address, that switch can go fast.

Installing & using the Windows Server Migration Tools To Migrate Local Users & Groups


Introduction

I was working on a little project for a company that was running TS Gateway on 32bit Windows 2008. The reason they did not go for x64 at the time was that they used Virtual Server as their virtualization platform for some years and not Hyper-V. One of the drawbacks was that they could not use x64 guest VMs. Since then they have move to Hyper-V and now also run Window Server 2012. So after more than 5 years of service and to make sure they did not keep relying on aging technology it is time to move to Windows Server 2012 RD Gateway and reap the benefits of the latest OS.

All in all the Microsoft documentation is not too bad, all be it that the information is a bit distributed as you need to use various tools to complete the process. Basically, depending on the original setup of the source server you’ll need to use the TS/RD Gateway Export & Import functionality, Web Deploy (we’re at version 3.0 at the time of writing) and the Windows Server Migration Tools that were introduced with Windows 2008 R2 and are also available in Windows Server 2012.

In a number of posts I’ll be discussing some of the steps we took. You are reading the second post.

  1. x86 Windows Server 2008 TS Gateway Migration To x64 Windows Server 2012 RD Gateway
  2. Installing & using the Windows Server Migration Tools To Migrate Local Users & Groups
  3. TS/RD Gateway Export & Import (Fixing Event ID 2002 “The policy and configuration settings could not be imported to the RD Gateway server "%1"" because they are associated with local computer groups on another RD Gateway server”)

As discussed in the first part we need to migrate some local users & groups on the TS Gateway (source) server as they are also being used for some special cases of remote access, next to Active Directory users & groups for the Remote Access Policies (RAPs) & Connection Authorization Policies (CAPs). The tool the use is the Windows Server Migration Tools. These were introduced with Windows 2008 R2 and are also available in Windows Server 2012.

Some people seem to get confused a bit about the installation of the Server Migration Tools but it’s not that hard. I have used these tools several times before in the past and they work very well. You just need to read up a bit on the the deployment part and once you have it figured out they work very well.

Installing the Windows Server Migration Tools on the DESTINATION Server

First we have to install the on the DESTINATION host (W2K12 in our case, the server to which you are migrating)). For this we launch Server Manager and on the dashboard select Manage and choose Add Roles & Feature.clip_image001

Navigate through the wizard until you get to Features. Find and select Windows Server Migration Tools. Click Next.clip_image001[4]

Click Install to kick of the installation.clip_image001[9]

After a while your patience will be rewarded.clip_image001[11]

Installing the Windows Server Migration Tools on the SOURCE Server

To install the Windows Server Migration Tools on the SOURCE server, you need to run the appropriate PowerShell command on the DESTINATION server. This is what trips people up a lot of the time. You deploy the correct version of the tools from the destination server to the source server, where you will than register them for use. Do this with an admin account that has admin privileges on both the DESTINATION & SOURCE Computer.

Start up the Windows Server Migration Tools from Server Manager, Tools.image

This launches the Windows Server Migration Tools PowerShell window.image

Our SOURCE server here is the32 bit (X86)  Windows 2008 TS Gateway Server. The documentation tells us the correct values to use for the parameters /architecture and /OS to use.

SmigDeploy.exe /package /architecture X86 /os WS08 /path \\SourcerServer\c$\sysadmin

Now before you run this command be sure to go to the ServerMigrationTools folder as the UI fails to do that for you.

Also this is PowerShell so use .\ in front of the command otherwise you’ll get the error below.image

While you want this:image

Now you have also deployed the correct tools to the SOURCE server, our old legacy TS Gateway Server. Next we need to register these tools on the SOURCE Server to be able to use them. You might have gotten the message already you need PowerShell deployed on the SOURCE Server as documented.

If you have PowerShell, launch the console with elevated permissions (Runs As Administrator) and run the following command: .\SmigDeploy.exeimage

Congratulations you are now ready to use the Windows Server Migration Tools! That wasn’t so hard was it? Smile

Using the Windows Server Migration Tools To Migrate Local Users & Groups

To export the local users and groups from the source TS/RD Gateway server you start up the Windows Server Migration Tools on the SOURCE server (see the documentation for all ways to achieve this) and run the following PowerShell command:
Export-SmigServerSetting -User All  -Group –Path C:\SysAdmin\ExportMigUsersGroups –Verboseimage

As you can see I elected to migrate all user accounts not just the enabled or disabled ones. We’ll sort those out later. Also note the command will create the folder for you.

To import the local users and groups to the target RD Gateway server you start up the Windows Server Migration Tools on the Destination server (see the documentation) , i.e. our new Windows Server 2012 RD Gateway VM.

image

and run the following PowerShell command:

Import-SmigServerSetting  -User Enabled  -Group -Path C:\SysAdmin\ExportMigUsersGroups -Verbose

Do note that the migrated user accounts will be disabled and have their properties set to "Next Logon". This means you will have to deal with this accordingly depending on the scenarios and communicate new passwords & action to take to the users.image

image

Do note that the local groups have had the local or domain groups/users added by the import command. Pretty neat.image

You’re now ready for the next step. But that’s for another blog post.

x86 Windows Server 2008 TS Gateway Migration To x64 Windows Server 2012 RD Gateway


Introduction

I was working on a little project for a company that was (still) running TS Gateway on a 32 bit  x86) version Windows 2008. The reason they did not go for x64 at the time of deployment was that they then used Microsoft Virtual Server as their virtualization platform and had been for some years.

In a number of posts I’ll be discussing some of the steps we took. You are reading the first one.

  1. x86 Windows Server 2008 TS Gateway Migration To x64 Windows Server 2012 RD Gateway
  2. Installing & using the Windows Server Migration Tools To Migrate Local Users & Groups
  3. TS/RD Gateway Export & Import (Fixing Event ID 2002 “The policy and configuration settings could not be imported to the RD Gateway server "%1"" because they are associated with local computer groups on another RD Gateway server”)

In those early days of W2K8 they had not yet switched to Hyper-V. As an early adopter I was able to show the the reliability of Hyper-V, so later they did.

One of the drawbacks of using Microsoft Virtual Server was that they could not use x64 guest VMs and that’s how they ended up with x86, which was still available for a server OS for W2K8. Since then they have move to Hyper-V and now also run Window Server 2012. Happy customers! So after more than 5 years of service and to make sure they did not keep relying on aging technology it is time to move to Windows Server 2012 RD Gateway and reap the benefits of the latest OS.

The Migration

Their is no in place upgrade from a x86 to an x64 OS. So this has to be a migration. No worries this is supported. With some insight, creativity and experience you can make this happen. The process reasonably well documented on TechNet, but not perfectly, and your starting point is right here RD Gateway Migration: Migrating the RD Gateway Role Service. These docs are for Windows Server 2008 R2 but still work for Windows Server 2012. Another challenge was we needed to also migrate their custom website used for the employees to check whether their PC is still on and if not wake it up or start it up remotely.

There are some things to take care of and I’ll address these I some later blog posts but I want you to take to heart this message. While an in place upgrade of an 32 bit X86 operating system to X64 version of that OS is not possible that doesn’t mean you’re in  a pickle and will have to start over from scratch. For many scenario’s there are migration paths and this is just one example of them, or better two combined,TS Gateway and a Website.

KB2770917 Updating Host & Guest Integration Services Components – Most Current Version Depends on Guest OS


As after installing http://support.microsoft.com/kb/2770917 on Windows Server 2012 Hyper-V hosts the integration services components are upgraded from 6.2.9200.16384 to 6.2.9200.16433. Windows Server 2012 guest get that same upgrade and as such also the newer integration services components. The guest with older OS version needed a different approach. So I turned to all the great PowerShell support now available for Hyper-V to automate this. Pretty pleased with the results of our adventures in PowerShell scripting I let the script go on Hyper-V cluster dedicated to test & development. As such there are some virtual machines on there running Windows 2003 SP2 (X64) and Windows XP SP3 (x86).  Guess what, after running my script and verifying the integration services version I see that those VM still report version 6.2.9200.16384 . No update. Didn’t my new scripting achievement “take” on those older guests?

So I try the install manually and this is what I get:

clip_image001

 

Why is there no upgrade for these guests?  Are they not needed or do I have an issue? So I mount the ISO and dig around in the files to find a clue in the date:

clip_image001[10]

 

It looks like there are indeed no update components in there for Windows XP/ W2K3. So then I look at the following registry key on the host where I normally use the Microsoft-Hyper-V-Guest-Installer-Win6x-Package value to find out what integration services version my hosts are running:

image

 

Bingo, there it seems indicated that we indeed need version for XP/W2K3 and version for W2K8(R2)/W2K12 and Vista/Windows 7/Windows 8. Cool, but I had to check if this was indeed as it should be and I’m happy to confirm all is well. Ben Armstrong (http://blogs.msdn.com/b/virtual_pc_guy/) confirmed that this is how it should be. There was a update needed for backup that only applied to Windows 8 / Windows Server 2012 guests.  As this fix was in a common component for Windows Server 2008 and later they all got the update. But for the older OS versions this was not the case and hence no update is need. Which is reflected in all the above. In short, this means your XP SP3 & W2K3SP2 VMs are just fine running the version of the integration services and are not in any kind of trouble.

This does leave me with an another task. I was planning to do enhancements to my script like feedback on progress, some logging, some better logic for clustered and non clustered environments, but now I have to also address this possibility and verify using the registry keys on the host which IC version I should check against per OS version. Checking against just for the one related to the host isn’t good enough Smile.

Know What Receive Side Scaling (RSS) Is For Better Decisions With Windows 8


Introduction

As I mentioned in an introduction post Thinking About Windows 8 Server & Hyper-V 3.0 Network Performance there will be a lot of options and design decisions to be made in the networking area, especially with Hyper-V 3.0. When we’ll be discussing DVMQ (see DMVQ In Windows 8 Hyper-V), SR-IOV in Windows 8 (or VMQ/VMDq in Windows 2008 R2) and other network features with their benefits, drawbacks and requirements it helps to know what Receive Side Scaling (RSS) is. Chances are you know it better than the other mentioned optimizations. After all it’s been around longer than VMQ or SR-IOV and it’s beneficial to other workloads than virtualization. So even if you’re a “hardware only for my servers” die hard kind of person you can already be familiar with it. Perhaps you even "dislike” it because when the Scalable Networking Pack came out for Windows  2003 it wasn’t such a trouble free & happy experience. This was due to incompatibilities with a lot of the NIC drivers and it wasn’t fixed very fast. This means the internet is loaded with posts on how to disable RSS & the offload settings on which it depends. This was done to get stability or performance back for application servers like Exchange and others applications or services.

The Case for RSS

But since Windows 2008 these days are over. RSS is a great technology that gets you a lot better usage of out of your network bandwidth and your server. Not using RSS means that you’ll buy extra servers to handle the same workload. That wastes both energy and money. So how does RSS achieve this? Well without RSS all the interrupt from a NIC go to the same CPU/Core in multicore processors (Core 0).  In Task Manager that looks not unlike the picture below:

image

Now for a while the increase in CPU power kept the negative effects at bay for a lot of us in the 1Gbps era. But now, with 10Gbps becoming more common every day, that’s no longer the case. That core will become the bottle neck as that poor logical CPU will be running at 100%, processing as much network interrupts in can handle, while the other logical CPU only have to deal with the other workloads. You might never see more than 3.5Gbps of bandwidth being used if you don’t use RSS. The CPU core just can’t keep up. When you use RSS the processing of those interrupts is distributed across al cores.

With Windows 2008 and Windows 2008 R2 and Windows 8 RSS is enabled by default in the operating system. Your NIC needs to support it and in that case you’ll be able to disable or enable it. Often you’ll get some advanced features (illustrated below) with the better NICs on the market. You’ll be able to set the base processor, the number of processors to use, the number of queues etc. That way you can divide the cores up amongst multiple NICs and/or tie NICs to specific cores.

image

image

So you can get fancy if needed and tweak the settings if needed for multi NIC systems. You can experiment with the best setting for your needs, follow the vendors defaults (Intel for example has different workload profiles for their NICs) or read up on what particular applications require for best performance.

Information On How To Make It Work

For more information on tweaking RSS you take a look at the following document http://msdn.microsoft.com/en-us/windows/hardware/gg463392. It holds a lot more information than just RSS in various scenarios so it’s a useful document for more than just this.

Another good guide is the "Networking Deployment Guide: Deploying High-Speed Networking Features". Those docs are about Windows 2008 R2 but they do have good information on RSS.

If you notice that RSS is correctly configured but it doesn’t seem to work for you it’ might be time to check up on the other adaptor offloads like TCP Checksum Offload, Large Send Offload etc. These also get turned of a lot when trouble shooting performance or reliability issues but RSS depends on them to work. If turned off, this could be the reason RSS is not working for you..

Upgrading Windows Server 2008R2 Editions With DISM


When an environment evolves (growth, mergers, different needs) you have might very well have resource needs above and beyond the  limits of the original Windows edition that was installed. Scaling out might not the right (or possible) solution you so scale up is alternative option. Today with Windows Server 2008 R2 this is very easy. However, again and again I see people resorting labor intensive and often tedious solutions. Some go the whole 9 yards and do a complete clean install and migration. Others get creative and do a custom install with the windows media to achieve an in place upgrade. But all this isn’t needed at all. Using DISM (Windows Edition-Servicing Command-Line Options) you can achieve anything you need and every role, feature, app on your server will remain in good working condition. Recently I had to upgrade some standard edition Hyper-V guest servers to the enterprise edition to make use of more than 32 GB of RAM. Another reason might be to move from Windows Server 2008 R2 Enterprise Edition to Data Center Edition for hyper-v host to make use of that specific licensing model for virtual machines.

Please note the following:

  • You can only do upgrades. You CANNOT downgrade
  • The server you upgrade cannot be a domain controller (demote, upgrade, promote)
  • This works on Standard, Enterprise edition, both full & core installations.
  • You cannot switch form core to full or vice versa. It’s edition upgrade only, not  for switching type of install.

This is how to find the possible target editions for your server:

C:\Windows\system32>DISM /online /Get-TargetEditions

Deployment Image Servicing and Management tool
Version: 6.1.7600.16385

Image Version: 6.1.7600.16385
Editions that can be upgraded to:

Target Edition : ServerDataCenter
Target Edition : ServerEnterprise

The operation completed successfully.

So I went to Enterprise Edition by executing this process takes some time but is painless but for one reboot.

C:\Windows\system32>Dism /online /Set-Edition:ServerEnterprise /ProductKey:489J6-VHDMP-X63PK-3K798-CPX3Y

Deployment Image Servicing and Management tool
Version: 6.1.7600.16385

Image Version: 6.1.7600.16385

Starting to update components...
Starting to install product key...
Finished installing product key.

Removing package Microsoft-Windows-ServerStandardEdition~31bf3856ad364e35~amd64~~6.1.7601.17514
[==========================100.0%==========================]
Finished updating components.

Starting to apply edition-specific settings...
Restart Windows to complete this operation.
Do you want to restart the computer now (Y/N)?

You either use a MAK key (if you don’t have a KMS server) or the default key for your volume license media. When you have KMS in place (and the matching server group KMS key A, B, or C) the activation will be done automatically and transparent for you. Standard trouble shooting applies if you run into an issue there.

These are the public keys for use with a KMS server:

  • Windows 7 Professional – FJ82H-XT6CR-J8D7P-XQJJ2-GPDD4
  • Windows 7 Professional N – MRPKT-YTG23-K7D7T-X2JMM-QY7MG
  • Windows 7 Enterprise – 33PXH-7Y6KF-2VJC9-XBBR8-HVTHH
  • Windows 7 Enterprise N – YDRBP-3D83W-TY26F-D46B2-XCKRJ
  • Windows 7 Enterprise E – C29WB-22CC8-VJ326-GHFJW-H9DH4
  • Windows Server 2008 R2 HPC Edition – FKJQ8-TMCVP-FRMR7-4WR42-3JCD7
  • Windows Server 2008 R2 Datacenter – 74YFP-3QFB3-KQT8W-PMXWJ-7M648
  • Windows Server 2008 R2 Enterprise – 489J6-VHDMP-X63PK-3K798-CPX3Y
  • Windows Server 2008 R2 for Itanium-Based Systems – GT63C-RJFQ3-4GMB6-BRFB9-CB83V
  • Windows Server 2008 R2 Standard – YC6KT-GKW9T-YTKYR-T4X34-R7VHC
  • Windows Web Server 2008 R2 – 6TPJF-RBVHG-WBW2R-86QPH-6RTM4

Don’t worry this is public information (KMS Client Setup Keys), these will only activate if you have a KMS server and the to key make that KMS server work.

Either way there is no need for reinstall & migration or upgrade installation in for a simple upgrade scenario So do your self a  favor and always check if you can use DSIM to achieve your goals!

Virtualization with Hyper-V & The NUMA Tax Is Not Just About Dynamic Memory


First of all to be able to join in this little discussion you need to know what NUMA is and does. You can read up on that on the Intel (or AMD) web site like http://software.intel.com/en-us/blogs/2009/03/11/learning-experience-of-numa-and-intels-next-generation-xeon-processor-i/ and http://software.intel.com/en-us/articles/optimizing-software-applications-for-numa/. Do have a look at the following SQL Skills Blog http://www.sqlskills.com/blogs/jonathan/post/Understanding-Non-Uniform-Memory-AccessArchitectures-(NUMA).aspx which has some great pictures to help visualize the concepts.

What Is It And Why Do We Care?

We all know that a CPU contains multiple cores today. 2,4,6,8,12,16 etc. cores. So in terms of a physical CPU we tend to talk about a processor that fits in a socket and about cores for logical CPUs. When hyper threading is enabled you double the logical processors seen and used. It is said that Hyper-V can handle hyper threading so you can leave it on. The logic being that it will never hurt performance and can help to improve it. I suggest you test it Smile as there was a performance bug with it once.  A processor today contains it own memory controller and access to memory from that processor is very fast. The NUMA node concept is older than the multi core processor technology but today you can state that a NUMA node translates to one processor/socket and all cores contained in that processor belong to the same NUMA node. Sometimes a processors contains two NUMA node like the AMD 12 core processors. In the future, with the ever increasing number of cores, we’ll perhaps see even more NUMA nodes per processor. You can state that all Intel processors since Nehalem with Quick Path Interconnect and AMD processors with Hyper-Transport are NUMA processors. But To be sure, check with your vendors before buying. Assumptions right?

Beyond NUMA nodes there is also a thing called processor groups which help Windows to use more than 64 logical processors (its former limit) by grouping logical processors into groups of which Windows handle 4 meaning in total Windows today can support 4*64=256 logical processors. Due to the fact that memory access within a NUMA node is a lot faster than between NUMA nodes you can see where a potential performance hit is waiting to happen. I tried to create a picture of this concept below. Now you know why I don’t make my living as a graphical artist Eye rolling smile

imageimage

 

To make it very clear NUMA is great and helps us in a lot of ways. But under certain conditions and with certain applications it can cause us to take a (serious) performance hit. And if there is anything certain to ruin a system administrators day than it is a brand new server with a bunch of CPUs and loads of RAM that isn’t running any better (or worse?) than the one you’re replacing. Current hyper visors like Hyper-V are NUMA aware and the better servers like SQL Server are as well. That means that under the hood they are doing their best to optimize the CPU & memory usage for performance. They do an very good job actually and you might, depending on your environment never, ever know of any issue or even the existence of NUMA.

But even with a NUMA knowledgeable hyper visor and NUMA aware applications you run the risk of having to go to remote memory. The introduction of Dynamic Memory in Windows 2008 R2 SP1 evens increases this likelihood as there is a lot of memory reassigning going on. Dynamic Memory actually educated a lot of Hyper-V people on what NUMA is and what to look out for. Until Dynamic Memory came on the scene, and the evangelizing that came with it by Microsoft, it was "only" the people virtualizing  SQL Server or Exchange & other big hungry application that were very aware of NUMA with its benefits and potential draw backs. If  you’re lucky the application is NUMA aware, but not all of them are, even the big names.

A Peak Into The Future

As it bears on this discussion, what is interesting that leaked screenshots from Hyper-V 3.0 or vNext  … have NUMA configuration options for both memory and CPU at the virtual machine level! See Numa Settings in Hyper-V 3.0 for a picture. So the times that you had to script WMI calls (see http://blogs.msdn.com/b/tvoellm/archive/2008/09/28/looking-for-that-last-once-of-performance_3f00_-then-try-affinitizing-your-vm-to-a-numa-node-.aspx) to assign a VM to a NUMA node might be over soon (speculation alert) and it seems like a natural progression from the ability to disable NUMA with W2K8R2SP1 Hyper-V in case you need it to avoid NUMA issues at the Hyper-V host level. Hyper-V today is already pretty NUMA aware and as such it will try to get all memory for a virtual machine from a single NUMA node and only when that can’t be done will it span across NUMA nodes. So as stated, Hyper-V with Windows Server 2008 R2 SP1 can prevent this form happening as we can disable NUMA for a Hyper-V host now. The downside is that you can’t get more memory even if it’s available on the host.

NumaSpanning

A working approach to reduce possible NUMA overhead is to limit the number of CPUs to 2 as this gives the largest amount of memory to the CPUs, in this case 50%. 4 CPUs only control 25%, etc.So with more CPU (and NUMA nodes) the risk of NUMA spanning is getting bigger very fast. For memory intensive applications scaling out is the way to go. Actually you could state that we do scale up the NUMA nodes per socket (lots of cores with the most amount of direct accessible memory possible) and as such do not scale up the server. If you can keep your virtual machines tied to a single CPU on a dual socket server to try and prevent any indirect memory access and thus a performance hit. But that won’t always work. If you ever wondered when an 8/12/16 core CPU comes in handy, well voila … here a perfect case: packing as many cores on a CPU becomes very handy when you want to limit sockets to prevent NUMA issues but still need plenty of CPU cycles. This should work as long as you can address large amounts of RAM per socket at fast speeds and the CPU internally isn’t cut up into to many multiple NUMA nodes, which would be scaling out NUMA node in the same CPU and we don’t want that or we’re back to a performance penalty.

Stacking The Deck

One way of stacking the deck in your favor is to keep the heavy apps on their own Hyper-V cluster. Then you can tweak it all you want to optimize for SQL Server, Exchange, … etc. When you throw these virtual machines in your regular clusters or for crying out loud on a VDI cluster your going to wreak havoc on the performance. Just like mixing server virtualization & VDI is a bad idea (don’t do it), throwing vCPU hungry, memory hogging servers on those cluster is just killing of performance and capacity of a perfectly good cluster. I have gotten into arguments over this as some thing one giant cluster for whatever need is better. Well no, you’ll end up micro managing placement of VM with very different needs on that cluster effectively “cutting” it up in smaller “cluster parts”. Now is separate clusters for different needs always the better approach? No, it depends, If you only have some small SQL Server needs you  get away with one nice cluster. It depends, I know, the eternal consultants answer, but I have to say it. I don’t want to get angry mails from managers because someone set up a 6 node clusters for a couple of SQL Server Express databases Winking smile There are also concepts called testing, proof of concept, etc. It’s called evidence based planning. Try it, it has some benefits that become very apparent  when you’re going to virtualize beefy SQL Server, SharePoint and Exchange servers.

How do you even know it is happening apart from empirical testing. Aha, excellent question! Take a look at the "Hyper-V VM Vid Numa Node" counter set and read this blog entry by on this subject http://blogs.msdn.com/b/tvoellm/archive/2008/09/29/hyper-v-performance-counters-part-five-of-many-hyper-vm-vm-vid-numa-node.aspx. And keep an eye on the event log for http://technet.microsoft.com/hi-in/library/dd582929(en-us,WS.10).aspx (for some reason there is no comparable entry for W2K8R2 on TechNet)

Conclusions

To conclude, all of the above people is why I’m interested in the some of the latest generation of servers. The architecture of the hardware allows for a the processor to address twice the "normal" amount of memory when you only put dual CPUs on a quad socket motherboard. The Dell PowerEdge R810 and the M910 have this and it’s called a FlexMemory Bridge and that allows more memory to be available without a performance hit. They also allow for more memory per socket at higher speeds. If you put a lot of memory directly addressable to one CPU you see a speed drop. A DELL R710 with 48 GB of RAM runs at 1033 MHZ  but put 96 GB in there and you fall back to 800 Mhz. So yes, bring on those new quad socket motherboards with just 2 sockets used, a bunch of fast direct accessible memory in a neat 2 unit server package with lost of space for NIC cards & FC HBAs if needed. Virtualization heaven :-) That’s what I want so I can give my VMs running SQL Server 2008 R2 & "Denali" (when can I call it SQL Server 2012?) a bigger amount of direct accessible memory form their NUMA node. This can be especially helpful if you need to run NUMA unaware applications like SAP or such. Testing is the way to go for knowing how well a NUMA aware hyper visor and a NUMA aware application figure out the best approach to optimize the NUMA experience together.  I’m sure we’ll learn more about this as more and more information becomes available and as technology evolves.  For now we optimize for performance with NUMA where we can, when we can with what we have :-) For Exchange 2010 (we even have virtualization support for DAG mailbox servers now as well) scaling out is easier as we have all the neatly separate roles and control just about everything down to the mail client. With SQL Server applications this is often less clear. There is a varied selection of commercial and home grown applications out there and a lot of them can’t even scale out, only up. So your mileage of what you can achieve may vary. But for resource & memory heavy applications under your control, for now, scaling out is the way to go.

Introducing 10Gbps With A Dedicated CSV & Live Migration Network (Part 2/4)


This is a 2nd post in a series of 4. Here’s a list of all parts:

  1. Introducing 10Gbps Networking In Your Hyper-V Failover Cluster Environment (Part 1/4)
  2. Introducing 10Gbps With A Dedicated CSV & Live Migration Network (Part 2/4)
  3. Introducing 10Gbps & Thoughts On Network High Availability For Hyper-V (Part 3/4)
  4. Introducing 10Gbps & Integrating It Into Your Network Infrastructure (Part 4/4)

Introduction

In this post we continue along the train of thought we set in a previous blog post “Introducing 10Gbps Networking In Your Hyper-V Failover Cluster Environment (Part 1/4)”. Let’s say you want to set up a Hyper-V cluster for SQL Server virtualization. Your business & IT manager told you the need to provide them with the best performance you can get. They follow up on that statement with a real budget so you can buy high end servers (blades or rack) and spec them out optimally for SQL Server. You take into consideration NUMA issues, vCPU:pCPU ratios, SQL memory demands, the current 4 vCPU limit in hyper-V, etc. By the way, this will be > 16vCPU with Windows Server 8, which leads me to believe the 64GB memory ceiling for virtual machines will also be broken. But for now this means that with regard to CPU & memory you’ve done all you can. That leaves only networking and IO to deal with. Now the IO is food for another & very extensive discussion, but basically you have to design that around the needs of the application(s) or you’ll be toast. The network part is what we’ll tackle here.

Without going into details, what does a Hyper-V cluster need in terms of networking?

Who/What

Function

Traffic

Connection Type

Host Management

Hyper-V host connectivity.

Relatively low bandwidth. But don’t forget about deploying VMs or backups.

Public

       

VM Network

Provides network connectivity to the VMs

Very dependent on the VMs using it.

Dedicated Hyper-V

Cluster Heartbeat

Internal cluster communication to determine the status of other cluster nodes

Not much traffic but low latency or cluster might think it’s in trouble due to dropped packets. OK to combine with CSV.

Private Cluster Network

Cluster Shared Volume (CSV)

For updating CSM metadata & scenarios where redirected I/O is required

Mostly idle. When in redirected I/O it demands high bandwidth & low latency required.

Private Cluster Network

Live Migration

Used to transfer the running VM’s from one cluster node to another

Mostly idle. When Live Migrating it demands high bandwidth & low latency required.

Private Cluster Network

Host Management: It is fine to leave this on 1Gbps, unless you have a need to deploy massive amounts of VMs or you backups are consuming all bandwidth. If so consider dedicated NICs for those roles and/or 10Gbps. Also note that you might be able to leverage your SAN for virtual machine deployment / backups.

VM Network: Use multiple “single” NICs or NIC teams to spread both the load and the risk. Remember that you can lose the host management or CSV network of a node, without affecting your virtual machine connectivity but not the virtual machine network(s). So don’t put all your eggs in one basket. So do consider multiple NICs and NIC teaming. Do remember that there are other bottle necks than bandwidth to a virtual machine running apps so don’t go completely overboard as there is no single magic bullet here for virtual machine performance. 2 or 3 will do perfectly fine. What about backups in the guest? Yes, that’s an extra burden but there are better solutions than that and if you hit and bandwidth issue with guest based backups it’s time to investigate them seriously. As you will see in these series I’m not a mincer with NIC ports but there’s no need to have one for every 2 Virtual machines. If you have really high bandwidth needs consider 10Gbps, not a truck load of NIC ports.

Heartbeat: Due to the mostly moderate needs it is often combined with the CSV traffic.

Cluster Shared Volume (CSV): Well you have the need for metadata of the clustered shared volumes. But that’s not all. You also have redirected access when you’re doing backups, defragmenting your CSV storage or when the storage paths are unavailable. So go for 10Gbps when you can, especially since this is your backup path for Live Migration traffic!

Side Note: Don’t say that Redirected Access over the CSV network will never happen when you have redundant storage paths. We’ve seen it happen in an environment with dual FC HBA cards, dual SAN controllers and the works. Redirected Access saved our service availability during that event! What happened exactly and how it all ties together is a long story and complicated but in essence an arbitrated loop management module when haywire and caused a loop, the root cause of this was a defective disk. When that event was over one of the controllers went nuts and decided this wasn’t his cup of tea and called it a day. Guess what? Some servers could not failover to the other controller as something went wrong in the internal workings of the SAN itself, dual HBA didn’t help here. How did our services stay available? Thanks to Redirected Access. It was at 1Gbps speeds so that hurt a little but we kept ‘m running. Our vendor worked through this with us but things where pretty bad and it was pucker time. However this is one example where we kept our services running for 24 hours (whilst working at the issue with the vendor) via redirected access. The bad thing was we needed to take the spare controller of line & restart both to get the replacement controller to be recognized, yes a complete shutdown of the cluster nodes to restart both SAN controllers. I still remember the mail I send and the call I made to management that is was shutting down the business for 30 minutes. But it was not because of Hyper-V, quite the opposite; it helped us out a lot!

Also note that when you run software VSS based backups and disk defragmentation on your CSV storage you’ll be running in Redirected Access mode. Also see http://workinghardinit.wordpress.com/2011/06/02/some-feedback-on-how-to-defrag-a-hyper-v-r2-cluster-shared-volume/ Some Feedback On How to defrag a Hyper-V R2 Cluster Shared Volume

Live Migration: The bigger and better the pipe the faster Live Migration gets done. With high density or resource (memory) intensive servers this becomes a lot more important. Think of SQL Server, Exchange consuming 16, 24, 32 or more GB of memory. So do consider 10Gbps.

iSCSI: As we are using Fiber Channel in our SAN we did not include iSCSI in the networking needs table above. Now I do want to draw your attention to the need for iSCSI in the virtual machines themselves. This is needed for clustering within the virtual machines. Today this is almost a requirement as clustering in the guest becomes more and more important. You’ll need at least two NIC ports in production for this, if possible in on two separate cards for ultimate redundancy. Now as a best practice we won’t share the iSCSI NICS between the hosts and the guests. I do this in the lab but won’t have it in production. So that could mean at least two more NIC ports. With 10Gbps you’ll have ample performance but depending on your IO needs you might want 4 if you’re using 1Gbps so those NIC numbers are rising fast.

What

Function

Traffic

Connection Type

iSCSI Guest

Virtual machine shared storage.

High bandwidth need, low latency is required to get good I/O

Dedicated to Hyper-V

iSCSI host

Host shared storage

High bandwidth need, low latency is required to get good I/O

Excluded from cluster, dedicated to the host.

What to move to 10Gbps?

Cool, you think, let’s throw some 10Gbps NICs & switches into our network. After that, depending on the rest of your network equipment & components, your virtual machines might be able to talk to other virtual and physical servers on the network at speeds up to 10Gbps or at least 1Gbps. I kind of hope that none of you are running 100 Mbps in your server racks today. And last but not least, with your 10Gbps network you’ll be able to do get the best performance for your CSV and Live Migration traffic. Life is good!

Until your network engineer hears about your plans. All of a sudden it’s no so cool anymore. You certainly woke the network people up! They’re nervous now they have seen all the double (redundancy) lines you’ve drawn on your copy of the schema representing the rack / server room network. They start mumbling things about redundancy, loops, RSTP, MSTP, LAG, stacking and a boatload other acronyms that sound like you’ve heard ‘m before but can’t quite place. They also talk about doom and gloom scenarios that might very well bring down the network. So unless you are the network admin you should dust of your communication skills and get them on board. So for your sake I hope they’re not the kind of engineers that states that most network problems that can’t be solved by removing servers and applications that ruin the nirvana of their network design. If so they’ll be vary weary of that “virtual switch” you’re talking about as well.

The Easy Way Out – A Dedicated CSV & Live Migration Network

Let’s say that you need a lot more time to get to a fully integrated solution for the 10Gbps network architecture figured out and set up. But your manager states you need to improve the Live Migration and other cluster network speeds today. What are your options? Based on the above information your boss is right, the networks that will benefit the most from a move to 10Gbps are CSV and Live Migration (and Heart Beat that piggy backs along with CSV). Now you have to remember that those cluster networks (subnets/VLANs) are for the Heart Beat, CSV and Live Migration cluster traffic only. So basically the only requirement you have is that these run on separate subnets/VLANs (to present them as distinct networks to your failover cluster) and that every node of the cluster can communicate over those subnets/VLANs. This means that you can leave the switches for those networks completely isolated from the rest of the network as shown in the picture below. I used some very common and often used DELL PowerConnect switches (5424, 6248, 8024F) in some scenario drawings for this blog series. They could make that 8024F an unbeatable price/quality deal if they would make them stackable. The sweet thing about stackable switches is that you can do Active-Active NIC teaming across switches rather than active-passive. I never went that way as I’m waiting to see what virtual switch innovations Hyper-V 3.0 will bring us. You see I’m a little cheap after all

But naturally, feel free think about these scenarios with your preferred ProCurves, CISCO, Juniper, NetGear … switches in mind. Smile

clip_image002[1]

Suddenly things are cool again. The network people get time to figure out an integrated & complete long term solution and you can provide you nodes with 10Gbps for cluster only traffic. By a couple of 10Gbps switches & NICs and you’re on your way. Is this a good idea? I can’t make that call for you. I just provide some ideas. You decide.

The Case For Physically Isolating Them

Now you might wonder if this isn’t very wasteful in resources. Well not necessarily. If your cluster is big enough, let’s say 12-16 nodes or if you have a couple of clusters (4 clusters with 5 nodes for example) this might be not overly expensive. Unless you’re on a converged network, you do (I hope) the same for your storage networks, isolate them that is. You have to when you’re using fiber and you’d better do it when using iSCSI. It provides for the best performance and less complex switch configurations. Remember I mentioned that high availability requires some complexity. Try to keep that complexity as low as possible and when you introduce complexity make sure you can manage it. This serves two purposes. One is making sure that the complexity doesn’t ruin you high availability and two is that you’ll be happy you did it when it comes to trouble shooting and fixing issues. Now you might say that this ruins the concept of converged networks. Academically this is true but when you are filling up ports on switches for a single purpose there is no room for anything else anyway. Don’t lose sight of the aim of a converged network. That is to have the ability to use the same hardware/technology when possible for multiple needs. This gives you options and capabilities where and when needed. It’s not about always using all technology and protocols on each and every switch. Don’t forget also that you’ll need to address QOS/Performance on a converged network per type of traffic. There is also the fact that in brown field scenario’s you’re dealing with replacing a part of the infrastructure and this example is a good way to get 10Gbps where needed and not making any change on the existing network infrastructure. This reduces risk and impact. As a matter effect if you plan this right you can do this without service interruption. That means going node by node (maintenance mode, evacuate all VMs), moving the CSV network first for example and only then the Live Migration network. You’re leveraging the ability of the cluster networks to take on each other’s role here to achieve this.

Another good reason to physically isolate the networks is for security. There was an exploit for manipulating VMs during live migrations in 2008 (http://www.eecs.umich.edu/techreports/cse/2007/CSE-TR-539-07.pdf). You can protect against this via very careful switch configuration and VLAN design. But isolating the switches is very easy, clean and effective as well. Overkill? I don’t know, but perhaps not if you do works for intelligence agencies.

Ethernet Out-of-Band (OOB) Port For Management

Don’t forget you still need to be able to manage those switches but today, in this class of equipment you get an Ethernet Out-of-Band (OOB) port for that. This one you can safely uplink to your regular management network. So if you really don’t need communication with the rest of the network you have no functional reason not to isolate them.

Money, Cost? No Value!

Still you think, isn’t this very expensive? Well look at the purpose. Manageable complexity, high availability and your management stated to eliminate, where possible, any limitation on performance and approved the budget for it all. Put this into perspective. The SQL Server data center editions running on these clusters, combined with the cost of development & maintenance of the databases and applications relying on this infrastructure put those extra € spent on a couple of switches really into perspective. On top of that you’re not wasting those switches. When the network people get their plans finished they’ll be integrated into the final solution if still needed and possible. Don’t forget that you might use all ports for just cluster traffic depending on the number of hosts you have! So even without integrating them into the rest of the network, you’re still getting very solid results. On top of that, sometimes you get to build solutions where budget is not the first, last and only concern. Sweet! I do know some people who’ll call me a money wasting nut case J. But get real, when you’re building high available, highly performing failover clusters and you’re in a discussion about the cost of a couple of NIC ports and you are going to adjust your design over that, perhaps you have a sponsorship issue. Put in into perspective. Hyper-V cluster are not a competition where the one who uses the least NIC ports/cards and switch ports/ switches wins. That’s why it hurts when I see designs like this claiming victory:

image

What I want to see is more like this:

image

But that will never fit into a blade design! Really? Have you seen the blades like the DELL M910? It’s a beast, comparable to the R810. It’s was the first blade I really felt like buying. Cisco also entered that market with guns drawn and is pushing HP to keep performing. So Again put the NIC/Switch and NIC port : Switch Port count into perspective against what you’re trying to achieve. To quote Anton Ego “… you know what I’m craving? A little perspective, that’s it. I’d like some fresh, clear, well-seasoned perspective.”

Hotfixes For Hyper-V & Failover Clustering Can Be Confusing KB2496089 & KB2521348


As I’m building or extending a number of Hyper-V Clusters in the next 4 months I’m gathering/updating my list with the Windows 2008 R2 SP1 hotfixes relating to Hyper-V and Failover Clustering. Microsoft has once published KB2545685: Recommended hotfixes and updates for Windows Server 2008 R2 SP1 Failover Clusters but that list is not kept up to date, the two hotfixes mentioned are in the list below. I also intend to update my list for Windows Server 2008 SP2 and Windows 2008 R2 RTM. As I will run into to these and it’s nice to have a quick reference list.

I’ll include my current list below. Some of these fixes are purely related to Hyper-V, some to a combination of hyper-V and clusters, some only to clustering and some to Windows in general. But they are all ones that will bite you when running Hyper-V (in a failover cluster or stand alone). Now for the fun part with some hotfixes I’ll address in this blog post. Confusion Smile Take a look at the purple text and the green text hotfixes and the discussion below. Are there any others like this I don’t know about?

* KB2496089 is included in SP1 according to “Updates in Win7 and WS08R2 SP1.xls” that can be downloaded here (http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=269) but the Dutch language KB article states it applies to W2K8R2SP1 http://support.microsoft.com/kb/2496089/nl

Artikel ID: 2498472 – Laatste beoordeling: dinsdag 10 februari 2011 – Wijziging: 1.0

Vereisten

Deze hotfix moet worden uitgevoerd een van de volgende besturings systemen:

  • Windows Server 2008 R2
  • Servicepack 1 (SP1) voor Windows Server 2008 R2
Voor alle ondersteunde x64 versies van Windows Server 2008 R2

6.1.7600.20881
4,507,648
15-Jan-2011
04: 10
x64

Vmms.exe
6.1.7601.21642
4,626,944
15-Jan-2011
04: 05
x64

When you try to install the hotfix it will. So is it really in there? Compare file versions! Well the version after installing the hotfix on a W2K8R2 SP1 Hyper-V server the version of vmms.exe was 6.1.7601.21642 and on a Hyper-V server with SP1 its was 6.1.7061.17514. Buy the way these are English versions of the OS, no language packs installed.

With hotfix installed on SP1

Withhotfix_thumb[1]

Without hotfix installed on SP1

Withoutpatch_thumb[1]

To make matters even more confusing while the Dutch KB article states it applies to both W2K8R2 RTM and W2K8R2SP1 but the English version of the article has been modified and only mentions W2K8R2 RTM anymore.

http://support.microsoft.com/kb/2496089/en-us

Article ID: 2496089 – Last Review: February 23, 2011 – Revision: 2.0

For all supported x64-based versions of Windows Server 2008 R2

Vmms.exe
6.1.7600.20881
4,507,648
15-Jan-2011
04:10
x64

So what gives? Has SP1 for W2K8R2 been updated with the fix included and did the SP1 version I installed (official one right after it went RTM) in the lab not yet include it? Do the service packs differ with language, i.e. only the English one got updated?. Sigh :-/ Now for the good news: ** It’s all very academic because of this KB 2521348 A virtual machine online backup fails in Windows Server 2008 R2 when the SAN policy is set to “Offline All” which brings the vmms.exe version to 6.1.7601.21686 and this hot fix supersedes KB2496089 Smile. See http://blogs.technet.com/b/yongrhee/archive/2011/05/22/list-of-hyper-v-windows-server-2008-r2-sp1-hotfixes.aspx where this is explicitly mentioned.

Ramazan Can mentions hotfix 2496089 and whether it is included in SP1 in the comments on his blog post http://ramazancan.wordpress.com/2011/06/14/post-sp1-hotfixes-for-windows-2008-r2-sp1-with-failover-clustering-and-hyper-v/ but I’m not very convinced it is indeed included. The machine I tested on are W2K8R2 English RTM updated to SP1, not installations for the media including SP1 so perhaps there could also be a difference. It also should not matter that if you install SP1 before adding the Hyper-V role, so that can’t be the cause.

Anyway, keep your systems up to date and running smoothly, but treat your Hyper-V clusters with all due care and attention.

  1. KB2277904: You cannot access an MPIO-controlled storage device in Windows Server 2008 R2 (SP1) after you send the “IOCTL_MPIO_PASS_THROUGH_PATH_DIRECT” control code that has an invalid MPIO path ID
  2. KB2519736: Stop error message in Windows Server 2008 R2 SP1 or in Windows 7 SP1: “STOP: 0x0000007F”
  3. KB2496089: The Hyper-V Virtual Machine Management service stops responding intermittently when the service is stopped in Windows Server 2008 R2
  4. KB2485986: An update is available for Hyper-V Best Practices Analyzer for Windows Server 2008 R2 (SP1)
  5. KB2494162: The Cluster service stops unexpectedly on a Windows Server 2008 R2 (SP1) failover cluster node when you perform multiple backup operations in parallel on a cluster shared volume
  6. KB2496089: The Hyper-V Virtual Machine Management service stops responding intermittently when the service is stopped in Windows Server 2008 R2 (SP1)*
  7. KB2521348: A virtual machine online backup fails in Windows Server 2008 R2 (SP1) when the SAN policy is set to “Offline All”**
  8. KB2531907: Validate SCSI Device Vital Product Data (VPD) test fails after you install Windows Server 2008 R2 SP1
  9. KB2462576: The NFS share cannot be brought online in Windows Server 2008 R2 when you try to create the NFS share as a cluster resource on a third-party storage disk
  10. KB2501763: Read-only pass-through disk after you add the disk to a highly available VM in a Windows Server 2008 R2 SP1 failover cluster
  11. KB2520235: “0x0000009E” Stop error when you add an extra storage disk to a failover cluster in Windows Server 2008 R2 (SP1)
  12. KB2460971: MPIO failover fails on a computer that is running Windows Server 2008 R2 (SP1)
  13. KB2511962: “0x000000D1″ Stop error occurs in the Mpio.sys driver in Windows Server 2008 R2 (SP1)
  14. KB2494036: A hotfix is available to let you configure a cluster node that does not have quorum votes in Windows Server 2008 and in Windows Server 2008 R2 (SP1)
  15. KB2519946: Timeout Detection and Recovery (TDR) randomly occurs in a virtual machine that uses the RemoteFX feature in Windows Server 2008 R2 (SP1)
  16. KB2512715: Validate Operating System Installation Option test may identify Windows Server 2008 R2 Server Core installation type incorrectly in Windows Server 2008 R2 (SP1)
  17. KB2523676: GPU is not accessed leads to some VMs that use the RemoteFX feature to not start in Windows Server 2008 R2 SP1
  18. KB2533362: Hyper-V settings hang after installing RemoteFX on Windows 2008 R2 SP1
  19. KB2529956: Windows Server 2008 R2 (SP1) installation may hang if more than 64 logical processors are active
  20. KB2545227: Event ID 10 is logged in the Application log after you install Service Pack 1 for Windows 7 or Windows Server 2008 R2
  21. KB2517329: Performance decreases in Windows Server 2008 R2 (SP1) when the Hyper-V role is installed on a computer that uses Intel Westmere or Sandy Bridge processors
  22. KB2532917: Hyper-V Virtual Machines Exhibit Slow Startup and Shutdown
  23. KB2494016: Stop error 0x0000007a occurs on a virtual machine that is running on a Windows Server 2008 R2-based failover cluster with a cluster shared volume, and the state of the CSV is switched to redirected access
  24. KB2263829: The network connection of a running Hyper-V virtual machine may be lost under heavy outgoing network traffic on a computer that is running Windows Server 2008 R2 SP1
  25. KB2406705: Some I/O requests to a storage device fail on a fault-tolerant system that is running Windows Server 2008 or Windows Server 2008 R2 (SP1) when you perform a surprise removal of one path to the storage device
  26. KB2522766: The MPIO driver fails over all paths incorrectly when a transient single failure occurs in Windows Server 2008 or in Windows Server 2008 R2

KB Article 2522766 & KB Article 2135160 Published Today


At this moment in time I don’t have any more Hyper-V clusters to support that are below Windows Server 2008 R2 SP1. That’s good as I only have one list of patches to keep up to date for my own use. As for you guys still taking care of Windows 2008 R2 RTM Hyper-V cluster you might want to take a look at KN article 2135160 FIX: "0x0000009E" Stop error when you host Hyper-V virtual machines in a Windows Server 2008 R2-based failover cluster that was released today. The issue however is (yet again) an underlying C-State issue that already has been fixed in relation to another issue published as KB article 983460 Startup takes a long time on a Windows 7 or Windows Server 2008 R2-based computer that has an Intel Nehalem-EX CPU installed.

And for both Windows Server 2008 R2 RTM and SP1 you might take a look at an MPIO issue that was also published today (you are running Hyper-V on a cluster and your are using MPIO for redundant storage access I bet) KB article 2522766 The MPIO driver fails over all paths incorrectly when a transient single failure occurs in Windows Server 2008 or in Windows Server 2008 R2

It’s time I add a page to this blog for all the fixes related to Hyper-V and Failover Clustering with Windows Server 2008 R2 SP1 for my own reference Smile