First experiences with a rolling cluster upgrade of a lab Hyper-V Cluster (Technical Preview)


Introduction

In vNext we have gotten a long awaited  & very welcome new capability: rolling cluster upgrades. Which for the Hyper-V roles is a 100% zero down time experience. The only step that will require some down time is the upgrade of the virtual machine configuration files to vNext (version 5 to 6) as the VM has to be shut down for this.

How to

The process for a rolling upgrade is so straight forward I’ll just give you a quick bullet list of the first part of the process:

  • Evacuate the workload from the cluster node you’re going to upgrade
  • Evict the node to upgrade to vNext from the cluster
  • Upgrade (no in place upgrade supported but in your lab you can get away with it)
  • Add the upgraded node to the cluster
  • Rinse & repeat until all nodes have been upgraded (that can take a while with larger clusters)

Please note that all actions you administration you do on a cluster in mixed mode should be done from a node running vNext or a system running Windows 10 with the vNext RSAT installed.

Once you’ve upgraded all nodes in the cluster, the situation you’re in now is basically that you’re running a Windows Server vNext Hyper-V cluster in cluster functional level 8 (W2K12R2) and the next step is to upgrade to 9, which is vNext, no there no 10 yet in server Winking smile

You do this by executing the Update-ClusterFunctionalLevel cmdlet. This is an online process.  Again, do this from a node running vNext or a system running Windows 10 with the vNext RSAT installed. Note that this is where you’re willing to commit to the vNext level for the cluster. That’s where you want to go but you get to decide when. When you’ve do this you can’t go back to W2K12R2. It’s a matter of fact that as long as you’re running cluster functional level 8, you can reverse the entire process. Talk about having options! I like having options, just ask Carsten Rachfahl (@hypervserver), he’ll tell you it’s one of my mantras.

image

When this goes well you can just easily check the cluster functional level as follows:

image

When this is done you can do the upgrade of the VM configuration by running the Update-VMConfigurationVersion cmdlet. This is an off line process where the VMs you’re updating have to be shut down. You can do this for just one VM, all or anything in between. This is when you decided you’re committing to all the goodness vNext brings you.  But the fact that you have some time before you need to do it means you can  easily get those machine to run smoothly on a W2K12R2 cluster in case you need to roll back. Remember, options are good!

Doing so updates VM version from 5 to 6 and enables new Hyper-V features (hit F5 a lot or reopen Hyper-V Manager to see the value change.

image

image

Note: If in the lab you’re running some VMs on a cluster node are not highly available (i.e. they’re not clustered) they cannot be updated until the cluster functional level has been upgraded to version 9.

Advertisements

Exchange 2010 SP3 Rollup 5 Added Support for Windows Server 2012 R2 Active Directory


6 weeks ago (February 25th 2014) Microsoft finally took away the last barrier to upgrading some of our Windows Server 2012 Active Directory Environments to R2.  Most of them are still running Exchange 2010 SP3 and not Exchange 2013. The reason is that Exchange 2013 was not deployed is whole other discussion Eye rolling smile.

However that dis mean that until the release of  Exchange Server 2010 SP3 Update Rollup 5 last month we could not upgrade Active Directory to Windows Server 2012 R2. Rollup 5 brought us support for exactly that. We can now:

  • Support Domain Controllers running Windows Server 2012 R2
  • Raise the Active Directory Forest Function Level and Domain Functional Level to Windows Server 2012 R2

Please note that you cannot deploy Exchange Server 2010 (SP3 RU5) on Windows Server 2012 R2 and you’ll probably never will be able to do that. I’m not sure Microsoft has any plans for this.

Now our office moves have been concluded, meaning I can get back to IT Infrastructure instead of being an glorified logistics & facility peon, we’re doing the upgrade.

This also means we can move the Active Directory environments to the latest version so we have the best possible position for any future IT projects at very low risk. The environments are already at W2K12 functional level. If the budgets get so tight they lose/scrap EA or volume licensing it also allows them to run at this level for many years to come without causing any blocking issues.

Reverting the Forest & Domain Functional Levels in Window Server 2008 R2, 2012, 2012 R2


Since Windows Server 2008 R2 and now with Windows Server 2012(R2)you can roll back the domain and forest functional level under certain conditions. This was not possible before with previous versions of Windows. In these cases you would have to revert to a restore from backup. Yup pretty hefty so raising functional levels has to be done with care.

Now this isn’t a free fire zone there are some conditions as listed in the table below.

image

So you cannot have advanced features like the AD recycle bin enabled in some conditions. Enabling this is irreversible, so you cannot revert the Forest Functional Level of your environment to a level that supports the AD recycle bin when it has been enabled. Today that means from Windows Server 2012(R2) to Windows Server 2008 R2.

You also need Enterprise Administrator rights to do so, which I hope you’ll understand. It’s also a Windows PowerShell only feature (Set-ADDomainMode).

I used this information recently during an upgrade of an Windows Server 2008 R2 domain to Windows Server 2012 where they wanted to raise the domain and forest functional level. As they had a Forest Trust between the (now) Windows Server 2012 forest/domain and another Windows Server 2008 R2 forest/domain. They had enabled the Recycle Bin when still at Windows 2008 R2. They wanted to know if they would have issues with the trust and if so whether they could revert the levels in that case.

Well I could put their mind at ease. Look at the table. Yes you can go back to Windows 2008 R2 Forest Functional level as that’s a version that also supports AD Recycle bin so it doesn’t matter that is enabled.  And no, the forest trust capability is not affected by the forest functional level in this case as all you need there is to be at a minimum level of Windows 2003 to be able to do a forest trust. Forest Trust is enabled from and above Windows Server 2003 Forest functional Level. In a Windows Server 2000 Forest functional Level, Forest Trust is disabled. That means you can do them between forests at different functional levels a long as non of them is lower than Windows 2003. In this case it’s Windows 2008 R2 that’s the lowest, so again, not an issue.

How? Very simple:

Set-ADDomain Mode mydomain.com -DomainMode Windows2008R2Domain

Set-ADForestMode mydomain.com -ForestMode Windows2008R2Forest

Take a look at these TechNet Resources Understanding Active Directory Domain Services (AD DS) Functional Levels  and Set-ADDomainMode for more information.

Upgrading Your DELL Compellent Storage Center Firmware (Part 2)


This is Part 2 of this blog. You’ll find Part 1 over here.

In part 1 we prepared our Compellent SAN to be ready and install Storage Center 6.3.10 that has gone public.  As said, 6.3.10 brings interesting features like ODX and UNMAP to us Windows Server 2012 Hyper-V users. It also introduces some very nice improvements to synchronous replication and Live Volumes. But here we’ll just do the actual upgrade, the preparations & health check have been done in part 1 so we can get started here right away.

Log in to your Compellent system and navigate to the Storage Management menu. Click on “System”, select Update and finally click on “Install Update”.  It’s already there as we downloaded it in Part 1. Click on “Install Now” to kick it all off.

image

Click on Install now to launch the upgrade.

image

After initialization you can walk away for 10 minutes but you might want to keep an eye on things and the progress of the process.

image

So go have a look at your storage center. Look at the Alert Monitor for example and notice that the “System is undergoing maintenance”.

image

When the controller the VIP address of the SAN reboots it becomes unavailable. After a while you can login again to the other controller via the VIP, if you cant’ wait a few seconds just use the IP address of the active controller. That will do.

image

When you log in again you’ll see the evidence of an ongoing SAN firmware upgrade. Nothing to panic about.image

This is also evident in Alert Monitor. CoPilot knows you’re doing the upgrade so no unexpected calls to make sure your system is OK will come in. They’re there every step of the way. The cool thing is that is the very first SAN we ever owned that we don’t need engineers on site or complex and expensive procedure to do all this. It’s all just part of an outstanding customer service Compellent & DELL deliver.image

You can also take a peak at your Enterprise manager software to see paths going down and so on. The artifacts of a sequential controller failovers during an upgrade. Mind you you’re not suffering downtime in most cases.image

Just be patient and keep an eye on the process. When you log in again after the firmware upgrade and your system is up and running again, you’ll be asked to rebalance the ports & IO load between the controllers on the system. You do, so click yes.image

image

When done you’ll return to the Storage Center interface. Navigate to “Help”" and click on About Compellent Storage Center. image

You can see that both controllers are running 6.3.10.

image

You’re rocking the new firmware. As you kept an eye on your hosts you should know these are good to go. Send of an e-mail to CoPilot support and they’ll run a complete health check on your system to make sure you’re good to go. Now it’s time to start leveraging the new capabilities you just got.

Upgrading Your DELL Compellent Storage Center Firmware (Part 1)


This is Part 1 of this blog. You’ll find Part 2 over here

Well the Compellent firmware 6.3.10 has gone public and it’s time to put it on our systems. 6.3 brings interesting features like ODX and UNMAP to us Windows Server 2012 Hyper-V users. It also introduces some very nice improvements to synchronous replication and Live Volumes. But’s those are matters for other blog posts. Here We’ll focus on the upgrade.

In part 1 we’ll look at how we prepare the Compellent to be ready to apply the upgrade. We make sure on our side we have no outstanding issues on the SAN. Then we made sure we upgraded Enterprise Manager and Replay Manager to the latest versions. At this time of writing that is EM 6.3.5.7 and RM 7.0.1.1. We could do this prior to the firmware upgrade because 6.2.2. is also supported by these versions. Once we established all was working well with this software we contacted CoPilot to check our systems (the check it’s health an applicability as well). When all is in order they’ll release the firmware to us. Then It’s time to run a check for update on the systems.

Log in to your Compellent system and navigate to the Storage Management menu. Click on “System”, select Update and finally click on “Check for Update”.
image

The tool will check for updates.

image

If no new firmware has been released to your systems you’ll see this.image

If new firmware has been released you see this in the update status.image

This also shows in the Storage Center GUI

image

Downloading the update.

image

The download takes a while. Once it’s done you’ll see that the update is ready to install. Note that this update is non service affecting in OUR case (green arrow). We won’t install it yet however. We’ll look at the details & validate the components. Due diligence pays off Winking smileimage

Click on details to get some more information about what’s in the update. image

You can see that our disk and enclosure firmware is up to date already from a previous update. The ones related to 6.3.10 are mandatory( required, not optional). When done, hit Return.

We now select “Validate Components” to make sure we’re good to go and won’t get any surprises. Trust but verify is one of our mantras.image

image

So now we are ready to run the update.  We’ll leave that for Part 2.

FIX: iSCSI Initiators Cannot Connect To iSCSI Target Anymore After Upgrading Windows Server 2012 iSCSI Target To Windows Server 2012 R2 Preview


I recently did an in place upgrade of my Windows Server 2012 iSCSI Target host in my home lab to Windows Server 2012 R2 Preview. That went well for one “minor” issue. My iSCSI initiators could not connect to it anymore and where in perpetual “connecting mode”.

image

On the target the status remained in “Not Connected” for all the VHDs.

image

As you can imagine this sort of ruins the share storage experience on my test Hyper-V cluster. After checking the basics and the event logs I found nothing wrong with the iSCSI Target. It’s then that I turned to the firewall as I suspected that we could have an issue there. Sure enough, the inbound iSCSI rules on the target had not been enabled.

image

After taking care of that the iSCSI connections succeeded and live was good again in the home lab. You only need to enable the iSCSI target rules.

image

image

image

In Place Upgrade To Windows Server 2012 R2 Preview Fails with “Windows installation encountered an unknown error and cannot continue.”


When doing an in place upgrade of Windows server 2012 to Windows Server 2012 R2 Preview you hit the following error:

clip_image002

There is not even a compatibility report generated. We have enough free space, no error events are logged except for this one in C:\$WINDOWS.~BT\Sources\Panther setuperr.logerror log that just mentions  Error      [0x0807a7] MIG    Callback_CreateCompatibilityReport failed[gle=0x00000490]

The trick to deal with this is by not getting install updates online when offered as option in installation wizard.

image

Otherwise it downloads some bits and then creation of the compatibility report fails. I owe a big thank you to fellow MVP Tomica Kaniski for this tip. I saved me some trouble shooting & time.

My guess is Microsoft will soon have this fixed.

Anyway, now all physical hosts in my home lab are running Windows Server 2012 R2 Preview. We refer to this in our group as having seen the fish Smile.

image