Video Interview On Rolling Cluster Upgrades in Windows Server vNext


Carsten Rachfahl from Rachfahl IT-Solutions (quite possibly  Germany’s leading Hyper-V, Storage Spaces & Private cloud consultancy) and I got together in Berlin last November at the Microsoft Technical Summit 2014. Between presenting (I delivered What’s new in Failover Clustering in Windows Server 2012 R2), workshops, interviews we found some time to do a video interview.

We discussed a very welcome new capability in Windows Server vNext: “Rolling cluster updates” or “Cluster Operating System Rolling Upgrade” in Windows Server Technical Preview as Microsoft calls it. I blogged about this rather soon after the release of the Technical Preview First experiences with a rolling cluster upgrade of a lab Hyper-V Cluster (Technical Preview).

Videointerview with Didier Van Hoye about Rolling Cluster Upgrade Thumb1

We’ve been able to do rolling updates of Windows NLB for a long time and we’ve been asking for that same capability in Windows Failover Clustering for many years and now, it’s finally coming! And yes, as you will notice we like that a lot!

You need to realize that making the transition form one version to another as smooth, easy and risk free as possible is of great value to the customer as it enables them to upgrade faster and get the benefits of their investment quicker. For Microsoft it means they can have more people move to more modern environments faster which helps with support and delivering value in a secure and modern environment.

At the end we also joke around a bit about DevOps and how this is just as set of training wheels on the road to true site resilience engineering. All fun and all good. Enjoy!

Options For A Highly Available Load Balanced RD Gateway Server Farm on Hyper-V


When you need to make the RD Gateway service highly available you have some options. On the RD Gateway side you have capability of configuring a farm with multiple RD Gateway servers.image

When in comes to the actual load balancing of the connections there are some changes in respect load balancing from Windows Server 2008 R2 that you need to de aware of! With Windows 2008 R2 you could do:

  1. Load balancing appliances (KEMP Loadmaster for example, F5, A10, …) or Application Delivery Controllers, which can be hardware, OEM servers, virtual and even cloud based (see Load Balancing In An Ever More Demanding Virtualized & Cloudy World). KEMP has Hyper-V appliances, many others don’t. These support layer 4, layer 7, geo load balancing etc. Each has it’s use cases with benefits and drawback but you have many options for the many situations you might encounter.
  2. Software load balancing. With this they mean Windows NLB. It works but it’s rather limited in regards to intelligence for failure detection & failover. It’s in no way an “Application Delivery Controller” as load balancer are positioned nowadays.
  3. DNS Round Robin load balancing. That sort of works but has the usual drawbacks for problem detection and failover.  Don’t get me wrong for some use cases it’s fine, but for many it isn’t.

I prefer the first but all 3 will do the basic job of load balancing the end-user connections based on the traffic. I have done 2 when it was good enough or the only option but I have never liked 3, bar where it’s all what’s needed, because it just doesn’t fit many of the uses cases I dealt with. It’s just too limited for many apps.

In regards to RD Gateway in Windows Server 2012 (R2), you can no longer use  DNS Round Robin for load balancing with the new HTTP transport. The reason is that it uses two HTTP channels (one for input and one for output) and DNS round robin cannot guarantee that both these connections will be routed trough the same RD Gateways server which is a requirement for it to work. Basically RRDNS will only work for legacy RPC-HTTP. RPC could reroute a channel to make sure all flows over the same node at the cost of performance & scalability. But that won’t work with HTTP which provides scalability & performance. Another thing to note is that while you can work without UDP you don’t want to. The UDP protocol is used  to deliver graphics with a better user experience  over even low quality networks for graphics or high and experiences with RemoteFX. TCP (HTTP) is can be used without it (at the cost of a lesser experience) and is also used to maintain the sessions and actions. Do note that you CANNOT use UDP alone as these connections are established only after the main HTTP connection exists between the remote desktop client and the remote desktop server. See Don’t Forget To Leverage The Benefits of RD Gateway On Hyper-V & RDP 8/8.1 for more information

So you will need a least Windows Network Load Balancing (WNLB) because that supports IP affinity to make sure all channels stick to the same node. UDP & HTTP can be on different nodes by the way. Also please not that when using network virtualization WNLB isn’t a good choice. It’s time to move on.

So the (or at least my) preferred method is via a real “hardware” load balancer.  These support a bunch of persistence options like IP affinity, cookie-based affinity, … just look at the screenshot below (KEMP Loadmaster)

image

But they also support layer 7 functionality for better health checking and failover.  So what’s not to like?

So we need to:

  1. Build a RD Gateway Farm with at least two servers
  2. Load balance HTTP/HTTPS for the RD Gateway farm
  3. Load balance UDP for the RD Gateway farm.

We’ll do this 100% virtualized on Hyper-V and we’ll also make make the load balancer it self highly available. Remember, removing single points of failure are like bottle necks. The moment you take one away you just hit the next one Smile.

Kemp has a great deployment guide for RDS on how to do this but I should ass that you could leverage SUB Virtual Services (SUBVS) to deal with the other workloads such as RD Web Access if they’re on the same server. They don’t mention this in the white paper but it’s an option when using HTTP/HTTPS as service type for both configurations. #1 & #2 are the SUB Virtual Services where I used this in a lab.

image

But for RD Gateway you can also leverage the Remote Terminal Service type and in this case you won’t leverage SUBVS as the service type is different between RD Gateway (Remote Terminal) and RD Web Access (HTTP/HTTPS). This is actually used by their RDS template you can download form their support site.

image

Hope this helps some of you out there!

Quick Demo Video Of Site Failover With KEMP Loadmaster Global Balancing


Here’s a quick video that demonstrates how you can achieve site failover with via the KEMP Loadmaster Global Balancing feature. As long as you know what this can do for you and realize that it about site failover and high availability and not continuous availability without a second of service interruption you can deliver nice results with this technology across city campuses or between cities.

In our scenario we normally connect to the primary data center (weighted round robin) and fail over to the DRC when the primary site fails for some reason.

It’s very busy at the moment but I hope to address this topic a bit more in detail in the future. All of this runs virtualized on Hyper-V and performs just fine.

Don’t Forget To Leverage The Benefits of RD Gateway On Hyper-V & RDP 8/8.1


So you upgraded your TS Gateway virtual machine on W2K8(R2) to RDS Gateway on W2K12(R2) too make sure you get the latest and the greatest functionality and cut off any signs of technology debt way in advance. Perhaps you were inspired by my blog series on how to do this, and maybe you jumped through the x86 to x64 bit hoop whilst at it. Well done.

Now when upgrading or migrating from W2K8(R2) a lot of people forget about some of the enhancements in W2K12(R2). This is especially true of you don’t notice much by doing so. That’s why I see people forget about UDP. Why? Well things will keep working as they did before Windows Server 2012 RDS Gateway over HTTP or over RPC-HTTP (legacy clients). I have seen deployments where both the Windows and the perimeter firewall rules to allow UDP over 3391 were missing. Let alone that port 3391 was allowed in the RAP.  But then you miss out on the benefits it offers (a better user experience over less than great network connections and with graphics) ass well on those of that ever more capable thingy called RemoteFX, if you use that.

For you that don’t know yet:  HTTP and UDP protocols are both used preferably by RD Gateway and are more efficient than RPC over HTTP which is better for scaling and experience under low bandwidth and bad connectivity conditions. When HTTP transport channels are up (in & outgoing traffic), two UDP side channels are set up that can be used to provide both reliable (RDP-UDP-R) and best-effort (RDP-UDP-L) delivery of data. UDP also leveraged SSL via the RD gateway because is uses Datagram Transport Layer Security (DTLS). For more info RD Gateway Capacity Planning in Windows Server 2012. Further more it proves you have no reason not to virtualize this workload and I concur!

So why not set it up!?  So check you firewall rules on the RD Gateway Server and set the rules accordingly. Do the same for your perimeter firewalls or any other in between your users and your RD Gateway.

image

Under properties of your RS Gateway server you need to make sure UDP is enabled and listening on the needed IP address(es)

image

A client who connects over your RDS Gateway server, Windows Server 2012(R2) that is, and checks the network connection properties (click the “wireless NIC” like icon in the connection bar) sees the following: UDP is enabled. imageIf they don’t see UDP as enabled and they aren’t running Windows 8 or 8.1 (or W2K12R2) they can upgrade to RDP 8.1 on windows 7 or Windows Server 2008 R2! When they connect to a Windows 7 SP1 or Windows 2008R2  machine make sure you read this blog post Get the best RDP 8.0 experience when connecting to Windows 7: What you need to know as it contains some great information on what you need to do to enable RDP 8/8.1 when connecting to Windows 7 SP1 or Windows 2008 R2:

  1. “Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Remote Session Environment\Enable Remote Desktop Protocol 8.0” should be set to “Enabled”
  2. “Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Connections\Select RDP Transport Protocols” should be set to “Use both UDP and TCP” => Important: After the above 2 policy settings have been configured, restart your computer.
  3. Allow port traffic: If you’re connecting directly to the Windows 7 system, make sure that traffic is allowed on TCP and UDP for port 3389. If you’re connecting via Remote Desktop Gateway, make sure you use RD Gateway in Windows Server 2012 and allow TCP port 443 and UDP port 3391 traffic to the gateway

Cool you’ve done it and you verify it works. Under monitoring in the RD Gateway Manager you can see 3 connections per session: one is HTTP and the two others are UDP.

image

Life is good. But if you want to see the difference really well demonstrated try to connect to Windows 7 SP1 computer with RDP8 & TCP/UDP disabled and play a YouTube video, then to the same with RDP8 & TCP/UDP enabled, the difference is rather impressive. Likewise if you leverage RemoteFX in VM. The difference is very clear in experience, just try it! While you’re doing this look a the UDP “Kilobytes Sent” stats (refresh the monitoring tab, you’ll see UDP being put to work when playing a video on in your RDP session.

image

Load balancing Hyper-V Workloads With High To Continuous Availability With a KEMP Loadmaster


I’m working on some labs and projects with KEMP Loadmaster load balancing appliances (LM 2400, LM-R320) That will lead to some blog post on  load balancing several workloads, which are all on Windows Server 2012 R2  Hyper-V or integrate in to Azure. The load balancers used in the labs are the virtual appliances, depending on the needs and environment these are a very good, cost effective option for production as well and depending on the version you get they scale very well. Hence their use in cloud environments, they will not hold you back at all!

To stimulate your interest in load balancing and high availability I’ve put up a video on load balancing RD Gateway services. Consider it a teaser or introduction to more about the subject.

Why use an appliance (hardware/virtual)? Well let’s look at the 2 alternatives:

  • Round robin DNS, which is also sometimes used is just to low tech for most real life scenarios and sometimes can’t be used or is less efficient which impacts scalability and performance. On top of that it doesn’t provide health checking for failover purposes.
  • I’ve also said  before that while Windows NLB  provides layer 4 load balancing out of the box it’s pretty basic. It also often causes a lot of network grief and the implementation can be tedious. This has not improved in an ever more virtualized & cloud based world. On top of that, when network virtualization comes into play you might paint yourself into a corner as those two don’t mix. But if that’s not a concern and you’re on a budget, I’ve used it with success in the past as well.

Load Balancing In An Ever More Demanding Virtualized & Cloudy World


We’ve been using the Kemp Loadmasters for many years now and they have served us very well. You might know that Microsoft Azure has a partnership with Kemp technologies to provide full featured load balancing in your public & hybrid cloud solutions. I pretty happy with that as when talk about load balancing with Microsoft we always end up discussing the need for more features and layer 7 support. I sometimes jokingly tease them that this is due to their Windows NLB legacy. While I have done some magic with that, it is way too limited for today’s (and yesterdays) demands and needs. Also the hacks they use to get it to work can’t be used in network virtualization. In the cloud Microsoft has the Azure Load Balancer. Whilst nice when combined with availability sets many of the current workloads need more. That’s exactly what the KEMP Virtual LoadMaster for Azure delivers in their partnership with Microsoft:

  • Layer 4, Layer 7 Load Balancing
  • Layer 7 (or Cookie) Persistence
  • SSL Offload/SSL Acceleration
  • Application Health Checking
  • Adaptive (Server Resource) Load Balancing
  • Layer 7 Content Switching
  • Application Acceleration: HTTP Caching, Compression & IPS

To me (and many other IT Pros) Kemp is the company that opened load balancing up to everyone on this planet with budget friendly but high value solutions. They took away the barrier to better & more capable load balancing for the masses. Furthermore they keep improving and I have seen many existing customers, including me get ever more benefits with the newer firmware releases, even on their entry level, older models like the LM2200 that are not for sale anymore. So you can keep using them or move them to the lab. They have great support and respond very quickly to vulnerabilities like Heartbleed, Shellshock and Poodle.

image 

Another benefit of this partnership is that we can use the load balancing solution we know and trust in all our environments: on premises (physical or virtual appliance), in the cloud & at our hosting companies. Partner ships with OEMs ensure that you can use the hardware you prefer (the DELL R320 is a nice example) and their Virtual Load Master now even extends into the cloud. So our options are to …

… deploy an appliance …

image

…  virtualize the LoadMasters …

image

… leverage Kemp in the cloud

image

…. or select your own preferred OEM …

image

They cover all our bases with that line up and it helps with operational ease & efficiencies.

As I’m investigating some scenarios with KEMP LoadMasters in a Hyper-V environment (on premises, multi sites, Azure IAAS & Multifactor Authentication you can expect to see some blog posts on this. Some of these will leverage technologies available in Windows Server vNext (Technical Preview). Lot’s of very interesting ideas to support high availability & flexibility that are affordable and not just point solutions.

Ah the joy of being in virtualization is that one gets great exposure to storage, networking, cloud solutions and on premises. The experience & knowledge of the entire stack isn’t just fun (yes working can be fun) but it is also what allows to build great solutions.

Handy DELL Storage Tool: Dell Virtual rack


If you need to see and find information on DELL storage fast they have a nice page on line with a virtual rack where you can look at the components of their various storage offerings.

image

They update it regularly and it’s a fast and easy way to get started. Naturally you’ll need to dive in a bit deeper or get some help to work out the last details. I’m like the Compellent as I have found it to be the best possible traditional storage solution for a Windows Server 2012 R2 Hyper-V environment and great value for money. I hope they can find a way to keep delivering that same value in the coming years in an ever changing storage landscape.

Windows Server vNext Soft Restart – A way to speed up reboots? Not in Technical Preview 9841


As you all probably know I’m also playing around with and testing Windows Server vNext Tech Preview and one of the nice new features in there I have my eye on is Soft Restart.

image

There is little information on this feature out there right now but from the description “Soft Restart” looks like a way to get faster Windows boot times by cutting down on device firmware initialization. When it’s not needed that would be a great thing to have as with > 10gbps live migration speeds the boot time of our hardware loaded (DRAC, NICs, HBA, BMC, …) servers is what makes it the longest single step per node during cluster aware updating. Interesting if this is indeed what it’s there for.

But let’s find out if this is indeed what we think it is Smile. First of all the installation of this feature requires a restart. Keep this in mind.

There are 2 ways to kick it off that I know of but to me there must be more … it would be a shame not to have this integrated as an option into Cluster Aware Updating for example.

Option 1: via shutdown

image

So let’s try shutdown /r /soft /t 000.  No joy, doesn’t make one bit of difference and nothing logged or so to indicate an issue.

Option 2: PowerShell via Restart-Computer –Soft

No joy here either …

image

What could be the problem?

So I figured I needed enterprise grade server hardware with some FC cards & lots of NIC and memory to notice the difference. On a VM it might do nothing, but I assure you I doesn’t do anything on the PC based home lab either. So I dragged a DELL PowerEdge R730 with exactly that into the game. But still no joy. Then I thought some more and decided it might integrate with the hardware capabilities to do so of I went to install the latest and greatest DELL Server Manager software to see if that make a difference. But again, no joy.

It’s probably not lit up yet in this release of the Technical Preview 9841. For now I’ll be content with the 28-30% improved reboot speeds the DELL R730 UEFI brought us. I’d love to speed things up a bit as time is money and valuable Winking smile but we’ll have to wait for the next code drop to see if and how it works …

Windows Server Technical Preview delivers integration services updates through Windows Update


Benefits of delivering updates to the integration services via Windows Updates

In Windows Server  vNext aka the Technical Preview the integration services are being delivered through Windows Update (and as such the well know tools such a s WSUS, …). This is significant in reducing the operational burden to make sure they are up to date. Many of us turned to PowerShell scripting to handle this task. So did I and I still find myself tweaking the scripts once in a while for a condition I had not dealt with before or just to get better feedback or reporting. Did I ever tell you that story about the cluster where a 100VMs did not have a virtual DVD drive (they removed them to improve performance) … that was yet another improvement to my script => detect the absence of a virtual DVD drive. In this day and age, virtualization has both scaled up and out with ever more virtual machines per host and in total. The process of having to load an ISO in a virtual DVD drive inside a virtual machine to install upgrades to integration services seems arcane and it’s very timely that it has been replaced by an operation process more befitting a Cloud OS Winking smile.

I have optimized this process with some PowerShell scripting and it wasn’t to painful anymore. The script upgrades all the VMs on the hosts and even puts them back in the state if found them in (Stopped, Saved, Running). A screenshot of the script in action below.

image

I’m glad that it’s now integrated through Windows Update and part of other routine maintenance that’s done on the guests anyway.

But is not only good news for us “on premises” system administrators and integrators. It’s also important for service/cloud providers and (hosted) private cloud hosters. This change means that the tenants  have control of updates to the integration services of their virtual machines. They update their Windows virtual machines with all updates during their normal patch cycles and now this includes the integration services. This provides operation ease (single method) and avoids some of the discussions about when to upgrade the integration services.

Legacy Operating Systems

Shortly after the release of the Windows Server Technical Preview, updates to integration services for Windows guests began being distributed through Windows Update. This means that on that version the vmguest.iso is no longer needed and as such it’s no longer included with Hyper-V.  This means that if you run an unsupported (most often legacy) version of Windows you’ll need to grab the latest possible vmguest.iso from an W2K12R2 Hyper-V host and try to install that and see if it works.

What about Linux and FreeBSD?

Well nothing has changed and how that’s taken care of you can read here: Linux and FreeBSD Virtual Machines on Hyper-V

Hyper-V Technical Preview Live Migration & Changing Static Memory Size


I have played with  Hot Add & Remove Static Memory in a Hyper-V vNext Virtual Machine before and I love it. As I’m currently testing (actually sometimes abusing) the Technical Preview a bit to see what breaks I’m sometimes testing silly things. This is one of them.

I took a Technical Preview VM with 45GB of memory, running in a Technical Preview Hyper-V cluster and live migrate it.  I then tried to change the memory size up and down during live migration to see what happens, or at least nothing goes “BOINK”. Well, not much, we get a notification that we’re being silly. So no failed migrations, crashed or messed up VMs or, even worse hosts.

image

It’s early days yet but we’re getting a head start as there is a lot to test and that will only increase. The aim is to get a good understanding of the features, the capabilities and the behavior to make sure we can leverage our existing infrastructures and software assurance benefits as fast a possible. Rolling cluster upgrades should certainly help us do that faster, with more ease and less risk. What are your plans for vNext? Are you getting a feeling for it yet or waiting for a more recent test version?