In Defense of Switch Independent Teaming With Hyper-V


For many old timers (heck, that includes me) NIC teaming with LACP mode was the best of the best, at least when it comes to teaming options. Other modes often led to passive/active, less than optimal receiving network traffic aggregation. Basically, and perhaps over simplified, I could say the other options were only used if you had no other choice to get things to work. Which we did a lot … I used Intel’s different teaming modes for various reasons in the past (before we had MLAG, VLT, VPC, …). Trying to use LACP where possible was a good approach in the past in physical deployments and early virtualized environments when 1Gbps networking dominated the datacenter realm and Windows did not have native support for LBFO.

But even LACP, even in those days, had some drawbacks. It’s the most demanding form of teaming. For one it required switch stacking. This demands the same brand and type of switches and that means you have no redundancy during firmware upgrades. That’s bad, as the only way to work around that is to move all workload to another rack unit … if you even had the capability to do that! So even in days past we chose different models if teaming out of need or because of the above limitations for high availability. But the superiority of NIC teaming with LACP still stands for many and as modern switches support MLAG, VLT, etc. the drawback of stacking can be avoided. So does that mean LACP for NIC teaming is always the superior choice today?

Some argue it is and now they have found support in the documentation about Microsoft CPS system documentation about Microsoft CPS system. Look, even if Microsoft chose to use LACP in their solutions it’s based on their particular design and the needs of that design I do not concur that this is the best overall. It is however a valid & probably the choice for their specific setup. While I applaud the use of MLAG (when available to you a no or very low cost) to have all bases covered but it does not mean that LACP is the best choice for the majority of use cases with Hyper-V deployments. Microsoft actually agrees with me on this in their Windows Server 2012 R2 NIC Teaming (LBFO) Deployment and Management guide. They state that Switch Independent configuration / Dynamic distribution (or Hyper-V Port if on Hyper-V and if not on W2K12R2)  is the best possible default choice is for teaming in both native and Hyper-V environments. I concur, even if perhaps not that strong for native workloads (it depends). Exceptions to this:

  • Teaming is being performed in a VM (which should be rare),
  • Switch dependent teaming (e.g., LACP) is required by policy, or
  • Operation of a two-member Active/Standby team is required by policy.

In other words in 2 out of 3 cases the reason is a policy, not a technical superior solution …

Note that there are differences between Address Hash, Hyper-V Port mode & the new dynamic distribution modes and the latter has made things better in W2K12R2 in regards to bandwidth but you’ll need the read the white papers. Use dynamic as default, it is the best. Also note that LACP/Switch Dependent doesn’t mean you can send & receive to and from a VM over the aggregated bandwidth of all team members. Life is more complicated than that. So if that’s you’re main reason for switch dependent, and think you’re done => be ware Winking smile.

Switch Independent is also way better for optimization of VMQ. You have more queues available (sum-of-queues) and the IO path is very predictable & optimized.

If you don’t control the switches there’s a lot more cross team communication involved to set up teaming for your hosts. There’s more complexity in these configurations so more possibilities for errors or bugs. Operational ease is also a factor.

The biggest draw back could be that for receiving traffic you cannot get more than the bandwidth a single team member can deliver. That’s true but optimizing receiving traffic has it’s own demands and might not always be that great if the switch configuration isn’t that smart & capable. Do I ever miss the potential ability to aggregate incoming traffic. In real life I do not (yet) but in some configurations it could do a great job to optimize that when needed.

When using 10Gbps or higher you’ll rarely be in a situation where receiving traffic is higher than 10Gbps or higher and if you want to get that amount of traffic you really need to leverage DVMQ. And a as said switch independent teaming with port of dynamic mode gives you the most bang for the buck. as you have more queues available. This drawback is mitigated a bit by the fact that modern NICs have way larger number of queues available than they used to have. But if you have more than one VM that is eating close to 10Gbps in a non lab environment and you planning to have more than 2 of those on a host you need to start thinking about 40Gbps instead of aggregating a fistful of 10Gbps cables. Remember the golden rules a single bigger pipe is always better than a bunch of small pipes.

When using 1Gbps you’ll be at that point sooner and as 1Gbps isn’t a great fit for (Dynamic) VMQ anyway I’d say, sure give LACP a spin to try and get a bit more bandwidth but will it really matter? In native workloads it might but with a vSwith?  Modern CPUs eat 1Gbps NICs for breakfast, so I would not bother with VMQ. But when you’re tied to 1Gbps it’s probably due to budget constraints and you might not even have stackable, MLAG, VLT or other capable switches. But the arguments can be made, it depends (see Don’t tell me “It depends”! But it does!). But in any case I start saving for 10Gbps Smile

Today as the PC8100 series and the N4000 Series (budget 10Gbps switches, yes I know “budget” is relative but in the 10Gbps world, but they offer outstanding value for money), I tend to set up MLAG with two of these per rack. This means we have all options and needs covered at no extra cost and without sacrificing redundancy under any condition. However look at the needs of your VMs and the capability of your NICs before using LACP for teaming by default. The fact that switch independent works with any combination of budget switches to get redundancy doesn’t mean it’s only to be used in such scenarios. That’s a perk for those without more advanced gear, not a consolation price.

My best advise: do not over engineer it. Engineer it for the best possible solution for the environment at hand. When choosing a default it’s not about the best possible redundancy and bandwidth under certain conditions. It’s about the best possible redundancy and bandwidth under most conditions. It’s there that switch independent comes into it’s own, today more than ever!

There is one other very good, but luckily also a very rare case where LACP/Switch dependent will save you and switch independent won’t: dead switch ports, where the port becomes dysfunctional. So while switch independent protects against NIC, Switch, cable failures, here it doesn’t help you as it doesn’t know (it’s about link failures, not logical issues on a port).

For the majority of my Hyper-V deployments I do not use switch dependent / LACP. The situation where I did had to do with Windows NLB in combination with ICMP Multicast.

Note: You can do VLT, MLAG, stacking and still leverage switch independent teaming, LACP or static switch dependent is NOT mandatory even when possible.

GPS service issues resolved fast by Hyper-V & site resilience engineering


Diminished services on a GPS positioning network

The past couple of days there had been latencies negatively affecting a near real time GPS positioning service that allow the users the correct their GPS measurements in real time.

Flemish Positioning Service (FLEPOS)

That service is really handy when you’re a surveyor and it safes money by avoiding extra GIS post processing work later. It becomes essential however when you are relying on your GPS coordinates to farm automatically, fly aerial photogrammetry patterns, create mobile mapping data, build dams or railways, steer your dredging ships and maneuver ever bigger ships through harbor locks.

Flemish Positioning Service (FLEPOS)

It was clear this needed to be resolved. After checking for network issues we pretty much knew that the recently spiking CPU load was the cause. Partially due to the growth in users, more and more use cases and partially due to a new software version that definitely requires a few more CPU cycles.

The GPS positioning service is running on multiple virtual machines, on separate LUNS, on separate hosts, those hosts are on separate racks. All this is being replicated to a second data center. They have high to continuous availability with Microsoft Failover Clustering and leverage Kemp Loadmaster load balancing. Together with the operations team we moved the load away from every VM, shut it down, doubled the vCPU count and restarted the VM. Rinse and repeat until all VMS have been assigned more vCPUs.

The results where a dramatic improvement in the response times and services response times that went back to normal.

Breathing room with more vCPUs

They can move fast and efficient

All this was done fast. They have the power to decide and act to resolve such issues on our own responsibility. Now the fact that they operate in tight night team that span over bureaucracy, hierarchies and make sure that people who need to involved can communicate fats en effective (even if they are spread over different locations) makes this possible. They have a design for high availability and a vertically integrated approach to the solution stack that spans any resource (CPU, Memory, Storage, Network and Software) combined with a great app owner and rock solid operational excellence (Peopleware) to enable the Site Resilience side of the story. Fast & efficient.

I’m proud to have help design and deliver this service and I’ll be ready and willing to help design vNext of this solution in the near future. We moved it from hardware to a virtualized solution based on Hyper-V in 2008 and have not regretted one minute of it. The operational capabilities it offers are too valuable for that and banking on Hyper-V has proved to be a winner!

Would Hot Add CPU Capability have made this easier?

Yes, faster for sure Smile. The process they have now isn’t that difficult. Now would I not like hot add vCPU capabilities in Hyper-V? Yes, absolutely. I do realize however that not every application might be able to handle this without restarting making the exercise a bit of a moot point in those cases.

Why some people have not virtualized yet I do not know (try and double the CPUs on your hardware servers easily and fast without leaving the comfort of your home office). I do know how ever that you are missing out on a lot of capabilities & operational benefits.

Video Interview On Rolling Cluster Upgrades in Windows Server vNext


Carsten Rachfahl from Rachfahl IT-Solutions (quite possibly  Germany’s leading Hyper-V, Storage Spaces & Private cloud consultancy) and I got together in Berlin last November at the Microsoft Technical Summit 2014. Between presenting (I delivered What’s new in Failover Clustering in Windows Server 2012 R2), workshops, interviews we found some time to do a video interview.

We discussed a very welcome new capability in Windows Server vNext: “Rolling cluster updates” or “Cluster Operating System Rolling Upgrade” in Windows Server Technical Preview as Microsoft calls it. I blogged about this rather soon after the release of the Technical Preview First experiences with a rolling cluster upgrade of a lab Hyper-V Cluster (Technical Preview).

Videointerview with Didier Van Hoye about Rolling Cluster Upgrade Thumb1

We’ve been able to do rolling updates of Windows NLB for a long time and we’ve been asking for that same capability in Windows Failover Clustering for many years and now, it’s finally coming! And yes, as you will notice we like that a lot!

You need to realize that making the transition form one version to another as smooth, easy and risk free as possible is of great value to the customer as it enables them to upgrade faster and get the benefits of their investment quicker. For Microsoft it means they can have more people move to more modern environments faster which helps with support and delivering value in a secure and modern environment.

At the end we also joke around a bit about DevOps and how this is just as set of training wheels on the road to true site resilience engineering. All fun and all good. Enjoy!

Don’t Forget To Leverage The Benefits of RD Gateway On Hyper-V & RDP 8/8.1


So you upgraded your TS Gateway virtual machine on W2K8(R2) to RDS Gateway on W2K12(R2) too make sure you get the latest and the greatest functionality and cut off any signs of technology debt way in advance. Perhaps you were inspired by my blog series on how to do this, and maybe you jumped through the x86 to x64 bit hoop whilst at it. Well done.

Now when upgrading or migrating from W2K8(R2) a lot of people forget about some of the enhancements in W2K12(R2). This is especially true of you don’t notice much by doing so. That’s why I see people forget about UDP. Why? Well things will keep working as they did before Windows Server 2012 RDS Gateway over HTTP or over RPC-HTTP (legacy clients). I have seen deployments where both the Windows and the perimeter firewall rules to allow UDP over 3391 were missing. Let alone that port 3391 was allowed in the RAP.  But then you miss out on the benefits it offers (a better user experience over less than great network connections and with graphics) ass well on those of that ever more capable thingy called RemoteFX, if you use that.

For you that don’t know yet:  HTTP and UDP protocols are both used preferably by RD Gateway and are more efficient than RPC over HTTP which is better for scaling and experience under low bandwidth and bad connectivity conditions. When HTTP transport channels are up (in & outgoing traffic), two UDP side channels are set up that can be used to provide both reliable (RDP-UDP-R) and best-effort (RDP-UDP-L) delivery of data. UDP also leveraged SSL via the RD gateway because is uses Datagram Transport Layer Security (DTLS). For more info RD Gateway Capacity Planning in Windows Server 2012. Further more it proves you have no reason not to virtualize this workload and I concur!

So why not set it up!?  So check you firewall rules on the RD Gateway Server and set the rules accordingly. Do the same for your perimeter firewalls or any other in between your users and your RD Gateway.

image

Under properties of your RS Gateway server you need to make sure UDP is enabled and listening on the needed IP address(es)

image

A client who connects over your RDS Gateway server, Windows Server 2012(R2) that is, and checks the network connection properties (click the “wireless NIC” like icon in the connection bar) sees the following: UDP is enabled. imageIf they don’t see UDP as enabled and they aren’t running Windows 8 or 8.1 (or W2K12R2) they can upgrade to RDP 8.1 on windows 7 or Windows Server 2008 R2! When they connect to a Windows 7 SP1 or Windows 2008R2  machine make sure you read this blog post Get the best RDP 8.0 experience when connecting to Windows 7: What you need to know as it contains some great information on what you need to do to enable RDP 8/8.1 when connecting to Windows 7 SP1 or Windows 2008 R2:

  1. “Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Remote Session Environment\Enable Remote Desktop Protocol 8.0” should be set to “Enabled”
  2. “Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Connections\Select RDP Transport Protocols” should be set to “Use both UDP and TCP” => Important: After the above 2 policy settings have been configured, restart your computer.
  3. Allow port traffic: If you’re connecting directly to the Windows 7 system, make sure that traffic is allowed on TCP and UDP for port 3389. If you’re connecting via Remote Desktop Gateway, make sure you use RD Gateway in Windows Server 2012 and allow TCP port 443 and UDP port 3391 traffic to the gateway

Cool you’ve done it and you verify it works. Under monitoring in the RD Gateway Manager you can see 3 connections per session: one is HTTP and the two others are UDP.

image

Life is good. But if you want to see the difference really well demonstrated try to connect to Windows 7 SP1 computer with RDP8 & TCP/UDP disabled and play a YouTube video, then to the same with RDP8 & TCP/UDP enabled, the difference is rather impressive. Likewise if you leverage RemoteFX in VM. The difference is very clear in experience, just try it! While you’re doing this look a the UDP “Kilobytes Sent” stats (refresh the monitoring tab, you’ll see UDP being put to work when playing a video on in your RDP session.

image

Load balancing Hyper-V Workloads With High To Continuous Availability With a KEMP Loadmaster


I’m working on some labs and projects with KEMP Loadmaster load balancing appliances (LM 2400, LM-R320) That will lead to some blog post on  load balancing several workloads, which are all on Windows Server 2012 R2  Hyper-V or integrate in to Azure. The load balancers used in the labs are the virtual appliances, depending on the needs and environment these are a very good, cost effective option for production as well and depending on the version you get they scale very well. Hence their use in cloud environments, they will not hold you back at all!

To stimulate your interest in load balancing and high availability I’ve put up a video on load balancing RD Gateway services. Consider it a teaser or introduction to more about the subject.

Why use an appliance (hardware/virtual)? Well let’s look at the 2 alternatives:

  • Round robin DNS, which is also sometimes used is just to low tech for most real life scenarios and sometimes can’t be used or is less efficient which impacts scalability and performance. On top of that it doesn’t provide health checking for failover purposes.
  • I’ve also said  before that while Windows NLB  provides layer 4 load balancing out of the box it’s pretty basic. It also often causes a lot of network grief and the implementation can be tedious. This has not improved in an ever more virtualized & cloud based world. On top of that, when network virtualization comes into play you might paint yourself into a corner as those two don’t mix. But if that’s not a concern and you’re on a budget, I’ve used it with success in the past as well.

Load Balancing In An Ever More Demanding Virtualized & Cloudy World


We’ve been using the Kemp Loadmasters for many years now and they have served us very well. You might know that Microsoft Azure has a partnership with Kemp technologies to provide full featured load balancing in your public & hybrid cloud solutions. I pretty happy with that as when talk about load balancing with Microsoft we always end up discussing the need for more features and layer 7 support. I sometimes jokingly tease them that this is due to their Windows NLB legacy. While I have done some magic with that, it is way too limited for today’s (and yesterdays) demands and needs. Also the hacks they use to get it to work can’t be used in network virtualization. In the cloud Microsoft has the Azure Load Balancer. Whilst nice when combined with availability sets many of the current workloads need more. That’s exactly what the KEMP Virtual LoadMaster for Azure delivers in their partnership with Microsoft:

  • Layer 4, Layer 7 Load Balancing
  • Layer 7 (or Cookie) Persistence
  • SSL Offload/SSL Acceleration
  • Application Health Checking
  • Adaptive (Server Resource) Load Balancing
  • Layer 7 Content Switching
  • Application Acceleration: HTTP Caching, Compression & IPS

To me (and many other IT Pros) Kemp is the company that opened load balancing up to everyone on this planet with budget friendly but high value solutions. They took away the barrier to better & more capable load balancing for the masses. Furthermore they keep improving and I have seen many existing customers, including me get ever more benefits with the newer firmware releases, even on their entry level, older models like the LM2200 that are not for sale anymore. So you can keep using them or move them to the lab. They have great support and respond very quickly to vulnerabilities like Heartbleed, Shellshock and Poodle.

image 

Another benefit of this partnership is that we can use the load balancing solution we know and trust in all our environments: on premises (physical or virtual appliance), in the cloud & at our hosting companies. Partner ships with OEMs ensure that you can use the hardware you prefer (the DELL R320 is a nice example) and their Virtual Load Master now even extends into the cloud. So our options are to …

… deploy an appliance …

image

…  virtualize the LoadMasters …

image

… leverage Kemp in the cloud

image

…. or select your own preferred OEM …

image

They cover all our bases with that line up and it helps with operational ease & efficiencies.

As I’m investigating some scenarios with KEMP LoadMasters in a Hyper-V environment (on premises, multi sites, Azure IAAS & Multifactor Authentication you can expect to see some blog posts on this. Some of these will leverage technologies available in Windows Server vNext (Technical Preview). Lot’s of very interesting ideas to support high availability & flexibility that are affordable and not just point solutions.

Ah the joy of being in virtualization is that one gets great exposure to storage, networking, cloud solutions and on premises. The experience & knowledge of the entire stack isn’t just fun (yes working can be fun) but it is also what allows to build great solutions.

Windows Server Technical Preview delivers integration services updates through Windows Update


Benefits of delivering updates to the integration services via Windows Updates

In Windows Server  vNext aka the Technical Preview the integration services are being delivered through Windows Update (and as such the well know tools such a s WSUS, …). This is significant in reducing the operational burden to make sure they are up to date. Many of us turned to PowerShell scripting to handle this task. So did I and I still find myself tweaking the scripts once in a while for a condition I had not dealt with before or just to get better feedback or reporting. Did I ever tell you that story about the cluster where a 100VMs did not have a virtual DVD drive (they removed them to improve performance) … that was yet another improvement to my script => detect the absence of a virtual DVD drive. In this day and age, virtualization has both scaled up and out with ever more virtual machines per host and in total. The process of having to load an ISO in a virtual DVD drive inside a virtual machine to install upgrades to integration services seems arcane and it’s very timely that it has been replaced by an operation process more befitting a Cloud OS Winking smile.

I have optimized this process with some PowerShell scripting and it wasn’t to painful anymore. The script upgrades all the VMs on the hosts and even puts them back in the state if found them in (Stopped, Saved, Running). A screenshot of the script in action below.

image

I’m glad that it’s now integrated through Windows Update and part of other routine maintenance that’s done on the guests anyway.

But is not only good news for us “on premises” system administrators and integrators. It’s also important for service/cloud providers and (hosted) private cloud hosters. This change means that the tenants  have control of updates to the integration services of their virtual machines. They update their Windows virtual machines with all updates during their normal patch cycles and now this includes the integration services. This provides operation ease (single method) and avoids some of the discussions about when to upgrade the integration services.

Legacy Operating Systems

Shortly after the release of the Windows Server Technical Preview, updates to integration services for Windows guests began being distributed through Windows Update. This means that on that version the vmguest.iso is no longer needed and as such it’s no longer included with Hyper-V.  This means that if you run an unsupported (most often legacy) version of Windows you’ll need to grab the latest possible vmguest.iso from an W2K12R2 Hyper-V host and try to install that and see if it works.

What about Linux and FreeBSD?

Well nothing has changed and how that’s taken care of you can read here: Linux and FreeBSD Virtual Machines on Hyper-V

Hyper-V Technical Preview Live Migration & Changing Static Memory Size


I have played with  Hot Add & Remove Static Memory in a Hyper-V vNext Virtual Machine before and I love it. As I’m currently testing (actually sometimes abusing) the Technical Preview a bit to see what breaks I’m sometimes testing silly things. This is one of them.

I took a Technical Preview VM with 45GB of memory, running in a Technical Preview Hyper-V cluster and live migrate it.  I then tried to change the memory size up and down during live migration to see what happens, or at least nothing goes “BOINK”. Well, not much, we get a notification that we’re being silly. So no failed migrations, crashed or messed up VMs or, even worse hosts.

image

It’s early days yet but we’re getting a head start as there is a lot to test and that will only increase. The aim is to get a good understanding of the features, the capabilities and the behavior to make sure we can leverage our existing infrastructures and software assurance benefits as fast a possible. Rolling cluster upgrades should certainly help us do that faster, with more ease and less risk. What are your plans for vNext? Are you getting a feeling for it yet or waiting for a more recent test version?

Hyper-V Amigos Showcast Episode 7: The Year 2014 in Review


In the seventh edition of the Hyper-V Amigos Carsten Rachfahl and I looked back at 2014, talk  a bit about our future plans and basically have some fun Smile

You can watch it here or on YouTube. We’ll be back this year with more show casts & we hope to see you there!

A MVP once more in 2015 – happy New Year from a renewed MVP


Happy New Year people! May 2015 bring you happiness,  good health,  and good jobs/projects/customers with real opportunities for growth & advancement. Don’t forget to step out of the office, away from the consoles once in a while to enjoy the wonderful experiences and majestic views this world has to offer.

image

Being an January 1st MVP (my expertise is Hyper-V) means that every year on new years day I might get an e-mail to inform me I have been renewed, or not. Prior to that our MVP lead will contact us to make sure we have updated our community activities and they’ll decide on whether we’re MVP material, or not.  Today I received this e-mail awarding me the MVP award for 2015.

image

It remains a special feeling to receive the award.  It’s recognition for what you’ve done and it means that I can enjoy the benefits that come with it: the MVP Global Summit and the interaction with the product groups at Microsoft. The summit is very valuable to me and if I knew the dates I would already book my flights and the hotel right now.

Some people think it makes us fan boys but I can assure you that’s not the case. Microsoft hears the great, the good, the bad and the ugly from us. And yes, they appreciate that as they cannot and do not want to live in an Ivory tower. So they need feedback and we’re a part of the feedback loop. We MVPs are a good mix of customers, consultants, partners & businesses working with their technologies & helping out the community to make the best use of them. Microsoft puts it like this:

“The Microsoft Most Valuable Professional (MVP) Award is our way of saying thank you to exceptional, independent community leaders who share their passion, technical expertise, and real-world knowledge of Microsoft products with others.”

The fact that we are independent is an important factor here. It makes us a valuable resource pool of hands on experience to mix in with other feedback channels. As Aidan Finn wrote in his blog post, Feedback Matters Once Again In Microsoft, it does indeed matter. Again? It always did but they listen more and better now Winking smile. They don’t need an "echo chamber" they value opinions, insights and experiences. The MVP award is for the things you’ve done and do. Sure, there is a code of conduct but that doesn’t mean you cannot voice your concerns. "Independent" means that what we say doesn’t have to be sugar coated marketing. Our value is in the fact that we help out the community (their customers, partners and Microsoft itself) in the better use and development of their solutions base on our real world experiences. Microsoft discusses that here.

It opens up doors and creates opportunities, and for that I’m grateful as well. For my employers/customers it means that when you hire me you get access to not just my skills and expertise but to the collective knowledge and experience of a global network of passionate experts that have a proven track record of engagement and are recognized internationally for that. Not too shabby is it Winking smile.