Using RAMDisk To Test Windows Server 2012 Network Performance

I’m testing & playing different Windows Server 2012 & Hyper-V networking scenarios with 10Gbps, Multichannel, RDAM, Converged networking etc. Partially this is to find out what works best for us in regards to speed, reliability, complexity, supportability and cost.

Basically you have for basic resources in IT around which the eternal struggle for the prefect balance finds place. These are:

• CPU
• Memory
• Networking
• Storage

We need both the correct balance in capabilities, capacities and speed for these in well designed system. For many years now, but especially the last 2 years it very save to say that, while the sky is the limit, it’s become ever easier and cheaper to get what we need when it comes to CPU, Memory. These have become very powerful, fast and affordable relative to the entire cost of a solution.

Networking in the 10Gbps era is also showing it’s potential in quantity (bandwidth), speed (latency) and cost (well it’s getting there) without reducing the CPU or memory to trash thanks to a bunch of modern off load technologies. And basically in this case it’s these qualities we want to put to the test.

The most trouble some resource has been storage and it has been for quite a while. While SSD do wonders for many applications the balance between speed, capacity & cost isn’t that sweet as for our other resources.

In some environments were I’m active they have a need for both capacity and IOPS and as such they are in luck as next to caching a lot of spindles still equate to more IOPS. For testing the boundaries of one resource one needs to make sure non of the others hit theirs. That’s not easy as for performance testing can’t always have a truck load of spindles on a modern high speed SAN available.

RAMDisk to ease the IOPS bottleneck

To see how well the 10Gbps cards with and without Teaming, Multichannel, RDMA are behaving and what these configuration are capable of I wanted to take as much of the disk IOPS bottle neck out of the equation as possible. Apart from buying a Violin system capable of doing +1 million IOPS, which isn’t going to happen for some lab work, you can perhaps get the best possible IOPS by combining some local SSD and RAMDisk. RAMDisk is spare memory used as a virtual disk. It’s very fast and cost effective per IOPS. But capacity wise it’s not the worlds best, let alone most cost effective solution.

I’m using free RAMDisk software provided by StarWind. I chose this as they allow for large sized RAMDisks. I’m using the ones of 54GB right now to speed test copying fixed sized VHDX files. It install flawlessly on Windows Server 2012 and it hasn’t caused me any issues. Throw in some SSDs on the servers for where you need persistence and you’re in business for some very nice lab work.

You also need to be aware it doesn’t persist data when you reboot the system or lose power. This is not an issue if all we are doing is speed testing as we don’t care. Otherwise you’ll need to find a workaround and realize those ‘”flush the data to persistent storage” isn’t full proof or super fast, the SSDs do help here.

You have to register but the good news is that they don’t spam you to death at all, which I find cool. As said the tool is free, works with Windows Server 2012 and allows for larger RAMDisks where other free ones are often way to limited in size.

It has allowed me to do some really nice testing. Perhaps you want to check this out as well. WARNING: The below picture is a lab setup … I’m not a magician and it’s not the kind of IOPS I have all over the datacenters with 4 Cheapo SATA disks I touched my special magic pixie dust.

Some quick tests with a 52GB NTFS RAMDisk formatted with a 64K NTFS Allocation unit size.

I also tested with another free tool from SoftPerfect ® RAM Disk FREE. It performs well but I don’t get to see the RAMDisk in the Windows Disk Management GUI, at least not on Windows Server 2012. I have not tested with W2K8R2.

NTFS Allocation unit size 4K	NTFS Allocation unit size 64K

MVP Carsten Rachfahl Visits & Interviews Me On Networking & Storage in Windows Server 2012

Last month Carsten (MVP – Virtual Machine) & Kerstin Rachfahl (MVP – Office 365) visited me in my home town. Apart from a short visit to the historic center & a sushi diner amongst friends we also did an interview where we discussed our ongoing Windows Server 2012 Hyper-V activities. We’re trying to leverage as much of the product we can to get the best TCO & ROI and as early adopters we’ve been reaping the benefits form the day the RTM bits were available to us. So far that has been delivering great results. Funny to hear me mention the Fast Track designs as a week later we saw version 3 of those at MMS2013. The most interesting to me about those was the fact that the small & medium sizes focus on Cluster in a Box and Storage Spaces!

While we were having fun talking about the above we also enjoyed some of the most beautiful landmarks of the City of Ghent as a back drop for the interview. It was filmed in a meeting room at AGIV, to whom I provide Infrastructure services with a great team of colleagues. Just click the picture to view the video.

You can also enjoy the video on Carsten’s blog http://www.hyper-v-server.de/videos/interview-mit-didier-van-hoye-ber-seinen-storage-netwerk-und-mehr/ All I need to do now is to arrange for Carsten to physically touch the Compellent storage I think.

Windows Hyper-V Server 2012 Live Migration DOES support pass-through disks–KB2834898 is Wrong

See update in yellow in line (April 11th 2013)

I recently saw KB2834898 (pulled) appear and it’s an important one. This fast publish statement is important as until recently it was accepted that Live Migration with pass through disks was supported with Windows Server 2012 Hyper-V Live Migration (just like with Windows Server 2008 R2 Hyper-V) as long as the live migration is managed by the Hyper-V cluster, i.e. the pass through disk is a clustered resource => see http://social.technet.microsoft.com/wiki/contents/articles/440.hyper-v-how-to-add-a-pass-through-disk-on-a-failover-cluster.aspx

UPDATE April 11th 2013: Now after consulting some very knowledgeable people at Microsoft (like Jeff Woolsey and Ben Armstrong) this KB article is not factual correct and leaves much to be desired. It’s wrong, as pass-through disks are still supported  with Live Migration in Windows Server 2012 Hyper-V, when managed by the cluster, just like before in Windows 2008 R2. The KB article has been pulled meanwhile.

Mind you that Shared Nothing Live Migration with pass through disks have never been supported as there is no way to move the pass through disk between hosts. Storage Live Migration is not really relevant in this scenario either, there are no VHDX file to copy apart fro the OS VHDX. Live migrations between stand alone host are equally irrelevant. Hence it’s a Hyper-V Cluster game only for pass through disks.

I have never been a fan of pass through disks and we have never used them in production. Not in the Windows Server 2008 R2 era let alone in the Windows Server 2012 time frame. No really we never used them, not even in our SQL Server virtualization efforts as we just don’t like the loss of flexibility of VHDX files and due to the fact that they tend to complicate things (i.e. things fail like live migration).

I advise people to strongly reconsider if they think they need them and only to use them if they are really sure they actually do have a valid use case. I know some people had various reasons to use them in the past but I have always found them to be a bit of over engineering. One of the better reasons might have been that you needed disks larger then 2TB but than I would advise iSCSI and now with Windows Server 2012 also virtual Fibre Channel (vFC), which is however not needed due to VHDX now supporting up to 64TB in size. Both these options support Live Migration and are useful for in guest clustering, but not as much for size or performance issues in Windows Server 2012 Hyper-V. On the performance side of things we might have eaten a small IO hit before in lieu of the nice benefits of using VHDs. But even a MSFT health check of our Virtualized SQL Server environment didn’t show any performance issues, Sure your needs may be different from ours but the performance argument with Windows Server 2012 and VHDX can be laid to rest. I refer you to my blog Hyper-V Guest Storage Performance: Above & Beyond 1 Million IOPS for more information of VHDX performance improvements and to Windows Server 2012 with Hyper-V & The New VHDX Format Leads The Way for VHDX capabilities in general (size, unmap, …).

Is see only one valid reason why you might have to use them today. You have  > 2TB disks in the VM and your backup vendor doesn’t support the VHDX format. Still a reality today unfortunately But that can be fixed by changing to another one

Saying Goodbye To Old Hardware Responsibly

Last year we renewed our SAN storage and our backup systems. They had been serving us for 5 years and where truly end of life as both technologies uses are functionally obsolete in the current era of virtualization and private clouds. The timing was fortunate as we would have been limited in our Windows 2012, Hyper-V & disaster recovery plans if we had to keep it going for another couple of years.

Now any time you dispose of old hardware it’s a good idea to wipe the data securely to a decent standard such as DoD 5220.22-M. This holds true whether it’s a laptop, a printer or a storage system.

We did the following:

• Un-initialize the SAN/VLS
• Reinitialize the SAN/VLS
• Un-initialize the SAN/VLS
• Swap a lot of disks around between SAN/VLS and disk bays in a random fashion
• Un-initialize the SAN/VLS
• Create new (Mirrored) LUNS, as large as possible.
• Mounted them to a host or host
• Run the DoD grade  disk wiping software against them.
• That process is completely automatic and foes faster than we were led to believe, so it was not really such a pain to do in the end. Just let it run for a week 24/7 and you’ll wipe a whole lot of data. There is no need to sit and watch progress counters.
• Un-initialize the SAN/VLS
• Have it removed by a certified company that assures proper disposal

We would have loved to take it to a shooting range and blast the hell of of those things but alas, that’s not very practical nor feasible . It would have been very therapeutic for the IT Ops guys who’ve been baby sitting the ever faster failing VLS hardware over the last years.

Here’s some pictures of the decommissioned systems. Below are the two old VLS backup systems, broken down and removed from the data center waiting disposal. It’s cheap commodity hardware with a reliability problem when over 3 years old and way to expensive for what is. Especially for up and out scaling later in the life time cycle, it’s just madness. Not to mention that those thing gave us more issues the the physical tape library (those still have a valid a viable role to play when used for the correct purposes). Anyway I consider this to have been my biggest technology choice mistake ever. If you want to read more about that go to Why I’m No Fan Of Virtual Tape Libraries

To see what replaced this with great success go to Disk to Disk Backup Solution with Windows Server 2012 & Commodity DELL Hardware – Part II

The old EVA 8000 SANs are awaiting removal in the junk yard area of the data center. They served us well and we’ve been early customers & loyal ones. But the platform was as dead as a dodo long before HP wanted to even admit to that. It took them quite a while to get the 3Par ready for the same market segment and I expect that cost them some sales. They’re ready today, they were not 24-12 months ago.

So they’ve been replaced with Compellent SANs. You can read some info on this on previous blogs Multi Site SAN Storage & Windows Server 2012 Hyper-V Efforts Under Way and Migration LUNs to your Compellent SAN

The next years the storage wares will rage and the landscape will change a lot. But We’re out of the storm for now. We’ll leverage what we got . One tip for all storage vendors. Start listening to your SME customers a lot more than you do now and getting the features they need into their hands. There are only so many big enterprises so until we’re all 100% cloudified, don’t ignore us, as together we buy a lot of stuff to. Many SMEs are interested in more optimal & richer support for their windows environments if you can deliver that you’ll see your sales rise. Keep commodity components, keep building blocks and from factors but don’t use a cookie cutter to determine our needs or “sell” us needs we don’t have. Time to market & open communication is important here. We really do keep an eye on technologies so it’s bad to come late to the party.

Are Data Tsunamis Inevitable Or Man Made Disasters?

What happens when people who have no real knowledge and context about how to handle data, infrastructure or applications insist on being in charge and need to be seen as taking strong decisive actions without ever being held responsible? It leads to real bad, often silly decisions with a bunch of unintended consequences. Storage vendors love this. More iron to sell. And yes, all this is predictable. When I’m able and allowed to poke around in storage and the data stored I often come to the following conclusion: there’s a bulk amount of data that is stored in an economical unsound fashion. Storage vendors & software vendors love this, as there are now data life cycle management tools & appliances to be sold.

The backlash of all this is? Cost cutting, which then leads to the data that has valid needs to be stored and protected not getting the resources it should. Why? Well who’s going to take responsibility to push the delete button to remove the other data? As we get ever better technology to store, transport and protect data we manage to do more with less money and personnel. But as is often the case, no good deed goes unpunished. Way to often these savings or efficiencies flow straight into the bottomless pit caused by that age old “horror vacui” principle in action in the world of data storage.

You get situations like this: “Can I have 60TB of storage?  It’s okay, I discussed this with your colleague last year, he said you’d have 60TB available at this time frame”

What is the use case? How do you need it? What applications or services will consume this storage? Do you really need this to be on a SAN or can we dump this in cost effective Windows Server Storage Spaces with ReFS? What are the economics involved around this data? Is it worth doing? What projects is this assigned to? Who’s the PM? Where is the functional analysis. Will this work? Has there been a POC? Was that POC sound? Was there a pilot? What the RTO? The RPO? Does it need to be replicated off site? What IOPS is required? How will it be accessed? What security is needed? Any encryption required? Any laws affecting the above? All you get is a lot of vacant blank stares and lot’s of “just get it done”. How can it be that with so many analysts and managers of all sorts running around to meeting after meeting, all in order to get companies running like a well oiled slick mean machine, we end up with this question at the desk of an operational systems administrator as a result? Basically what are you asking for? Why are you asking this and did you think this through?

Consider the following. What if you asked for 30 billion gallons of water at our desk and we say “sure” and just sent it to you. We did what you asked. Perhaps you meant bottled drinking water but below is what you’ll end up with. And yes it completely up to specifications, limited as they are.

The last words heard while drowning will be “Who ordered this? You can bet no one will be responsible, especially not when the bill arrives and when the resulting mess needs to be cleaned up. Data in the cloud will not solve this. Like the hosting business, who serve up massive amount of idle servers, the cloud will host massive amounts of idle data as in both situations it’s providing the service that generates revenue, not the real use of that service by you or it’s economic value to you.

Money Saving Hero of 2012: Windows 2012 In Box Deduplication Delivers Big Value

To wave goodbye to 2012 I’m posting the latest screenshot of the easiest and very effective money saving feature you got in Windows Server 2012 than RTM in August. Below you’ll find the status report of a backup LUN in a small environment.  Yes those are real numbers in a production environment.

If you are not using it; you’re really throwing away vast amounts of money on storage right this moment. If you’re in the market for a practical, economical and effective backup solution my advice you to  is the following. Scrap any backup vendor or product that prevents it files of LUNs being duplicated  by Windows Server 2012.  They might as well be robbing you at gun point.

You can pay for a very nice company new years party with these savings

I wish you all a great end of 2012 and a magnificent 2013 ahead. In 2013 we’ll push Windows Server 2012 into service where we couldn’t before (waiting for 3 party vendor support and if they keep straggling they are out of the door) and work at making our infrastructure ever more resilient an protected.  With System Center SP1 some products of that suite will make a come back in our environment. 10Gbps is bound to become the standard all over our little data center network and not just our most important workloads.

You Got To Love Windows Server 2012 Deduplication for Backups

I’ve discussed this before in Windows Server 2012 Deduplication Results In A Small Environment but here’s a little updated screenshot of a backup volume:

Not to shabby I’d say and 100% free in box portable deduplication … What are you waiting for

Carsten Rachfahl Interviews Me On Windows Server 2012 Storage Improvements

Carsten Rachfahl, a German Hyper-V Expert, friend and fellow MVP, interviewed me after the joint MVP effort at TEC 2012 in Barcelona. The subject was storage in Windows Server 2012. We found a great setting in the garden and got into quite a nice discussion on the subject.

It’s no surprise to anyone I guess that I’m very enthusiastic about what Microsoft is doing with storage on all levels in Windows Server 2012 and is trying to achieve for us, the customers from both a cost and performance and reliability perspective. It was a lot of fun to do and I see blinking lights in our eyes at many moments during this interview. Yes, working is important for many reasons, but when you can enjoy your work and have fun whilst doing it, life is pretty good . So enjoy, we certainly did.

Windows Server 2012 VHDX Thin Provisioning Benefits Explored

Thin Provisioning With Hyper-V

Windows Server 2012 provides thins provisioning at the virtual layer via the VHDX file format. It also provides it at the physical storage layer when your storage supports it. For the later don’t forget that this also means Storage Spaces! So even in environments where budgets are really tight you can leverage this on the physical storage now. So its not just for the feature rich SAN owners anymore .

Even if you use a storage sub system that does not support thin provisioning at the physical layer you will benefit from this mechanism when you use dynamic VHDX files. Not only will these grow less but during shut down they shrink by the size of the empty blocks. Pretty cool! I do however see a potential risk for increased fragmentation. This has a negative impact on performance and needs defragmentation to remediate which also has an impact on IO performance. How much this is a concern depends on your environment and needs. We’ll also have to see in real life how well dynamic VHDX files live up to their performance improvements they got with Windows Server 2012 to entice more people to use this. You have proponents and naysayers. I’m selective and let the circumstances and needs/requirements decide.

Thin Provisioning at the Virtual Layer

You can take a look at the TechEd 2012 session VIR301 by Senthil Rajaram to see how VHD versus VHDX behaves in regards to thin provisioning. I will not repeat all of this here. What I am going to do is look at some other situations.

Important note: You get this UNMAP feature automatically in Windows. There’s no need to manually run the Disk-Optimize command we’ll use in the scenarios below. It’s run automatically for us when the standard Defrag scheduled task runs.  So this mechanism that will normally take care of all that. But as that “only” runs every week by default you might want to tweak it or create your own scheduled task in your environment if needed. In demos and labs we’re rather inpatient geeks so we run “Optimize-Volume  -DriveLetter X –ReTrim” to get immediate gratification while testing or kick of the job manually. In real life it’s zero touch feature, you don’t need to baby sit it.

Fixed VHDX versus Dynamic VHDX

Apart from the fact that you’ll have no shrink on shutdown this optimization does nothing for the file size. The only benefit here is that the UNMAP can be passed to the physical storage where it can help if that supports it. At the virtual layer it doesn’t matter for a fixed sized VDHX disk.

Dynamic VHDX Disk

You’ll profit from the savings in storage when the dynamically expanding VHDX file doesn’t need to grow as much this. This reduces the overhead of expanding the disk, which is a performance benefit and it even helps your non thin provisioning capable storage go further.

Watch Senthil’s presentation (from around minute 20) to see the benefits in action. With VHDX, If you “shift delete” the files inside the VM, then run “Optimize-Volume -DriveLetter X –ReTrim” or  the defrag job and then copy new files  you’ll see that there is no additional file growth as long as you don’t exceed the current size of the VHDX. If you don’t do this both the VHD and VHDX file will grow.

But is another potential benefit why this might be important. Even with the block sizes that have been increased to have less overhead when growing dynamic VDHX files we still have to deal with fragmentation of the VHDX files on the storage where they live. The better/more empty blocks are reused, the less the dynamic files will have to grow. This means you’ll have less opportunity for fragmentation. Whether this compensates for potential of more fragmentation due to the shrinking when they are shutdown I don’t know. If all the performance improvements for dynamic disks are good enough will depend on your environment and needs. Defragmentation can help mitigate this but IO performance during the defragmentation process suffers. Do it or better, schedule it, wisely!

Virtual SCSI controller attached versus virtual IDE controller attached

What about a guest (boot) VHDX disk attached to an IDE controller? I see a lot of one disk virtual machines out there, so it would be a pity if it didn’t work for those and just for the one who have extras vSCSI disk attached.So let’s test this.

Below you see the disk size of the VHD and VHDX files and what type of controller they are attached to. As you can see this they had one or two 3.3 GB ISO files copied to them and where then “shift deleted”. The size of the VHD(X) files reflects the amount of data that they stored.

Now after running the defrag job or executing “Optimize-Volume -DriveLetter X –ReTrim” inside the VM you’ll see the results below after you shut down the VM

So as it turns out, the thin provisioning benefits it work with an IDE attached VHDX files as well! Yes inside a Windows Server 2012 virtual machine you get the UNMAP support with IDE attached VHDX disks to. Think of Hosting companies with many thousands single disk virtual machines who can leverage this as well. So this is something you might not expect when having watch the video as there they only talk about virtual SCSI/ FC controllers.

Conclusion

Doing tests like these are a bit artificial but they do demonstrate how the technology works. In real life it will translate into efficiencies over time, based on the data creation and deletion in your VHDX files. Think about hundreds or thousands of virtual machines in your environment leveraging this mechanism. Over time, on that scale, the amount of storage consumed will be reduced which results in better economies. Now leverage that together with thin provisioning support in Storage Spaces and you see that there are some very interesting scenarios to investigate. Some how it’s starting to look like you can have your cookie and eat it to . You don’t need an expensive SAN to get these efficiencies at the physical storage layer, but if you have and use to have to mess around with sdelete or agents, it’s easy to see the benefit you get from this here as well.

Dell Storage Forum 2012 Paris – Fluid Forward Think Tank

Thanks to some great people at Dell in Germany (yes, you Florian), Belgium and of course Alison Krause (@AlisonatDell), Maryna Frolova  (@MarineroF) and Stephanie Woodstrom I got invited to attend the “Fluid Forward Think Tank” at the Dell Storage Forum in Paris.

We had a healthy variation in customers, partners, consultants and DELL employees discussing various aspects of IT related to storage. The task of herding the cats fell upon the shoulders of Simon Robinson (@simonrob451) who’s an Analyst and VP at 451 Research, a firm that deals with storage and information management. I for one think he did so brilliantly. This interactive discussion was streamed live and if you missed it you can click on this live stream link to look at our ramblings

I had to pitch some of my dreams of leveraging al the new mobility features as well as the high to continuous available that is being enabled with Windows Server 2012 Hyper-V on inherently unreliable components what opportunities these present to us customers and storage vendors.

Here’s the gang around the table:

• Andrew Mauro, @andrea_mauro
• Fabio Rapposelli, @fabiorapposelli
• Barry Coombs, @virtualisedreal
• Didier Van Hoye, @WorkingHardInIT
• Bruno Sousa, @bsousapt
• Rasmus Haslund, @haslund
• Charles Gillanders, @cgillanders
• Andreas Erson, @ersontech
• Tom Heskestad  @hypervfan

It was a fun, educational discussion as the mix of backgrounds, industries, job functions was diverse enough to address all sides of the storage story, the good, the bad and the ugly. We gave them some food for taught I think. Well the folks at DELL can now take this back to Austin and reflect on it all. If need be, I’ll drop by some day to provide some feedback and remember @WarrenByle I ‘d like to try out that STI of his   After an interview I ran of to a Compellent customer panel to learn something and provide some feedback on our first experiences.