Windows Server 2012 64TB NTFS Volumes and the Flush Command

As you might very well have read or even tried you can use 64TB volumes in Windows Server 2012 in a supported scenario. You can do more, NTFS is quite capable of this. I created a 300TB LUN once that I could format up to 256TB But as no one can realistically stress test this for real, it’s not supported.

That’s a lot of storage and data. It’s also expensive and incurs some risk … all that data on one volume. Windows 2012 tries to address the cost issue with commodity storage in combination with the excellent resilience of storage space to reduce both cost and risk.

Apart from introducing ReFS they also did some work on NFTS to help with reliability:

• A new approach for detecting and repairing corruptions in NTFS which optimizes uptime through on line repair and with spot fixing that keeps off line repairs minimized and very short.
• Using the flush command instead of FUA.

In this post this we’ll focus on the flush command.

Flushing Your Data

No, not that kind of flushing You have always been able to “throw” data away with some very bad practices and unreliable technology, no need for much innovation there.

I’m talking about the fact that NTFS in Windows Server 2012 has switched to the flush command instead of relying on Forced Unit Access (FUA) to increase reliability for SATA disk and performance with SCSI disks. The good news is you don’t lose anything and gain on both fronts. Especially making cheaper SATA disks more reliable is a big one. It allows SATA disks to be used in business/enterprise scenarios and as such helps reduce costs.

What is Forced Unit Access (FUA)?

Well it’s a flag that indicates a given write should go directly to media, writing through a devices write cache. The NTFS Journaling File System uses FUA to guarantee write ordering which is important to maintain its metadata integrity. It was  implemented in the SCSI (T10) specification but not in the original  ATA (T13) specification. This was added in the 2002 version of the ATA specs but FUA has never been guaranteed to implemented on all ATA devices and as such Windows could not rely on it being there with ATA/SATA disks. As a result it was never used by Windows with SATA disks.

That meant that with SATA disks there is a bigger change of corruption due to a power failure or the likes as NTFS was designed to rely on FUA implementation for robust metadata writes.With ever increasing capacity needs an larger SATA disk being needed and used for business purposes something had to be done. So with Windows Server 2012 (and Windows 8) NTFS switched to using a  flush command to the drives write cache instead of using FUA.

The Benefits

1. The switch to using the flush command for all operations that require write ordering to ensure file system metadata integrity realizes better reliability and robustness when using commodity SATA storage as it reduces possibility of corruption due to power loss
2. It Improves performance on SCSI devices because it allows the disk to cache data for as long as safely possible instead of having to do write-through using FUA

I’m off to Attend MMS 2012 In Las Vegas

Life is good people. I have to good fortune to work in an interesting industry, doing great projects with modern technologies. On top of that my employer allows me to fully develop my skills . In that respect it makes a serious difference to have a good boss & management that understands the benefits of ongoing education. They look a both the short & long term value of people educating & developing themselves a lot more than at that nagging Excel sheet on the screen. Professional development is not just a cookie cutter 4 day training course once or twice a year but real opportunities to become a better professional if and when you’re willing to put in the effort. They’ve figured out that you cannot just use utmost cost reduction to catapult both your business and employees in to prosperity & wellbeing. You need to keep learning, evolving, networking, … The contacts I make and the education I get by working with and learning along very smart & motived people are priceless. Sure it costs money and effort form everyone involved but it beats doing nothing and saving a few € as a long term strategy for growth & success. On top of that I feel appreciated & valued for my contributions and the efforts I put in.So to the tunes of some eighties rockers I’m off again.

Here I go again on my own, goin’ down the only road I’ve ever known.
Like a drifter I was born to walk alone. An’ I’ve made up my mind, I ain’t wasting no more time. I’m attending MMS 2012

Alone, heck no, many thousands of us will be descending on Las Vegas (Nevada, USA) to attend the summit. This event sells out fast each year. A friend told me to register a.s.a.p. or miss out, so I did as soon as I got the go ahead to attend, securing my spot. So now I’m travelling over LHR to LAS following my buddies & other attendees journey from their respective countries to Las Vegas on line, mostly via Twitter.

If you can’t come, whatever the reason, you can always enjoy a good number of sessions here MMS 2012 goes digital: LIVE streaming and On-Demand for attendees AND non-attendees! 48 hours after the live presentation.

I don’t have to tell you what System Center 2012 means to the IT Pro in the Microsoft ecosystem. Combine that with the RTM of Windows 8 later this year and I just had to go and attend the Microsoft management Summit 2012 in Las Vegas.  It’s more than training. It’s networking and an education.

Apart from the formal agenda & sessions I already a have some meetings lined up with vendors, colleagues from around the globe. We’re making the most of this opportunity to meet face to face with people we other wise only get to talk to on line and often with huge time zone difference.

I’ve you’re going and you read my blog or follow me on twitter. Give us a shout out and perhaps we can have a meet & greet.

To all my geek & nerd friends, colleagues, MEET members, business partners, Microsoft employees & MVPs in route to Vegas & the Summit at The Venetian, I’m looking forward to seeing you all again! But first I have some traveling to do in the next 24 hours, to make my way over there.

Build Windows Key Note 2011/09/13

Updated as we follow the key note

After the talk about Windows 8 being even better and greater for all form factors (hardware people, the ARM architecture, it will be fun to see how the competition responds) I want to dive into Windows Server 8. Yes I’m here for the server side. But as the Hyper-V is now brought to the client there is a lot to say about Hyper-V here as well. No problem. But not yet, not yet.

First, mobile devices. Lots of touch, looks all very cool with the Metro UI. As I live in the country with the most expensive smart phones & mobile data subscriptions in the world I’m not a heavy user. It’s a great market, it’s cool, it’s important, but it’s not my primary theater of operation so to speak. But I might need to get me some of those devices to play with It really looks cool. It looks all very fast & fluid. And the resource hogging should be reduced. Bring it on I say But don’t worry if you’re a “Grand Pa Box” keyboard & mouse jockey. Windows 8 works just as well for you. the idea is Windows 8 everywhere on every device & form factor.

Now they first need to talk about all the developers will be writing applications for Windows 8. Here comes Metro Style applications development. The bold WinRT API bet (yet another one). The languages used are the one we all know, love or hate . No worries you’re coding skills have not been dumped into the toilet. Oh yes, Silverlight is not dead. An no .NET is not dead either. Really? Even COM+ is not dead yet. But Metro style development is the way ahead. But please dump the hyped drama and o continue coding on your current projects They promised everything that runs on Windows 7 today will run on Windows 8. There you go You might say with less drama that Win Forms & co will be less dominant. Nothing that new. New form factors & mobility ask for new tools. But guess what you’ll be coding those apps in? Metro Style apps will be written in C, C++, C#, Visual Basic, HTML5/JavaScript and/or using XAML. XAML is for “Jupiter,”which is the XAML/UI layer on top of Windows 8 needed for Silverlight and Windows Presentation Foundation (WPF) apps to work on the platform.

They are now coding on stage. Perhaps not the best use of time during a keynote but hey, we’ll get to the good stuff eventually. Once again we see the impression launched you can write apps in a couple of minutes with no knowledge at all. Take that devies! We IT Pro’s are not the only ones facing unemployment (cloud) we’re all going to be replaced by a very small easy script with drag & drop. I know some hard core consultants/developers who are now buying stocks in their own company to cash in on the fixing of all that

We’re treated to some very impressive hardware demos. Really impressive. Mobile device OS people we have met your competition and it is called Microsoft. The crowd goes wild when they are told they are getting a Samsung slate machine. Hmmm, why am I working instead of being at Build? My priorities are wrong I guess

We’re shown deep freeze, the new task manager that look pretty neat.  The command line  to set a base line for your machine refresh is very appealing to me. At a point you have your machine just right => grab it for refresh if/when needed.

Metro over RDP looks awesome remote charms, virtual keyboard and of cause touch! I bet the VDI crow is going a little wild dreaming of the possibilities straight out of the box.

Hyper-V on Windows 8 client! We’re there . The guy is storming through the features. He’s on the clock. We arrived at the business crowd. A lot of stuff for the desktop is also improved. Multiple Monitor support, control of Metro & desktop with shortcuts within the monitors. The UP button should be a good alternative to select delete in Window explorer paths. Lots of stuff to explore.

Windows Live integration with Windows 8 is extensive. The SkyDrive examples are impressive. Windows 8 will be the first mesh /hybrid / integrated OS. WinRT API exposes this so you can use that cloud extensibility in your Metro style apps!

Sorry if all this reads hectic, but it’s kind of hard to keep up. This is a tsunami of information! Keynote is wrapping up. The Hyper-V Windows 8 Server stuff will be for another day.

In the end a call to action for developers. Get the preview and get ahead of the pack delivering Metro style apps to a billion potential users. Up and at them developers!

Hyper-V Cluster Nodes Upgrade: Zero Down Time With Intel VT FlexMigration

Well the oldest Hyper-V cluster nodes are 3 + years old. They’ve been running Hyper-V clusters since RTM of Hyper-V for Windows 2008 RTM. Yes you needed to update the “beta” versions to the RTM version of Hyper-V that came later Bit of a messy decision back then but all in all that experience was painless.

These nodes/clusters have been upgraded to W2KR2 Hyper-V clusters very soon after that SKU went RTM but now they have reached the end of their “Tier 1” production life. The need for more capacity (CPU, memory) was felt. Scaling out was not really an option. The cost of fiber channel cards is big enough but fiber channel switch ports need activation licenses and the cost for those border on legalized extortion.

So upgrading to more capable nodes was the standing order. Those nodes became DELL R810 servers. The entire node upgrade process itself is actually quite easy. You just live migrate the virtual machines over to clear a host that you then evict from the cluster. You recuperate the fiber channel HBAs to use in the new node that you than add to the cluster. You just rinse and repeat until you’re done with all nodes. Thank you Microsoft for the easy clustering experience in Windows 2008 (R2)! Those nodes now also have 10Gbps networking kit to work with (Intel X520 DA SPF+).

If you do your home work this process works very well. The cool thing there is not much to do on the SAN/HBA/Fiber Switch configuration side as you recuperate the HBA with their World Wide Names. You just need to updates some names/descriptions to represent the new nodes. The only thing to note is that the cluster validation wizard nags about inconsistencies in node configuration, service packs. That’s because the new nodes are installed with SP1 integrated as opposes to the original ones having been upgraded to SP1 etc.

The beauty is that by sticking to Intel CPUs we could live migrate the virtual machines between nodes having Intel E5430 2.66Ghz CPUs (5400-series "Harpertown") and those having the new X7560 2.27Ghz CPUs (Nehalem EX “Beckton”). There was no need to use the “Allow migration to a virtual machine with a different processor” option.  Intel’s investment (and ours) in VT FlexMigration is paying of as we had a zero down time upgrade process thanks to this.

You can read more about Intel VT FlexMigration here

And in case you’re wondering. Those PE2950 III are getting a second life. Believe it or not there are software vendors that don’t have application live cycle management, Virtualization support or roadmaps to support. So some hardware comes in handy to transplant those servers when needed. Yes it’s 2011 and we’re still dealing with that crap in the cloud era. I do hope the vendors of those application get the message or management cuts the rope and lets them fall.

Download Microsoft Standalone System Sweeper Beta

Microsoft has released the beta version of Microsoft Standalone System Sweeper Beta. You can find more information and both the x86  and x64 versions to download over here on the connect site.

I know that all our environments and clients are well protected, patched and maintained but unfortunately this is not the case all over the board. So this tool can help you to address malware issues.

Microsoft describes the product as follows:

A recovery tool that can help you start an infected PC and perform an offline scan to help identify and remove rootkits and other advanced malware. In addition, Microsoft Standalone System Sweeper Beta can be used if you cannot install or start an antivirus solution on your PC, or if the installed solution can’t detect or remove malware on your PC.

Microsoft Standalone System Sweeper Beta is not a replacement for a full antivirus solution providing ongoing protection; it is meant to be used in situations where you cannot start your PC due to a virus or other malware infection. For no-cost, real-time protection that helps guard your home or small business PCs against viruses, spyware, and other malicious software, download Microsoft Security Essentials*.

To get started, please make sure that you have a blank CD, DVD, or USB drive with at least 250 MB of space. Next, download and run the tool – the tool will help you to create the bootable media required to run the software on your PC.

Yet another tool in your box. Happy hunting and if you do use it, please provide your feedback via the connect site. It only helps make the product better.

New KB Article 2494016 Related to Windows Server 2008 SP1 Hyper-V: Stop error 0x0000007a When Using CVS in Redirected Access

Well not a day after my blog post Extra Info on Clustering & Hyper-V with Dynamic Memory When You Start With Windows Server 2008 R2 SP1on important hotfixes for Hyper-V clustering with Windows Server 2008 R2 SP1 Microsoft releases a new hot fix for issue below. I’ll add it to the post to keep up to date.

Stop error 0x0000007a occurs on a virtual machine that is running on a Windows Server 2008 R2-based failover cluster with a cluster shared volume, and the state of the CSV is switched to redirected access

The KB article with instructions on how to get the hot fix is here: http://support.microsoft.com/kb/2494016/en-us?sd=rss&spid=14134

The scenario is detailed as follows:

Consider the following scenario:

• You enable the cluster shared volume (CSV) feature on a Windows Server 2008 R2-based failover cluster.
• You create a virtual machine on the CSV on a cluster node.
• You start the virtual machine on the cluster node.
• You move the CSV owner to another cluster node, and you change the state of CSV to redirected access.
• The connection that is used for redirected access is switched to another connection when one of the following scenarios occurs:
  • The cable for local area network (LAN) is disconnected.
  • The related network adapter is disabled.
  • The connection is switched by using Failover Cluster Manager.

In this scenario, you receive a Stop error message that resembles the following in the virtual machine:

STOP 0x0000007a ( parameter1 , parameter2 , parameter3 , parameter4 )
KERNEL_DATA_INPAGE_ERROR

Note

• The parameters in this Stop error message vary, depending on the configuration of the computer.
• Not all "0x0000007a" Stop error messages are caused by this issue.
• You may also receive other Stop error messages when this issue occurs. For example, you may receive a "0x0000004F" Stop error message.

Déjà vu Bug: The network connection of a running Hyper-V virtual machine may be lost under heavy outgoing network traffic on a computer that is running Windows Server 2008 R2 SP1

Anyone who’s been doing virtualization with Hyper-V on Windows 2008 R2 has a good change of having seen the issue described in http://support.microsoft.com/kb/974909/en-us

You install the Hyper-V role on a computer that is running Windows Server 2008 R2.

• You run a virtual machine on the computer.
• You use a network adapter on the virtual machine to access a network.
• You establish many concurrent network connections, or there is heavy outgoing network traffic.

In this scenario, the network connection on the virtual machine may be lost. Additionally, the network adapter is disabled.
Note You have to restart the virtual machine to recover from this issue.

We’ve seen this one on VM’s that have indeed a lot of outgoing traffic.  In our environment the situation looks like this:

• You can access the VM with Hyper-V Manager or SCVMM but not via RDP as all Network connectivity is lost.  The status the  guest NIS is always “Enabled” but there is no traffic/connectivity
• You can try to disable the NIC but this tales a  very long time and when you try to enable it again this never succeeds. Disconnecting the NIC form the virtual network and connecting it again doesn’t help either.
• You need to shut down the host but this takes an extremely long time, so long you really can’t afford to wait if it ever succeeds. It seems to hang at shutting down with a “non whirling whirly”.  So finally you’ll power off the VM and start it up again. Apart from entries related to having not connectivity the event logs are “clean” and there is no indication as to what happened.

Well this exact same issue is back with Windows 2008 R2 SP1. That’s the bad news. The good news is there is a hotfix for it already so you can fix it. You can read up on this issue in Knowledge Base article 2263829  and request the hotfix here. Instructions to get the hotfix are in there as well as a reference to the previous fixes for Windows 2008 R2 RTM.

Consider the following scenario:

• You install the Hyper-V role on a computer that is running Windows Server 2008 R2 Service Pack 1 (SP1).
• You run a virtual machine on the computer.
• You use a network adapter on the virtual machine to access a network.
• You establish many concurrent network connections. Or, there is heavy outgoing network traffic.

In this scenario, the network connection on the virtual machine may be lost. Additionally, the network adapter may be disabled.
Notes

• You must restart the virtual machine to recover from this issue.
• This issue can also occur on versions of Windows Server 2008 R2 that do not have SP1 installed. To resolve the issue, apply the hotfix that is described in one of the following Microsoft Knowledge Base articles:

  974909 (http://support.microsoft.com/kb/974909/ ) The network connection of a running Hyper-V virtual machine is lost under heavy outgoing network traffic on a Windows Server 2008 R2-based computer
  2264080 (http://support.microsoft.com/kb/2264080/ ) An update rollup package for the Hyper-V role in Windows Server 2008 R2: August 24, 2010

Oh yeah, people often seem confused  as to where to install the hotfix. Does it go on the Hyper-V hosts or and/or on the guest?  It’s a hyper visor bug in Hyper-V so it goes on the hosts. Have a nice weekend.

Kick Starting Your Windows 7 Deployments With Mastering Windows 7 Deployment

I have to hand it to Aidan Finn, he doesn’t stop at sharing information via his blogs or the community. He joined forces with Darril Gibson & Kenneth van Surksum went the extra mile. The wrote a readable, useful book Mastering Windows 7 Deployment about a subject on which consolidated documentation is scarce, scattered around the internet or written badly so you still can’t figure it out or is to boring you just don’t read it. If I need to define the goal of this book: get people a good head start for Windows 7 deployments in a planned and organized fashion.

This is not a book for the absolute newbie who doesn’t know the difference between a local and a domain account. It isn’t targeted at the WDS/MDT experts who’ve solved, fixed and worked around any and all PXE boot, network errors, cryptic WDS or MDT deployment errors & configuration challenges known to man kind. In that case this stuff is known to you (or should be). The point is those experts have already learnt a lot the hard way and they put in a considerable effort to do so. But knowledge needs to be transferred and spread around and to do that you need to cover the basics and work up from there, showing progress and results. The progress and results motivate people.

In that respect, this books get’s you started on that path from chapter one and by page 5 you’re already being guided into auditing & reporting via MAPS to prepare a roll out proposal. The effort put into discussing the Application Compatibility Toolkit (ACT) is important. I remember the work that we needed to do for Vista x64 bit and how that paid off when deploying Windows 7. What surprises me it that a lot of IT Pro’s don’t even know about the ACT, file and registry virtualization or shims. I recommend another blog on this subject http://blogs.msdn.com/b/cjacks/ , Chris Jackson, the “App Compat Guy” and a very good conference speaker on the subject. The scenarios with the User State Migration Tool will benefit system administrators who dread touching end users their PC and the precious data it might contain. If so, I hope you are backing up the data on those workstations, if not than that is really scary.

Perhaps some readers will already be using certain tools touched upon in the book but not others. In that case this is a great way to start with them and see where they fit in and what they can do for you. We did Vista x64 bit deployments in 2008 with WDS; rolled out Windows 7 x64 in 2010 using WDS/MDT and I still found this book interesting enough to buy some copies and add it to the toolkit of my team. What I’d like to add as a useful hint: look into disable rearming by using <SkipRearm>1</SkipRearm> in the unattended XML file you can pass to sysprep as in “/generalize /quiet /unattend:<file_name.xml” so you don’t run into a when you do it more than 4 times on the same image (An error message occurs when you run "Sysprep /generalize" in Windows Vista or Windows 7: "A fatal error occurred while trying to Sysprep the machine").

The Microsoft Deployment Toolkit (MDT) sections point you directly to some gems we found very useful in our deployments. That you can pre stage computers in the MDT database to help make the roll outs as “light touch” as possible is cool, but that you can automate that with the MDT PowerShell module makes it really very valuable. See http://blogs.technet.com/b/mniehaus/archive/2009/05/15/manipulating-the-microsoft-deployment-toolkit-database-using-powershell.aspx for more details. Michael Niehaus is to MDT what Chris Jason is to ACT. As identifier we use the MAC address as we get that on a label on the PC and we can easily get a list of those to mass import them together with creating the computer objects in Active Directory. We also added driver profiles depending on the client make & model. When you combine this with boot from PXE provided by WDS to boot to an MDT WinPE, and remember WDS also gives you multicast, you have a real sweet solution going. This is the route we went last year and has served us well (we came from a pure WDS solutions, and RIS before that when we still did XP rollouts but that was more than 4 years ago … time flies.

Task sequencer is a gem that we indeed also use to roll out certain default software like 7zip, a pdf reader, ISO burner, anti malware, etc. The fact that these are not in the image makes it very easy to deploy newer versions as they come available.

The chapter on KMS, VAMT, volume licensing will be of use to people who have never dealt with it coming from Windows 2003/XP

This book will come into its own for any SME or enterprise departmental system administrator with who needs to be launched swiftly and on his or her way to their targets, which are smooth Windows 7 deployments. A lot of production system administrators are in the progress of looking at Windows 7 and might have a lot of experience with Windows XP and Windows 2003 but not with Windows 2008(R2) and Vista/Windows 7. If you’re in that bracket you’re definitely going to get a kick start with this book and it contains some neat tips and tricks to get over some initial gotchas. Don’t think that this is for big enterprises only. Apart from the system center products most tools are free downloads or a part of the Windows server license you already own.

As always, the only way to understand technologies is to work with them, use them. That’s the way to gain insight, experience, and context. So play with this stuff in a lab. Run into a bunch issues and fix them. If you need to get up to speed with all this stuff then you should dig into this book with a hands on approach. The book will also help you make more sense of other information out there and you’ll be able to put that into context better. As a bonus, I’m pretty sure that anything you learn from it will help you with deploying Windows vNext as well.

Hyper-V Dynamic Memory does not work on a Windows Server 2008 Standard Edition or Windows Web Server 2008 virtual machine (VM)

Here’s a pointer to a Microsoft Knowledge base article on Hyper-v Dynamic Memory not working on Windows 2008 Standard & Web edition Hotfix: Hyper-V Dynamic Memory does not work on a Windows Server 2008 Standard Edition or Windows Web Server 2008 virtual machine (VM) As you probably already know you have a similar issue with Window 2008 R2 Standard and Web edition virtual machines which is fixed by installing SP1 in the guest. For the predecessor of R2 you need to install this hotfix.

The cause is that on these versions of the operating systems the the required memory enlightenment is supported. After installing the hotfix (or SP1 in the case of Windows 2008 R2) memory addition enlightenment is available on these SKU and your good to go.

Windows 2008 R2 SP1 Publicly Available & Update on Error 0x00f0818

Yesterday, late last night in Europe, Windows Server 2008 R2 SP1 became available to the general public. That means it will be pushed via Windows Updates or that you can download it manually here Windows 7 and Windows Server 2008 R2 Service Pack 1 (KB976932)

I’ve been busy doing deployments in the lab but also in production at several locations. The speed of a deployment depends on the host and what’s running on that host. I’ve seen anything between 22 minutes and 65 minutes. For a walk through of an install see my previous blog post on this Upgrading a Hyper-V R2 Cluster to Windows 2008 R2 SP1

I’ve also noticed an increased amount of hits for my blog on error 0x00f0818 when installing the Windows 2008 R2 SP1 Beta. The same solution holds true for RTM. More info on that is available at Windows 2008 R2 SP1 Beta Install Gone Wrong: Service Pack Installation failed with error code 0x800f0818

I had one place where dozens of VMs had this issue so in order to progress quickly we scripted the replacement of the C:\Windows\servicing\Packages folder content with know good files in bulk. Read the above mentioned article to learn about the security settings you’ll need to address in any automated solution (use takeown F c:\Windows\Servicing\Packages /D y /R & cacls c:\Windows\Servicing\Packages /E /T /C /G “UserName”:F  to take care of business). That saved us the time to check on each affected server individually what packages were involved. This works but test this before using it in production and don’t forget to return the security settings to the default when you’re done. In this case it was a rather large lab environment.