Windows 2003? Let it go!


Reflecting on some of the discussions I was in recently I can only say that there is no escaping reality. Here are some reference blogs for you.

You can’t get of Windows 2003 you say? Held hostage by ancient software from a previous century?  Sure I understand your problems and perils. But we do not negotiate with hostage takers. We get rid of them. Be realistic, do you think this is somehow going to get any better with age? What in 24 months? What about 48? You get the drift. What’s bad now will only be horrible in x amount of time.

Look at some issues people run into already:

Issues like this are not going to go away, new ones will pop up. Are you going to keep everything in your infrastructure frozen in time to try an avoid these? That’s not even coping, that’s suffering.

image

What ever it is that’s blocking you, tomorrow is when you start planning to deal with it and execute on that plan. Don’t be paralyzed by fear or indecision. Over 12 years it will have been a supported OS by its end of life. Windows 2003 had a real good run but now it’s over. Let it go before it hurts you. You have no added value from a more recent version of Windows? Really? We need to talk, seriously.

UPDATE: Inspired by Aidan Finn (@joe_elway) who offered a very good picture to get the message across => click the picture to get the soundtrack! LET IT GO!

Embedded image permalink

Failed at dumping XP in a timely fashion? Reassert yourself by doing better with Windows Server 2003!


I could write a blog post that repeats the things I said bout XP here for Windows 2003 with even some more drama attached so I won’t. There’s plenty about that on the internet and you can always read these blogs again:

I also refer you to a old tweet of mine that got picked up by some one and he kind of agreed:

image

Replace “XP” with “Server 2003” and voila. Instant insight into the situation. You are blocking yourself from moving ahead and it getting worse by the day. All IT systems & solutions rot over time. They become an ever bigger problem to manage and maintain, costing you time, effort, money and lost opportunities due to blocking to progress. There comes a day that creative solutions won’t pop up anymore like the one in this blog post  Windows XP Clients Cannot Execute Logon Scripts against a Windows Server 2012 R2 Domain Controller – Workaround and more recently this on where people just waited to long to move AD over from Windows Server 2003 to something more recent It turns out that weird things can happen when you mix Windows Server 2003 and Windows Server 2012 R2 domain controllers. All situations where not moving ahead out of fear to break stuff actually broke the stuff.

In the environments I manage I look at the technology stack and plan the technologies that will be upgraded in the coming 12 months in the context of what needs to happen to support & sustain initiatives. This has the advantage that the delta between versions & technologies can never become to big. It avoids risk because it doesn’t let delta grow for 10 years an blocks introducing “solutions” that only supports old technology stacks. It make sure you never fall behind too much, pay off existing technology debt in a timely fashion and opens up opportunities & possibilities. That’s why our AD is running Windows Server 2012 R2 and our ADFS was moved to 3.0 already. It’s not because a lot of things have become commodities you should hand ‘m over to the janitor in break/fix mode. Oh the simplicity by which some wander this earth …

OODA

Observe, Orient, Decide, Act. Right now in 2014 we’ve given management and  every product/application owner their marching orders. Move away from any Windows 2008 / R2 server that is still in production. Why? They demand a modern capable infrastructure that can deliver what’s needed to grasp opportunities that exits with current technology. In return they cannot allow apps to block this. It’s as easy and simple as that. And we’ll stick to the 80/20 rule to call it successful and up the effort next year for the remainder. Whether it’s an informal group of dedicated IT staff or a full blown ITIL process that delivers that  doesn’t matter. It’s about the result and if I still see Windows 7 or Windows 2008 R2 being rolled out as a standard I look deeper and often find a slew of Windows 2003 or even Windows 2000 servers, hopefully virtualized by now. But what does this mean? That you’re in a very reactive modus & in a bad place. Courage & plans are what’s needed. Combine this with skills to deal with the fact that no plan ever woks out perfectly. Or as Mike Tyson said “Everybody has a plan until they get punched in the mouth. … Then, like a rat, they stop in fear and freeze.”

Organizations that still run XP and Windows Server 2003 are paralyzed by fear & have frozen even before they got hit. Hiding behind whatever process or methodology they can (or the abuse of it) to avoid failure by doing the absolute minimum for the least possible cost. Somehow they define that as success and it became a mission statement. If you messed up with XP, there’s very little time left to redeem yourself and avoid the same shameful situation with Windows Server 2003. What are you waiting for? Observe, Orient, Decide, Act.

NTFS Permissions On A File Server From Hell Saved By SetACL.exe & SetACL Studio


Most IT people don’t have a warm and fuzzy feeling when NTFS permissions & “ACLing” are being discussed. While you can do great & very functional things with it, in reality when dealing with file servers over time “stuff” happens. Some of it technical, most of it is what I’ll call “real life”. When it comes to file servers, real life, especially in a business environment, has very little respect, let alone consideration for NFTS/ACL best practices. So we all end up dealing with the fall out of this phenomena. If you haven’t I could state you’re not a real sys admin but in reality I’m just envious of your avoidance skills Smile.

You don’t want to fight NTFS/ACLs, but if it can’t be avoided you need the best possible knowledge about how it works and the best possible tools to get the job done (in that order).

If you have not heard of SetACL or DelProf2, you might also not have heard of uberAgent for Splunk, let alone of their creator, community rock star Helge Klein. If you new to the business I’ll forgive you but if you been around for a while you have to get to know these tools. His admin tools, both the free or the paying ones, are rock solid and come in extremely handy in day to day work. When the shit hits the fans they are priceless.

Helge is an extremely knowledgeable, experienced, talented and creative IT Professional and developer. I’ve met him a couple of times (E2EVC, where he’s an appreciated speaker) and all I can say is that on top of all that, he’s a great guy, with heart for the community.

Having the free SetACL.exe available for scripting of NTFS permissions is a luxury I cannot do without anymore. On top of that for a very low price you can buy SetACL Studio. This must be the most efficient GUI tool for managing NFTS permissions / ACLs I have ever come across.

Not long ago I was faced with a MBR to GPT LUN migration on a very large file server. It’s the proverbial file server from hell. We’ve all been there too many times and even after 15 years plus we still cannot get people to listen and follow some best practices and above all the KISS principle. So you end up having to deal with the fall out of every political, organizational, process and technical mistake you can imagine when it comes to ACLs & NTFS permissions. So what did I reach for? SetACL.exe and SetACL Studio, these are my go to tools for this.

image

Check out the web page to read up on what this tool can do for you. It very easy to use, intuitive and fast. It can do ACL on file systems, registry, services, printers and even WMI. It helps you deal with granting ownership and rights without messing up the existing NTFS permissions in an easy way. It works on both local and remote systems. Last but not least it has an undo function, how cool is that?!  Yup and admin tool that let you change your mind. Quite unique.

As an MVP I can get a license for free form Helge Klein but I recommend any IT Pro or consultant to buy this tool as it makes a wonderful addition to anyone’s toolkit, saving countless of hours, perhaps even days. It pays itself back within the 15 minutes you use it.

Other useful tools in your toolkit are http://www.editpadlite.com/ as it can handle the large (550-800 MB) log files RoboCopy can produce and some PowerShell scripting skills to parse these files.

Windows XP Clients Cannot Execute Logon Scripts against a Windows Server 2012 R2 Domain Controller – Workaround


The issue

The real issue is that you are still running Windows XP. The secondary issue is that you have Windows XP clients that cannot connect to a file share (NETLOGON) on a Windows Server 2012 R2 Domain Controller. If you try manually via \\domaincontroller\Netlogon it will throw an error like  "The specified network name is no longer available".  Security wise & moral pressure wise I kind of think this drives home the message you need to get off Windows XP. But I realize you’re in a pickle so here’s the workaround/fix.

Root Cause & Fix

Windows XP talks SMB 1.0 and that’s it. If this is not offered by the server (file server or domain controller) we have a problem. Now if you installed new Windows Server 2012 R2 servers they do not deploy the SMB 1.0 feature by default. If you upgraded from Windows 2008 R2 (perhaps even over Windows 2012) to get to Windows 2008 (R2) this feature kept in place. Other wise you’ll need to make sure SMB 1.0 is installed, it often (always?) is. Just check.

image

However there is a big change between Windows Server 2008 R2/Windows 2012. The LanmanServer service has a dependency set to SMB 2.0 and no longer to SMB 1.0

This is what it looks like on a Windows Server 2012 (or lower) domain controller:

image

This is what it look like on a Windows Server 2012 domain controller

image

So we need to change that on Windows 2012 R2 to support Windows XP. We can do this in the registry. Navigate to

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\DependOnService

  1. Change SamSS Srv2 to SamSS Srvimage
  2. Restart the Server (Lanmanserver) service (it will restart the dependent services like netlogon, DFS Namespace, .. as well)

You’re XP clients should be able to authenticate again. You can test this by navigating to \\domaincontroller\Netlogon on a XP client. This should succeed again.

If you have issues with Windows Server 2012 R2 file servers … this is also valid. When you do get rid of Windows XP. Go back to the original settings please Smile.

If you want to read more on SMB read this blog Windows Server 2012 R2: Which version of the SMB protocol (SMB 1.0, SMB 2.0, SMB 2.1, SMB 3.0 or SMB 3.02) are you using? by Jose Barreto (File Server team at Microsoft)

Finally, get off XP!

I think I said it enough on twitter and my blog Legacy Apps Preventing Your Move From Windows XP to Windows 8.1? Are you worried about HeartBleed? Good! Are you worried about still being on XP? No? Well dump SSL and use clear text authentication as XP is a free fire zone  anyway (as of April 8th 2014) and it’s just a matter of time before you’re road kill. Any company who has CIO/CTO/IT managers and other well paid functions and have let their organization be held hostage on XP (I’m not talking about a few PCs or VMs left and right) by legacy apps & ISV should realize they are the one who let this happen. Your watch. Your responsibility. No excuses.

SSL Certs And Achieving “A” Level Security With Older Windows Versions


So a mate of mine pings me. Says they have an problem with their web mail SSL security  (Exchange 2010) running virtualized on Hyper-V.  The security guy states they need to move to a more secure platform that supports “modern SSL standards” and proposes to migrate from Exchange 2010 to Exchange 2013 in an emergency upgrade. Preferably to VMware as “MickeySoft” is insecure. Oh boy! Another profit of disaster who says the ship is lost unless …

You immediately know that the “security guy” is an incompetent fraud who only reads the IT press tabloids, runs some  freely available vulnerability toys (some are quite good) to determine what to check off on his list and shout out some “the sky is falling” rubbish to justify his daily rate and guarantee his paycheck. I’ve said it before, your mother told you not to trust strangers just like that, so why do so many companies do this with “consultants”? Choose your advisers wisely and remember Machiavelli’s notes on the use of mercenaries Winking smile!

  • VMware is not more secure than Hyper-V. That’s so wrong and so loaded with prejudice it immediately invalidates the persons credibility & reputation. If you need proof, do your research but as a recent example the “HeartBleed” issue left VMware scrambling, not Hyper-V. And for what it’s worth. IT security is like crime, statistically we’ll all be victims a couple of times in our life time.
  • Exchange 2010 running on Windows 2008R2 fully patched is just fine. So what was all the drama about? The issue was that the Qualys SSL Labs tool gave their Outlook Web Access a F grade. Why? Well they still allowed SSL 2.0, they didn’t run TLS 1.2 and they don’t have Forward Secrecy support.

My advice to my buddy? First he needs to get better security advice. Secondly, to get an “A” for secure SSL configuration all you need to is some easy tweaking. You don’t want to support any clients that can’t handle the better SSL configurations anyway. No one should be allowed to use these anyway. But what do I use? SSL 3.0? TLS 1.0/1.1/1.2? What to use & do? Here’s some documentation on how to enable/disable protocols: How to restrict the use of certain cryptographic algorithms and protocols in Schannel.dll. This will tell you how to do it? But which SSL versions can you dump today without suffering to many support calls. Server side, drop SSL 2.0 & SSL 3.0, keep TLS 1.0/1.1/1.2. On the client side you’ll need to do the same. That will keep most things working. Not ideal but the trick is to allow / enable the better protocols server side so all clients that can use it, can use it, while you block the really bad ones that just don’t have any use any more. We’ll play a bit with this.

Test 1: Disable SSL 2.0 and Enable SSL 3.0

image

As you can see this gave them an B grade. We need to enforce the current best TLS 1.2 protocol to get that and we might want to get rid of SSL 3.0 as XP &n IE 6.0 have had there time and that’s over.

Test 2: Enable TLS 1.2

There you go. I hope this helps you out if you need to make sure you environment supports only more modern, stronger protocols.

image

There it is. A- Smile Compliance achieved! Now it would best to disable SSL 2.0/3.0, TLS 1.0/1.1 on the server and forget about any browsers, operating systems and software that can’t handle it. But that’s not that easily done you’ll need Outlook 2013 for RPC over HTTP if you want to enforce TLS 1.2. But as far as the auditors go they are all so happy now and effectively you’re now supporting the more modern clients. Now my buddy can get to an A or A+ rating when they make sure to get Forward secrecy support in the future. I really advise the latter as HeartBleed made it obvious the wide use of this is long overdue.

Some Testing Fun

Grab a laptop, WireShark and a number of twitter clients, cloud storage products and take a peak a what version of SSL/TLS those apps use. Some tests you can do:

MetroTwit uses SSLv3, OneDrive uses TLSv1, Yammer seems to be at TLSv1 as well. Try disabling TSL 1.0 on a client and see how it breaks Outlook  2010 RPC over HTTPS and even OneDrive by the way.

image

What you can get away with depends on the roles of the servers and the level security the clients for that role can handle.

Won’t this break functionality?

As you’ve seen above it can but for what matters on the e-mail server, probably not. If it does you’re in need of some major work on your client infrastructure. But in most cases you’ll be fine, especially with web browsers. But I have a underpaid employee who needs food stamp support so she cannot afford to upgrade her PC from Windows XP! Dude, pay a decent living wage, please. That aside, yes you can turn on better protocol support and block the oldest, most insecure ones on your servers. You call the shots on the use of your businesses infrastructure and you are under no obligation to allow your employees to access your services with obsolete clients. You want to be in the green zone, in the right column with TLS 1.2 if possible, but that’s going to be a challenge for a lot of services.

image

Do as I say, don’t do as I do

The funny thing is that I ran the same test against the web (mainly e-mail) servers of 4 governments levels that are enforcing/promoting the (mandatory) use of security officers in an attempt to get to a more secure web for the benefit of all man kind. Not only does this fail because of such fine examples of security officers but 2/3 don’t seem to take their own medicine. The intentions are good I’m sure but the road to hell is paved with those and while compliancy is not the same a being secure, even this is hard to get to it seems.

Federal Government Department

image

Undisclosed State Government

image

Undisclosed Local Government

image

Medium Sized City (they did well compared to the above braches with more resources)

image

Don’t panic

That’s what it says on the cover of “The Official Hitchhiker’s Guide to the Galaxy Companion”. Get some good advise and if you want or read more about how the rating is done (as of 2014) then please read this SSL Labs: Stricter Security Requirements for 2014 which also provide a link to their SSL Server Rating Guide.

Legacy Apps Preventing Your Move From Windows XP to Windows 8.1?


Are old applications holding you back getting rid of Windows XP? It’s A reason we hear a lot and these apps do exist. But often it’s because the effort to make it work isn’t considered worth the cost. Year after year. So some people today are stuck on a Windows Server 2000/2003 & XP infrastructure. How does that cost compare now to the cost of dealing with the application? Was it worth not moving the application & have an out of date infrastructure holding your ENTIRE company down?

image

While some things can’t be fixed, putting in some effort could have prevented you of being in this mess. Yes it would have cost you a decent penny but nothing compared to where you are at now with your infrastructure “challenges”.

Here’s a little example for you. Over a period of 13 years we’ve moved an old application (using a Borland database engine & ISAPI DLLs in IIS). It ran on Windows Server 2000. It was P2V’d to VMware Server. Over the years the data base swapped from Informix to SQL Server 2000, 2005, 2008, 2008 R2. We upgraded the VM to Windows Server 2003(x86), moved to Hyper-V, upgraded to Windows 2008(x86) & final now put on W2K12R2(x64). So what do you mean you can’t get rid of XP? We’ve moved the client app for that VM to x64 with Vista in 2007.  We were not to let that app block our way to the future and Windows 7(x64) and Windows 8 & 8.1(x64). In 2014 you should be able to move to or you need to reconsider your approach to IT as you have totally painted the organization into a corner. We did not have installers for anything. We extracted registry entries & bits form installed systems and build installers ourselves with the free NSIS installer. We used  Windows SysInternals tools to figure out where the application wrote & read, what permissions where needed and add those to the installer to make sure it did not need local admin rights. It gave the business over a decade to get a grip on application live cycle management & replace the app. They failed twice, and while that’s bad and we do not like it, it was not deadly as they haven’t let the rest of the company suffer for it. Never, ever let your infrastructure get stuck in the past. But wait you say, what you did is not supported. That’s right. That’s one app, that works, and it beats being left with an unsupportable infrastructure blocking progress Winking smile

You might need some help and here’s a great place to start helping yourself The App Compat Guy. Read and view (TechEd presentations) anything Chris Jackson is offering on this subject and you’ll be on your way. Need a helping hand? Here’s a good place to start if your in Belgium: Microsoft Extended Experts Team (MEET). Chances are some of them known some one who knows how to get it done or are the person to talk to.

Windows Server 2012 64TB NTFS Volumes and the Flush Command


As you might very well have read or even tried you can use 64TB volumes in Windows Server 2012 in a supported scenario. You can do more, NTFS is quite capable of this. I created a 300TB LUN once that I could format up to 256TB Smile But as no one can realistically stress test this for real, it’s not supported.

That’s a lot of storage and data. It’s also expensive and incurs some risk … all that data on one volume. Windows 2012 tries to address the cost issue with commodity storage in combination with the excellent resilience of storage space to reduce both cost and risk.

Apart from introducing ReFS they also did some work on NFTS to help with reliability:

  • A new approach for detecting and repairing corruptions in NTFS which optimizes uptime through on line repair and with spot fixing that keeps off line repairs minimized and very short.
  • Using the flush command instead of FUA.

In this post this we’ll focus on the flush command.

Flushing Your Data

No, not that kind of flushing Smile You have always been able to “throw” data away with some very bad practices and unreliable technology, no need for much innovation there.

I’m talking about the fact that NTFS in Windows Server 2012 has switched to the flush command instead of relying on Forced Unit Access (FUA) to increase reliability for SATA disk and performance with SCSI disks. The good news is you don’t lose anything and gain on both fronts. Especially making cheaper SATA disks more reliable is a big one. It allows SATA disks to be used in business/enterprise scenarios and as such helps reduce costs.

What is Forced Unit Access (FUA)?

Well it’s a flag that indicates a given write should go directly to media, writing through a devices write cache. The NTFS Journaling File System uses FUA to guarantee write ordering which is important to maintain its metadata integrity. It was  implemented in the SCSI (T10) specification but not in the original  ATA (T13) specification. This was added in the 2002 version of the ATA specs but FUA has never been guaranteed to implemented on all ATA devices and as such Windows could not rely on it being there with ATA/SATA disks. As a result it was never used by Windows with SATA disks.

That meant that with SATA disks there is a bigger change of corruption due to a power failure or the likes as NTFS was designed to rely on FUA implementation for robust metadata writes.With ever increasing capacity needs an larger SATA disk being needed and used for business purposes something had to be done. So with Windows Server 2012 (and Windows 8) NTFS switched to using a  flush command to the drives write cache instead of using FUA.

The Benefits

  1. The switch to using the flush command for all operations that require write ordering to ensure file system metadata integrity realizes better reliability and robustness when using commodity SATA storage as it reduces possibility of corruption due to power loss
  2. It Improves performance on SCSI devices because it allows the disk to cache data for as long as safely possible instead of having to do write-through using FUA