SMB 3, ODX, Windows Server 2012 R2 & Windows 8.1 perform magic in file sharing for both corporate & branch offices


SMB 3 for Transparent Failover File Shares

SMB 3 gives us lots of goodies and one of them is Transparent Failover which allows us to make file shares continuously available on a cluster. I have talked about this before in Transparent Failover & Node Fault Tolerance With SMB 2.2 Tested (yes, that was with the developer preview bits after BUILD 2011, I was hooked fast and early) and here Continuously Available File Shares Don’t Support Short File Names – "The request is not supported" & “CA failure – Failed to set continuously available property on a new or existing file share as Resume Key filter is not started.”

image

This is an awesome capability to have. This also made me decide to deploy Windows 8 and now 8.1 as the default client OS. The fact that maintenance (it the Resume Key filter that makes this possible) can now happen during day time and patches can be done via Cluster Aware Updating is such a win-win for everyone it’s a no brainer. Just do it. Even better, it’s continuous availability thanks to the Witness service!

When the node running the file share crashes, the clients will experience a somewhat long delay in responsiveness but after 10 seconds the continue where they left off when the role has resumed on the other node. Awesome! Learn more bout this here Continuously Available File Server: Under the Hood and SMB Transparent Failover – making file shares continuously available.

Windows Clients also benefits from ODX

But there is more it’s SMB 3 & ODX that brings us even more goodness. The offloading of read & write to the SAN saving CPU cycles and bandwidth. Especially in the case of branch offices this rocks. SMB 3 clients who copy data between files shares on Windows Server 2012 (R2) that has storage an a ODX capable SAN get the benefit that the transfer request is translated to ODX by the server who gets a token that represents the data. This token is used by Windows to do the copying and is delivered to the storage array who internally does all the heavy lifting and tell the client the job is done. No more reading data form disk, translating it into TCP/IP, moving it across the wire to reassemble them on the other side and write them to disk.

image

To make ODX happen we need a decent SAN that supports this well. A DELL Compellent shines here. Next to that you can’t have any filter drives on the volumes that don’t support offloaded read and write. This means that we need to make sure that features like data deduplication support this but also that 3rd party vendors for anti-virus and backup don’t ruin the party.

image

In the screenshot above you can see that Windows data deduplication supports ODX. And if you run antivirus on the host you have to make sure that the filter driver supports ODX. In our case McAfee Enterprise does. So we’re good. Do make sure to exclude the cluster related folders & subfolders from on access scans and schedules scans.

Do not run DFS Namespace servers on the cluster nodes. The DfsDriver does not support ODX!

image

The solution is easy, run your DFS Namespaces servers separate from your cluster hosts, somewhere else. That’s not a show stopper.

The user experience

What it looks like to a user? Totally normal except for the speed at which the file copies happen.

Here’s me copying an ISO file from a file share on server A to a file share on server B from my Windows 8.1 workstation at the branch office in another city, 65 KM away from our data center and connected via a 200Mbps pipe (MPLS).

image

On average we get about 300 MB/s or 2.4 Gbps, which “over” a 200Mbps WAN is a kind of magic. I assure you that they’re not complaining and get used to this quite (too) fast Winking smile.

The IT Pro experience

Leveraging SMB 3 and ODX means we avoid that people consume tons of bandwidth over the WAN and make copying large data sets a lot faster. On top of that the CPU cycles and bandwidth on the server are conserved for other needs as well. All this while we can failover the cluster nodes without our business users being impacted. Continuous to high availability, speed, less bandwidth & CPU cycles needed. What’s not to like?

Pretty cool huh! These improvements help out a lot and we’ve paid for them via software assurance so why not leverage them? Light up your IT infrastructure and make it shine.

What’s stopping you?

So what are your plans to leverage your software assurance benefits? What’s stopping you? When I asked that I got a couple of answers:

  • I don’t have money for new hardware. Well my SAN is also pré Windows 2012 (DELL Compellent SC40 controllers. I just chose based on my own research not on what VARs like to sell to get maximal kickbacks Winking smile. The servers I used are almost 4 years old but fully up to date DELL PowerEdge R710’s, recuperated from their duty as Hyper-V hosts. These server easily last us 6 years and over time we collected some spare servers for parts or replacement after the support expires. DELL doesn’t take away your access to firmware &drivers like some do and their servers aren’t artificially crippled in feature set.
  • Skills? Study, learn, test! I mean it, no excuse!
  • Bad support from ISV an OEMs for recent Windows versions are holding you back? Buy other brands, vote with your money and do not accept their excuses. You pay them to deliver.

As IT professionals we must and we can deliver. This is only possible as the result of sustained effort & planning. All the labs, testing, studying helps out when I’m designing and deploying solutions. As I take the entire stack into account in designs and we do our due diligence, I know it will work. The fact that being active in the community also helps me know early on what vendors & products have issues and makes that we can avoid the “marchitecture” solutions that don’t deliver when deployed. You can achieve this as well, you just have to make it happen. That’s not too expensive or time consuming, at least a lot less than being stuck after you spent your money.

I’m Not Your (FREE) Personal Assistant


Volunteering in the community

As active community member and MVP I spend a lot of time and effort sharing information and experiences with the community. I also assist colleagues & peers across the globe when they have questions or issues I might be able to help them with. It’s part of sharing and caring. Just like my fellow community members & MVPs I blog, record video’s, web & screen casts, present at conferences & user groups. I hang out for the Ask The Experts moments of opportunity at both local and international. When possible I also attend the ChalkTalks nights like the one that local user group WinTalks organizes where people can bring their questions or problems to discuss.

The impossibility of answering the questions

I share a lot of information, ideas, opinions and experiences. Asking me directly, repeatedly, to give you quick & fast solutions for your current issues, problems and consulting challenges is not the way to go however. For one the complexity of the issues and the situation as exists is often ignored in these question. So it’s impossible to answer them in that fashion.

Also, as is the case with most of us, I’m a very, very busy man. A tremendous amount of knowledge many of my peers and I share is freely available to the community and we absolutely love doing that. If you ask a question on a blog post or contact me I will try and answer if it’s not too much work & is relevant to the blog post. It benefits everyone to see the question and the answer. But for real support you have forums and vendors service desks that are a lot better suited and have dedicated staff or thousands of volunteer eyes. For consulting engagements to solve the complex issues you’re running into you’ll just have to hire the expertise or make me an offer way too good to decline. When hiring expertise, you do get what you pay for if you do it smart. I’m not to blame and will not pay the bill for your previous bad hires, pseudo experts, marketing based decisions that got people into a pickle.

Keeping it real

We all have jobs with lots of work that we need to do to pay the bills. So we cannot be a free support desk, ad interim engineer, consultant or strategic advisor. This means e-mails and DMs with consulting questions or easily searchable questions are ignored unless the problem is personally interesting to me as a learning experience or it’s indeed “the opportunity of a life time”. The latter is highly unlikely.

You need to realize that you need to design your solutions to whatever level of complexity you can handle or afford. Many make this mistake. I understand all the issues around acquiring, building, maintaining, retaining & hiring expertise. Really I do, I do not live under a rock in the wilderness. It’s hard to find expertise and it’s hard to market expertise. So basically we end up with “best practices” & partially mediocrity. For good reason, that’s where you have to be and stay if you’re not willing/capable to pay for expertise. For a lot of commodity solutions that’s how it should be.

If you need better support & consultants than you currently have you should really consider hiring some of my fellow MVPs via their companies but don’t be surprised to be paying anything from € 200/hour and up for proven highly skilled experts for short very specialized assignments. Don’t balk at this, Ever hired MCS? Or a plumber? Right, these people are true consultants, not what passes for them nowadays but what is actually contracting or body shopping. Nothing wrong with temporary augmentation of your labor force, but is not high expertise consulting. Microsoft PFE/MCS aren’t expensive for the value they provide and the time and effort they put in. Next time you need to pay a plumber after a DIY project has gone wrong you’ll realize this.

You don’t have to engage experts. But if you do, you’ll need to bring a big wallet. You need to understand that your unwillingness to pay does not dictated rates, let alone value. Banks, doctors, shops, government … they only accept money and they laugh at me when I tell them I’d like to pay with some ones else’s gratitude.

Some of the people in my network know I have helped many in the past and know that I do this as a service to the community and learning experience. That benefits everyone out there, just like I benefit from them. That’s my choice, in my personal free time. I can assure you that neither those people or I  take this sort of help for granted, let alone demand it.

I can’t fix you being stupid, lazy, cheap or any combination of the above.

  • You’ll have to do your own searching of the internet via Bing or Google for you.
  • You’ll have to read the articles, blog & documentation.
  • You’ll have to analyze your own issues and come up with an plan of action.
  • You need to realize that developing yourself and skillsets is a time consuming, sustained effort. I understand you have other priorities, but that doesn’t mean I have to pick up the slack and put my own aside.
  • You’ll need to face reality. If your business needs something, they’ll need to make sure they are profitable enough to afford it.

Stupidity Is Not A Great Consulting Skill


Most companies don’t get that body shopping and contracting is not consulting. And today way too many consultants don’t match up to the requirements of being one. So basically these people pay way too much for too little. I have no mercy for fake consultants and companies so messed up in their political crap they fail to recognize quality and value. Just read The do’s and don’ts when engaging consultants Part I and The do’s and don’ts when engaging consultants Part II where I’ve given some rules of fist for optimizing the use of consultants. But here’s some advice to fake consultants. At least try to act professional and be nice. Because one on the main rules of long term happiness while staying employed is the fact that people not want to work with a*holes.

  • Don’t download or synchronize hacking tools while at your employer/customer.
  • If you’re an IT Pro or developer  and you cannot figure out how to expand or mount an ISO with the tools on a modern PC/OS you should realize they want to fire you on the spot for gross incompetence.
  • Playing games like a long term frustrated employee with help desk personnel regarding mundane tasks and leveraging policies or rules as excused not to get work done is annoying, immature & frowned upon.
  • Waiting until you get asked why you didn’t get work done to mention your PC got confiscated due to a security incident an the replacement was missing a tool proves you’re a dead weight.
  • Churning is nothing short of fraud. You deliver real work or get out.
  • It’s rather easy to see that the hours you telecommute are not matching your time sheets. It gets noted. At best they’ll pull a Yahoo on you.
  • When you state that optimizing parallelism in SQL Server and using multiple LUNs and data files is waste of time in an environment that’s on the record for being in the top 5 SQL server operations in the country you might perhaps realize you are being an a* hole. Yes it requires some work, the database is not designed to cater to your laziness, but for performance.
  • It’s designed and build at a great value point by skilled people who earn mediocre wages but are very good and love what they do. Do you even realize the level of your arrogance and the fact that you are now on the “do not hire list”?
  • Realize that not all people who’re working as an employee are clueless sheep that only serve to line your pockets. Most of them are good. Many of of them are great, some even excellent to exceptional. Even when not subject matter experts they’ll see through any fake expertise.

Look, when you’re dead, you don’t realize that, so for you there is no drawback, it’s all good. It’s your surroundings that notice it and have to deal with the loss, the pain. It’s the same when you are stupid. It’s the people around you that suffer. Get it?

We Need Your Opinion On This Strategy, Vision, Management Issue …


Could you give us your opinion on this?

Lately people, managers, have asked me to give advice or at least my opinion on how to organize & manage IT. In the broad sense of the term. Infrastructure, software, services, support, on premise, cloud, data protection, security …  “Just think about it a bit”.

That question “Could you give us your opinion on this?” is a hard one for me.  I could say “read my blog”, the non technical posts. But my opinion is often too high level and they don’t they actually want that. They want a solution. And it’s not that I don’t think about it or don’t have an opinion. But I can’t focus on areas out of my expertise, my control and priorities.

Basically I cannot help them. Not because I’m that stupid or the matter is beyond our control. It’s because the way managers and organizations think is getting more and faster obsolete by the day.

The Issue

Our world, both privately and work related, is becoming more and more connected every day. That means there is a tremendous amount of input, leading to an ever continuing increase of permutations of ever more variables that come in to play. In short, complexity is on the rise at an enormous rate and will overwhelm us. Even worse is that this complexity only shows itself after things have gone wrong. That’s bad but, that also means there are probably many more relationships of cause and effect that haven’t even shown themselves yet. That kind of sounds like a time bomb.

How do you deal with this? Not in the way so many are asking for. And I’m not here to tell my managers or customers what they want to hear. I’m in the business of telling them what they need to hear as I deal in results, not services or studies. More often than not they are looking for processes and methodologies to keep central control over planning, execution, operations and change. All this while the rug is literally pulled away under their feet. There’s the problem.

Situations, technologies, solutions, frameworks, processes all have a time limited value that’s becoming shorter. So the idea that you can plan and control for many years ahead is obsolete in many areas in our ecosystem. There are just to many moving parts, that are changing too fast. So how do we manage this? What kind of leadership do you need? Well there is no easy answer.

How do I deal with this?

Personally I deal with this by working, collaborating & cooperating in a network, in “the community”. My insights, knowledge, help and support come from my network. Some of my colleagues, the contractors and consultants we hire are in that network. A lot of colleagues are not. Most managers are not. Why is that? They are stuck in a hierarchal world of centralized command and control that is failing them fast. At best they achieve good results, but very slow and at a very high expense. We can only hope that the results also don’t turn out bad. They want procedures & processes. Predictability & consistency but I deal with complexity in wide area of expertise that cannot readily be put into manuals and documentation. Not in a timely fashion. I’m in a dog fight (insert “Top Gun” theme). The processes & logistics provide the platform. Learn where procedures & methodologies work and where they’ll kill you. The knowledge and the skills we need are a living thing that feeds on a networked collective and are very much in flux.  I’m so much more better skilled and effective at my job through participating my global community than I can be tied into the confines of my current workplace they’d be mad not to leverage that, let alone prevent me from doing so. You can’t do it alone or in isolation.

An example

Yesterday was an extreme example in a busy week. I started work at 05:30 AM yesterday to set up a testing environment for questions I needed answered by a vendor who leverages the community at large. That’s required some extra work in the datacenter that I could have done by a colleague that was there today because I found out in time. I went to the office at 08:30. I worked all day on an important piece of work I mentioned in my network and was alerted to a potential issue. That led to knowledge sharing & testing. Meaning we could prevent that very potential issue and meanwhile we’re both learning. I went home at 18:30, dinner & testing. I was attending an MVP web cast at 20:00 PM till 21:00 PM learning new & better ways to trouble shoot clusters. I got a call at 19:10PM of a mate in Switzerland who’s running into SAN issues and I helped him out with the two most possible causes of this through my experience with SANs and that brand of HP SAN.  We did some more testing & research until 22:00 after which I wrote this blog up.

We don’t get paid for this. This is true mutual beneficial cooperation. We don’t benefit directly and it’s not “our problem” or job goal. But oh boy do we learn and grow together and in such help each other and our employers/customers. It’s a true long term investment that pays of day by day the longer you are active in the community and network. But the thing is, I can’t put that into a process or manual. Any methodology that has to serve centralized command and control structure while dealing with agile subjects is bound to fail. Hence you see agile & scrum being abused to the level it’s just doing stuff without the benefits.

Conclusion

This is just one small and personal example. Management and leadership will have to find ways of nurturing collaboration and cooperation beyond the boundaries of their control. The skillset and knowledge needed are not to be found in a corporate manual or in never ending in house meetings & committees. Knowledge gained has to flow to grow As such it flows both in an out of your organization. You’re delusional if you think you can stop that today and it’s not the same a leaking corporate secrets. Hierarchies & management based on rank and pay grades are going to fail. And if those managers in higher pay grades can’t make the organization thrive in this ever more connected, faster moving world, they might not be worth that pay grade.

I assure you that employees and consultants who live in the networked global community will quickly figure out if an organization can handle this. They will not and should not do their managers job. In fact they are already doing managers areal big favor by working and operating the way they do. They are leading at their level, they are leveraging their networks and getting the job done. They are taking responsibilities, they solve problems creatively and get results. It just doesn’t fit easily in an obsolete model of neatly documented procedures in a centralized command and control structure. They don’t need a manager for that, they need one that will make it possible to thrive in that ultra-connected ever changing fast paced world. Facilitate, stimulate and reward learning and taking responsibilities, not hierarchies. That way all people in your organization will lead or at least contribute to the best of their ability. You’ll need to trust them for that to work. If you don’t trust them, fine, but act upon it. Letting people you don’t trust work for and with you doesn’t work.

How to do this is a managers & leaders challenge. Not mine. I know when I’m out of my depth or when not to engage. The grand visions, the strategic play of a company is their responsibility. Getting results & moving forward will come from your perpetually learning, and engaged workforce, if you don’t mess it up. And yes, that is your responsibility. Cultures are cultivated by definition. So if the culture of the company is to blame for things going south, realize you’re the ones supposed to make it a good one. People don’t leave organizations, they leave managers ;-) And to paraphrase the words of Walt Disney … you’re in a world of hurt if they leave you but stay at their desk and on the pay roll. It’s called mediocrity, which also serves a purpose, providing commodities & cookie template services whilst letting others shine. But if you want to be a thriving, highly skilled, expertise driven center of excellence … it’s going to take lot of hard and sustained work and it’s not a one way street.

Future Proofing Storage Acquisitions Without A Crystal Ball


Dealing with an unknown future without a crystal ball

I’ve said it before and I’ll say it again. Storage Spaces in Windows Server 2012 (R2) is are the first steps of MSFT to really make a difference (or put a dent into) in the storage world. See TechEd 2013 Revelations for Storage Vendors as the Future of Storage lies With Windows 2012 R2 (that was a nice blog by the way to find out what resellers & vendors have no sense of humor & perspective). It’s not just Microsoft who’s doing so. There are many interesting initiatives at smaller companies to to the same. The question is not if these offerings can match the features sets, capabilities and scenario’s of the established storage vendors offerings. The real question is if the established vendors offer enough value for money to maintain themselves in a good enough is good enough world, which in itself is a moving target due to the speed at which technology & business needs evolve. The balance of cost versus value becomes critical for selecting storage. You need it now and you know you’ll run it for 3 to 5 years. Perhaps longer, which is fine if it serves your needs, but you just don’t know. Due to speed of change you can’t invest in a solution that will last you for the long term. You need a good fit now at reasonable cost with some headway for scale up / scale out. The ROI/TCO has to be good within 6 months or a year. If possible get a modular solution. One where you can replace the parts that are the bottle neck without having to to a fork lift upgrade. That allows for smaller, incremental, affordable improvements until you have either morphed into a new system all together over a period of time or have gotten out of the current solution what’s possible and the time has arrived to replace it. Never would I  invest in an expensive, long term, fork lift, ultra scalable solution. Why not. To expensive and as such to high risk. The risk is due to the fact I don’t have one of these:

http://trustbite.co.nz/wp-content/uploads/2010/01/Crystal-Ball.jpg

So storage vendors need to perform a delicate balancing act. It’s about price, value, technology evolution, rapid adoption, diversification, integration, assimilation & licensing models in a good enough is good enough world where the solution needs to deliver from day one.

I for one will be very interested if all storage vendors can deliver enough value to retain the mid market or if they’ll become top feeders only. The push to the cloud, the advancements in data replication & protection in the application and platform layer are shaking up the traditional storage world. Combine that with the fast pace at which SSD & Flash storage are evolving together with Windows Server 2012 that has morphed into a very capable storage platform and the landscape looks very volatile for the years to come. Think about  ever more solutions at the application (Exchange, SQL server) and platform layer (Hyper-V replica) with orchestration on premise and/or in the cloud and the pressure is really on.

So how do you choose a solution in this environment?

Whenever you are buying storage the following will happen. Vendors, resellers & sales people, are going to start pulling at you. Now, some are way better than others at this, some are even down right good at this whole process a proceed very intelligently.

Sometimes it involves FUD, doom & gloom combined with predictions of data loss & corruption by what seem to be prophets of disaster. Good thing is when you buy whatever they are selling that day, they can save you from that. The thing is this changes with the profit margin and kickbacks they are getting. Sometimes you can attribute this to the time limited value of technology, things evolve and todays best is not tomorrows best. But some of them are chasing the proverbial $ so hard they portray themselves as untrustworthy fools.

That’s why I’m not to fond of the real big $ projects. Too much politics & sales. Sure you can have people take care of but you are the only one there to look out for your own interests. To do that all you need to do is your own due diligence and be brave. Look, a lot of SAN resellers have never ever run a SAN, servers, Hyper-V clusters, virtualized SQL Server environments or VDI solutions in your real live production environments for a sustained period of time. You have. You are the one whose needs it’s all about as you will have to live and work with the solution for years to come.  We did this exercise and it was worth while. We got the best value for money looking out for our own interests.

Try this with a reseller or vendor. Ask them about how their hardware VSS providers & snapshot software deals with the intricacies of CSV 2.0 in a Hyper-V cluster. Ask them how it works and tell them you need to references to speak to who are running this in production. Also make sure you find your own references. You can, it’s a big world out there and it’s a fun exercise to watch their reactions Winking smile

As Aidan remarked in his blog on ODX–Not All SANs Are Created Equally

These comparisons reaffirm what you should probably know: don’t trust the whitepapers, brochures, or sales-speak from a manufacturer.  Evidently not all features are created equally.

You really have to do your own due diligence. Some companies can afford the time, expense & personnel to have the shortlisted vendors deliver a system for them to test. Costs & effort rise fast if you need to get a setup that’s comparable to the production environment. You need to device tests that mimic real life scenario’s in storage capacity, IOPS, read/write patterns and make sure you don’t have bottleneck outside of the storage system in the lab.

Even for those that can, this is a hard thing to do. Some vendors also offer labs at their Tech Centers or Solutions Centers where customers or potential customers can try out scenarios. No matter what options you have, you’ll realize that this takes a lot of effort. So what do I do? I always start early. You won’t have all the information, question & answers available with a few hours of browsing the internet & reading some brochures. You’ll also notice that’s there’s always something else to deal with or do, so give your self time, but don’t procrastinate. I did visit the Tech Centers & Solution Centers in Europe of short listed vendors. Next to that I did a lot of reading, asked questions and talked to a lot of people about their view and experiences with storage. Don’t just talk to the vendors or resellers. I talked a lot with people in my network, at conferences and in the community. I even tracked down owners of the shortlisted systems and asked to talk to them. All this was part of my litmus test of the offered storage solutions. While perfection is not of this world there is a significant difference between vendor’s claims and the reality in the field. Our goal was to find the best solution for our needs based on price/value and who’s capabilities & usability & support excellence materialized with the biggest possible majority of customers in the field.

Friendly Advice To Vendors

So while the entire marketing and sales process is important for a vendor I’d like to remind all of them of a simple fact. Delivering what you sell makes for very happy customers who’s simple stories of their experiences with the products will sell it by worth of mouth. Those people can afford to talk about the imperfections & some vNext wishes they have. That’s great as those might be important to you but you’ll be able to see if they are happy with their choice and they’ll tell you why.

The Dilbert Life Series – A Bad Manager’s Priorities


As usual the normal disclaimer applies: don’t take yourself to seriously. Relax Smile

Where great managers can make a serious difference in many ways to both the success of a company and to the personal achievements of their employees the opposite also happens. Many types of managers exist. Dealing with or even controlling them, depending on whether you live above or under them is well documented. The aim of that is to get the best out of the resources and people available. The better the managers, the better this will work out. Perfection is not of this world and you won’t have the best possible manager for every possible position. That’s a given, just like they won’t have the best possible employee or consultant for every job or project. So there is no need to get emotional about it or expect perfection before calling something good. There is however one type, the bad manager, that should not be controlled. They should be dealt with in only one way which is termination. If that’s not possible you need to get as far away from them as possible. Mind you the latter is only an option if you’re a subordinate employee. If, as a boss you run away from bad subordinate manager than you really need to reconsider your career choices.

Me, Myself and I

A bad manager will never choose you over his or her own priorities, nor will they put the organizations needs first. The first is by definition. Don’t take it personal. The company does not exist for your needs. The second is more problematic as the organization’s needs are priority one. Let’s take a look their priorities in order of declining importance as determined by experience.

  1.  Me, myself and I. This is normal and it applies to everyone. But there is more to this than just plain self-interest. People who are given or rise to power, have a strong tendency to put their own needs and interests above those of others. Your manager’s ambitions & agenda (professional, personal and financial) will always take precedence of any need you might have. They need to fill their treasury and the pressure to “live up to expectations” of their overlords is on.
  2. Reputation. Managers need to be seen & act as very reliable, trustworthy persons who can get results. With some luck they are. But we all know about “perception is reality”. This is true until you hit the ground after jumping of the 36 floor because you pretend you can fly. Whether a bad manager actually delivers anything is irrelevant as long as the perception is there. Office politics are part of the game and they don’t take prisoners. Your boss is going to be more prone to protect his or her reputation than to protect yours. That’s why managers get pissed off about even only a perceived loss of reputation. In the dog eat dog world they’ll even ruin your reputation if and when needed as they can’t be seen as the root cause of problems. They’ll blatantly steal your work and take credit for all that goes well in the same way. You’re an expendable asset and you should never forget it.
  3. Their superiors. This is both hierarchical and functional. It’s not only the fact that a lot of people feel the need to please others for whatever reason. It is also just self-interest (promotions, ego) and self-preservation. So realize that your managers will almost always choose to follow their bosses or the peers they fear or need in order to gain a stronger or more favorable position with them. Yes, they will do so even if it is bad for the company or organization. This holds a warning: if you’re a functional superior to your managers than you’re a threat and they might try to get rid of you.
  4. Customers. You can forget about being more important than the needs of the customers. Whether these are external or internal customers is irrelevant. Your managers job is to serve the need of the customers. Your managers will not get ahead if he doesn’t serve their needs.
  5. The team. Yes the team, the assets are more important than you. As long as managers can have the team do what needs to be done, they have a way of serving the above priorities, which are more important. In that respect the ability of a manager to keep the team running is paramount. They’ll feed the teams just enough to keep them alive, hopeful enough to carry on and will challenges them to keep them sharp. Keep ‘m mean, lean & hungry.
  6. You. Sure you have some skills they needs. If not they might keep you around just to add another FTE to the head count in order to proof the importance or the weight of their jobs. So he won’t kick you most of the time and will even throw you a bone every now and then. Good doggy. But you know that saying “People are our biggest asset?” It’s a lie, especially to them.

How to deal with this?

The above is always true in a lesser way for all individuals and as such also for managers. The big difference is that the balance has totally shifted to the dark side with really bad ones. In essence you have a couple of options. Grow a pair of balls and make sure you have some power as well, play the same game and get them terminated. If your upper management is worth their pay they might be way ahead of you and that will bet the end of it for you. If it has to come from the bottom realize that this is not easy. Terminating a manager from lower in the hierarchy always upsets the powers that be. To them such an event is highly disconcerting and visions of guillotines, tar, feathers and pitch forks pop up. Another option is to take evasive maneuvers. You could do so by moving laterally or vertically in the organization out of harm’s way. Last but not least. Leave. Yes, that might not be fair on you and what you already accomplished at the company but life is not fair and is certainly too precious to put up with the above. In the end you must know your opponent and know yourself. Perhaps you can live with them and there are various ways of dealing with various types of managers, who all have their weaknesses and strengths. It’s a personal decision, but a real bad manager, that’s something you really can do without and shouldn’t tolerate ever, for your own health and well-being.

Key Take Aways From MMS2013


Introduction

I’ve parked myself at McCarran International Airport in Las Vegas awaiting the start of my long haul home to Europe. The new terminal is inspiring me to share as I reflect on the past week and on what’s happening at work.

ICT in the 21st Century

A lot is going on and moving in ICT and even more is coming our way. In the Microsoft sphere we got the official heads up at MMS2013 that new features would be publicly discussed at TechEd 2013 (USA or Europe). So you might want to attend that one. I for one think that’s great. We need that information to verify we’re still are on the right track and fine tune our course. Especially in those areas where we can get quick wins with sometimes significant cost savings & benefits.  I could start telling you about all the great sessions and products at MMS2013 whilst quacking like a duck “cloud, cloud, …, cloud, cloud, cloud, … cloud”. But I will not. You can watch it all here.  I will reflect on the key take away.

Cheaper & Faster

Cheaper AND faster are the new mantra or’ “fast is the new cheap”. Cheaper makes everyone happy especially when quality remains high. Faster is sometimes a bit more of a challenge to sell. “New features, already?”  you say. Yes. The nature of our economies and industry is being transformed by the cloud and commoditization. It brings a lot of benefits, especially in a high speed, low drag world.

Fast is actually faster. For many years now any strategy & execution plan that took more than a couple of years was doomed. You get bypassed and your big investments will never live up to their potential. So, apart from the necessary larger and more long term investments, we evolve more and more towards a perpetual improvements & rapid adoption model. Innovation and the subsequent commoditization of it is pushing this. That’s not bad. By making constant smaller (easier to fund) investments that deliver fast results we get to a more adaptable, agile environment for lesser costs. It’s not that all long term, large scale projects are going away but the ratio is shifting. In smart countries this is already being done for building hospitals and other infrastructure that evolves fast. It’s not unique to ICT. Massive projects taking too long and too much funding lead to out of date solutions at the time of delivery at huge costs. Use this approach where needed but forget about it for the other projects. Cloud will be an important tool in all this, not the goal.

A Word of Warning

Fast and cheap shouldn’t translate into mediocre crap at dump pricing that will bite us. It should also keep in mind the ecosystem and don’t act like a shock & awe offensive leaving everything in it’s track in disarray. It needs to fit into a plan with clear goals an knowing where it fits in and helps.  It’s about balance. That’s the art. Knowing what, where, when and with/for who to do it. Not easy. Now let’s hope some of my managers read this blog. It might help them. As the question beckons an answer: who is it that will lead us in this new era? Well not one single person, far from it. It’s a team effort and to lead a team takes competence and some character.

It takes competence and personality

Competence and personality, combined with  applying both these (skills and  drive) diligently in a sustained fashion. That requires a lot of effort, even when no one is watching you, or perhaps better stated, especially then. Do what needs to be done where and when needed. Not because it could get you promoted or more money. That’s the character part. That’s what drives us to learn by participating in our ICT communities, presenting, attending conferences and networking. But also in those hours spend reading, studying and working in the lab alone or with a buddy. That’s what will make us able to handle the though and bad situations you’ll encounter and overcome them. It’s your resourcefulness that will make you seek and find opportunity in adverse conditions. People like the team members amongst whom I have the distinct pleasure of working. You can’t find such synergy if it’s only about personal gain and getting ahead. There is both a broad and deep skill set needed by all involved and doesn’t come easy nor can it be bought. It has to be acquired through work and experience. The transformation of the ICT landscape is uncharted domain for all but a few of us so it’s going to ask a lot of effort, often outside of our comfort zone.

Sure there are cynics who laugh at this and can’t imagine why someone would do all that without personal and immediate reward. Those are the ones we don’t need and who won’t be there at crunch time. Only after the facts they seek the spotlight to poach the glory if things went well or to condemn those that failed whilst trying. Well, the last so called leader who did that doesn’t work with us anymore. Enough said.

A reality Check On Disaster Recovery & Business Continuity


Introduction

Another blog post in “The Dilbert Life Series®” for those who are not taking everything personal. Every time business types start talking about business continuity, for some reason, call it experience or cynicism, my bull shit & assumption sensors go into high alert mode. They tend to spend a certain (sometimes considerable) amount of money on connectivity, storage, CPUs at a remote site, 2000 pages of documentation and think that covers just about anything they’ll need. They’ll then ask you when the automatic or 5 minute failover to the secondary site will be up and running. That’s when the time has come to subdue all those inflated expectations and reduce the expectation gap between business and IT as much as possible. It should never have come to that in the first place. But in this matter business people & analysts alike, often read (or are fed) some marchitecture docs with a bunch of sales brochures which make it al sound very easy and quickly accomplished. They sometimes think that the good old IT department is saying “no” again just because they are negative people who aren’t team players and lack the necessary “can do attitude” in world where their technology castle is falling down. Well, sorry to bust the bubble, but that’s not it. The world isn’t quite that black and white. You see the techies have to make it work and they’re the ones who have to deal with the real. Combine the above with a weak and rather incompetent IT manager bending over to the business (i.e. promising them heaven on earth) to stay in there good grace and it becomes a certainty they’re going to get a rude awakening. Not that the realities are all that bad. Far from it, but the expectations can be so high and unrealistic that disappointment is unavoidable.

The typical flow of things

The business is under pressure from peers, top management, government & regulators to pay attention to disaster recovery. This, inevitably leads to an interest in business continuity. Why, well we’re in a 24/7 economy and your consumer right to buy a new coffee table on line at 03:00 AM on a Sunday night is worth some effort.  So if we can do it for furniture we should certainly have it for more critical services. The business will hear about possible (technology) solutions and would like to see them implemented. Why wouldn’t they? It all sounds effective and logical. So why aren’t we all running of and doing it? Is it because IT is a bunch of lazy geeks playing FPS games online rather than working for their mythically high salaries? How hard can it be? It’s all over the press that IT is a commodity, easy, fast, dynamic and consumer driven so “we” the consumers want our business continuity now! But hey it costs money, time, a considerable and sustained effort and we have to deal with the less than optimal legacy applications (90% of what you’re running right now).

Realities & 24/7 standby personnel

The acronyms & buzz words the business comes up with after attending some tech briefing by Vendors Y & Z (those are a bit like infomercials but without the limited value those might have Sarcastic smile) can be quite entertaining. You could say these people at least pay attention to the consumerized business types. Well actually they don’t, but they do smell money and lots of it. Technically they are not lying. In a perfect world things might work like that … sort of, some times and maybe even when you need it. But it will really work well and reliable. Sure that’s not the vendors fault. He can’t help  that the cool “jump of a cliff” boots he sold you got you killed. Yes they are designed to jump of a cliff but anything above 1 meter without other precautions and technologies might cause bodily harm or even death. But gravity and its effects in combination with the complexity of your businesses are beyond the scope of their product solutions and are entirely your responsibility. Will you be able to cover all those aspects?

Also don’t forget the people factor. Do you have the right people & skill sets at your disposal 24/7 for that time when disaster strikes? Remember that could be on a hot summer night in a weekend when they are enjoying a few glasses of wine at a BBQ party and not at 10:15 AM on a Tuesday morning.

So what terminology flies around?

They hear about asynchronous or even synchronous replication of storage of applications. Sure it can work within a data center, depending on how well it is designed and setup. It can even work between data centers, especially for applications like Exchange 2010. But let’s face it, the technical limitations and the lack of support for this in many of the legacy applications will hinder this considerably.

They hear of things like stretched clusters and synchronous storage replication. Sure they’ll sell you all kinds of licensed features to make this works at the storage level with a lot of small print. Sometimes even at the cost of losing functionality that makes the storage interesting in the first place. At the network level anything below layer 3 probably suffers from too much optimism. Sure stretched subnets seem nice but … how reliable are these solutions in real live?

Consider the latency and less reliable connectivity.You can and will lose the link once in a while. With active-active or active-passive data centers that depend on each other both become single points of failure. And then there are all the scenarios where only one part of the entire technology stack that makes everything work fails. What if the application clustering survives but not the network, the storage or the database? You’re toast any way. Even worse, what if you get into a split brain scenario and have two sides writing data. Recover from that one my friend, there’s no merge process for that, only data recovery. What about live migration or live motion (state, storage, shared nothing) across data centers to avoid an impending disaster? That’s a pipe dream at the moment people. How long can you afford for this to take even if your link is 99.999% reliable? Chances are that in a crisis things need to happen vast to avoid disaster and guess what even in the same data center, during normal routine operations, we’re leveraging <1ms latency 10Gbps pipes for this. Are we going to get solutions that are affordable and robust? Yes, and I think the hypervisor vendors will help push the entire industry forward when I see what is happening in that space but we’re not in Walhalla yet.

Our client server application has high availability capabilities

There are those “robust and highly available application architectures” (ahum) that only hold true if nothing ever goes wrong or happens to the rest of the universe. “Disasters” such as the server hosting the license dongle that is rebooted for patching. Or, heaven forbid, your TCP/IP connection dropped some packages due to high volume traffic. No we can’t do QoS on the individual application level and even if we could it wouldn’t help. If your line of business software can’t handle a WAN link without serious performance impact or errors due to a dropped packet, it was probably written and tested on  <1ms latency networks against a database with only one active connection. It wasn’t designed, it was merely written. It’s not because software runs on an OS that can be made highly available and uses a database that can be clustered that this application has any high availability, let alone business continuity capabilities. Why would that application be happy switching over to another link. A link that is possibly further away and running on less resources and quite possibly against less capable storage? For your apps to works acceptably in such scenarios you would already have to redesign them.

You must also realize that a lot of acquired and home written software has IP addresses in configuration files instead of DNS names. Some even have IP addresses in code.  Some abuse local host files to deal with hard coded DNS names … There are tons of very bad practices out there running in production. And you want business continuity for that? Not just disaster recovery  to be clear but business continuity, preferably without dropping one beat. Done any real software and infrastructure engineering in your life time have you? Keeping a business running often looks like a a MacGyver series. Lots creativity, ingenuity, super glue, wire, duct tape and Swiss army knife or multi tool. This is still true today, it doesn’t sound cool to admit to it, but it needs to be said.

We can make this work with the right methodologies and strict processes

Next time you think that, go to the top floor and jump of, adhering to the flight methodologies and strict processes that rule aerodynamics. After the loud thud due to you hitting the deck, you’ll be nothing more than a pool of human waste. You cannot fly. On top of unrealistic scenarios things change so fast that documentation and procedures are very often out of date as soon as they are written.

Next time some “consultants” drop in selling you products & processes with fancy acronyms proclaiming rigorous adherence to these will safe the day consider the following. They make a bold assumption given the fact they don’t know even 10% of the apps and processes in your company. Even bolder because they ignore the fact that what they discover in interviews often barely scratches the surface. People can only tell you what they actually know or dare tell you. On top of that any discovery they do with tools is rather incomplete. If the job consist of merely pushing processes and methodologies around without reality checks you could be in for a big surprise. You need the holistic approach here, otherwise it’s make believe. It’s a bit like paratrooper training for night drops over enemy strong holds, to attack those and bring ‘m down. Only the training is done in a heated class room during meetings and on a computer. They do not ever put on all their gear, let alone jump out of an aircraft in the dead of night, regroup, hump all that gear to the rally points and engage the enemy in a training exercise. Well people, you’ll never be able to pull of business continuity in real life either if you don’t design and test properly and keep doing that. It’s fantasy land. Even in the best of circumstances no plan survives it first contact with the enemy and basically you would be doing the equivalent of a trooper firing his rifle for the very first time at night during a real engagement. That’s assuming you didn’t break your neck during the drop, got lost and managed to load the darn thing in the first place.

You’re a pain in the proverbial ass to work with

Am I being to negative? No, I’m being realistic. I know reality is a very unwelcome guest in fantasy land as it tends to disturb the feel good factor. Those pesky details are not just silly technological “manual labor” issues people. They’ll kill your shiny plans, waste tremendous amounts of money and time.

We can have mission critical applications protected and provide both disaster recovery and business continuity. For that the entire solution stack need to be designed for this. While possible, this makes things expensive and often only a dream for custom written and a lot of the shelf software. If you need business continuity, the applications need to be designed and written for it. If not, all the money and creativity in the world cannot guarantee you anything. In fact they are even at best ugly and very expensive hacks to cheap and not highly available software that poses as “mission critical”.

Conclusion

Seriously people, business continuity can be a very costly and complex subject. You’ll need to think this through. When making assumptions realize that you cannot go forward without confirming them. We operate by the mantra “assumptions are the mother of al fuckups” which is nothing more than the age old “Trust but verify” in action. There are many things you can do for disaster recovery and business continuity. Do them with insight, know what you are getting into and maybe forget about doing it without one second of interruption for your entire business.

Let’s say disaster strikes and the primary data center is destroyed. If you can restart and get running again with only a limited amount of work and productivity lost, you’re doing very well. Being down for only a couple of hours or days or even a week, will make you one of the top performers. Really! Try to get there first before thinking about continuous availability via disaster avoidance and automatic autonomous failovers.

One approach to achieve this is what I call “Pandora’s Box”. If a company wants to have business continuity for its entire stack of operations you’ll have to leave that box closed and replicate it entirely to another site. When you’re hit with a major long lasting disaster you eat the down time and loss of a certain delta, fire up the entire box in another location. That way you can avoid trying to micro manage it’s content. You’ll fail at that anyway. For short term disasters you have to eat the downtime. Deciding when to fail over is a hard decision. Also don’t forget about the process in reverse order. That’s another part of the ball game.

It’s sad to see that more money is spend consulting & advisers daydreaming than on realistic planning and mitigation. If you want to know why this is allowed to happen there’s always my series on The do’s and don’ts when engaging consultants Part I and Part II. FYI, the last guru I saw brought into a shop was “convinced” he could open Pandora’s Box and remain in control. He has left the building by now and it wasn’t a pretty sight, but that’s another story.

Yahoo’s “Physically Together” is Management Failure


I’m awaiting boarding at SEATAC and browsing the news. I suggest you read “Physically Together”: Here’s the Internal Yahoo No-Work-From-Home Memo for Remote Workers and Maybe More and consider the quote below.

“… Speed and quality are often sacrificed when we work from home  …”

If I was working for Yahoo I’d be jumping the ship. That mentality just doesn’t compute. If anything I have seen the working conditions become worse and worse in offices over the past decade. All the new open/flex work office plans with the continuous interrupts, office chit chat & gossip, noise and countless never ending meetings (I guess partially to escape the lousy desk experience) are probably very good for the bottom line but all the rest of it seems to be working out a lot less well.

Granted, part of that is because of bad execution. It works if you can and will adopt that culture. But more often than not they just transplant the old ways into the new office environment with disastrous results. But the savings are there; so they don’t really mind. Just like they don’t mind outsourcing or consultants. Those don’t come into the office either but they do help reduce head count and CAPEX, whatever helps the Excel sheet look better. Speed and quality can often suffer as well in these cases but then the response is to have better governance and processes, not to drag them all into the landscape office meadow.

And as far as speed and quality … I’ll be crystal clear, I’m not buying that for one second. If I had not been responding to alerts (we have no on call) on weekends the company would no longer exist. It would have lost it’s entire infrastructure a couple of times with little or no hope of recovery. If they force me to be at the office between 08:30 and 17:30 every day they would not get that commitment and I would work a lot less hours. The same goes for my team. We expect a lot and we give a lot. Checks and balances. How are you supposed to build a top notch team on mediocre management practices is beyond me. We put in the effort because that’s what we give back to our employers in return for a lot of flexibility and freedom on how we organize ourselves and the team.

Some middle management that wants hot bodies in the seats to respond to every question they have is very worrying to me and those people have no sense of real priorities. Perhaps of self importance, yes, but not priorities Look organize yourself any way you need to to deliver what ever it is but the above quote executed across the board is sad in it’s simplification and denial of realities.

But go ahead. Sacrifice your agility and flexibility to be able to keep operations going during snow storms, flu pandemics and go on wasting time and resources commuting during peak traffic hours. The trick to making all of this work is to make it part of the normal way of working. The ratio of type of flex and telework might change during such times but that’s it. Any organization who cannot see this, act on it and leverage the new possibilities technology offers us is a victim of management failure. These across the board decisions are a clear sign of that and make me list Yahoo on the “Unsuitable Employers” list. Their speed and quality may very well suffer from this decision.

Are you perhaps saying your employees are goofing of at home and are under performing? Well if physical presence is the only way to make sure they are doing a good job you’re really in trouble. You have many other and more serious problems I think and good luck to you if you think pulling then back into the office will fix this. Probably this is really the issue. They’ve lost insight in who does what and why. End states are not defined, lack of accountability, … or otherwise put: management failure.

Or are you a serious professional who can’t stand the idea of your senior engineer sitting in his pajamas writing code or building a cluster at 10:00 or 22:00 hours? You think he needs to be in khakis and shirt? If it’s the pajama image you could consider hiring super models as engineers, the idea will become a lot more pleasant,  I guarantee it Winking smile. Or are you worried about the odd working hours and the impact on the well being of your employees? Changes are they’ll do that anyway or even more when having to be in the office. They can’t get the real work done when having to sit in that sub optimal cube all day and dealing with all the senseless interrupts.

What if people don’t flee you because of this policy but just zone out. They show up for whatever mandatory time they need to. When shuffled like cattle into their cubicles and or pastures (open landscape offices) they’ll put on their noise cancellation headsets, run of to meetings (anything to escape the chaos and interrupt hell the modern office environment has become. Their talent, engagement, motivation and zeal will go to what they love to do and those organizations will end up as mediocre players putting in the bear minimum. Well played. Look, today we’re expected to be able to work from anywhere at any time and indeed technology has enabled this for a significant amount of people. A lot of us do that and we’re very flexible about it as we commit to our jobs and working lives in ever more flexible ways. Now on top of that they expect us to show up on the clock and proof attendance in a rather than creating a win-win situation?

On top of that they do this in a time where managers claim that talent will flee companies that do not allow BYOD or other consumer IT.  Really, but having old school office organizations wont? Flexibility works both ways. Employees can be very efficient and committed. But any manager looking to extract every last ounce of profit or plays power games because they can’t deal with end state management will loose more then they will ever gain. A BYOD device policy cannot attract and retain the best of the best. Trust me, those fine employees will figure out very fast that they’ll choose flex time, telecommuting, better pay and extra paid holidays over that stupid iPad or iPhone. Consumerization of ICT means they don’t need your technology and devices. They’ll buy their own and use it for their own advancement and interest and you’ll be left in your holding the short end of the stick. You shouldn’t care that your  employees make you money while stepping on a cross trainer at home or even from their bath tub.

I really don’t buy into the fact that this is all complicating the creation of products or the delivery of services. It also doesn’t ruin any long term supportability. People will go where they think they are best off.  So what is this move? A need to reduce head count and trying to achieve this by people calling it quit voluntarily? So basically you’re even unable to fix performance issues with your feedback/planning and evaluation system? Oh boy. So what if your best quit and the worst show up at the office? Yahoo’s in a pretty bad state it seems.

Is it a power play and about limiting options for people to see how obedient they are? If all the “our employees are our biggest and most important resource” is true some things would be really different. For one your employees would tell you to stop considering and treating them a resource to move around at will. After all this is not an national crisis and this is not the military at war. In a real war for talent employees would interview you whether to see if you’re even worth working for. Most companies don’t like the power to shift to the employees to far. They have seen this for short periods of time in certain professions and they still haven’t recovered from that shock to their system. They’d rather have less of it, not more. It’s all way to complicated for them to handle and manage. It also costs them more.

Are Data Tsunamis Inevitable Or Man Made Disasters?


What happens when people who have no real knowledge and context about how to handle data, infrastructure or applications insist on being in charge and need to be seen as taking strong decisive actions without ever being held responsible? It leads to real bad, often silly decisions with a bunch of unintended consequences. Storage vendors love this. More iron to sell. And yes, all this is predictable. When I’m able and allowed to poke around in storage and the data stored I often come to the following conclusion: there’s a bulk amount of data that is stored in an economical unsound fashion. Storage vendors & software vendors love this, as there are now data life cycle management tools & appliances to be sold.

The backlash of all this is? Cost cutting, which then leads to the data that has valid needs to be stored and protected not getting the resources it should. Why? Well who’s going to take responsibility to push the delete button to remove the other data? As we get ever better technology to store, transport and protect data we manage to do more with less money and personnel. But as is often the case, no good deed goes unpunished. Way to often these savings or efficiencies flow straight into the bottomless pit caused by that age old “horror vacui” principle in action in the world of data storage.

You get situations like this: “Can I have 60TB of storage?  It’s okay, I discussed this with your colleague last year, he said you’d have 60TB available at this time frame”

What is the use case? How do you need it? What applications or services will consume this storage? Do you really need this to be on a SAN or can we dump this in cost effective Windows Server Storage Spaces with ReFS? What are the economics involved around this data? Is it worth doing? What projects is this assigned to? Who’s the PM? Where is the functional analysis. Will this work? Has there been a POC? Was that POC sound? Was there a pilot? What the RTO? The RPO? Does it need to be replicated off site? What IOPS is required? How will it be accessed? What security is needed? Any encryption required? Any laws affecting the above? All you get is a lot of vacant blank stares and lot’s of “just get it done”. How can it be that with so many analysts and managers of all sorts running around to meeting after meeting, all in order to get companies running like a well oiled slick mean machine, we end up with this question at the desk of an operational systems administrator as a result? Basically what are you asking for? Why are you asking this and did you think this through?

waterjugs

Consider the following. What if you asked for 30 billion gallons of water at our desk and we say “sure” and just sent it to you. We did what you asked. Perhaps you meant bottled drinking water but below is what you’ll end up with. And yes it completely up to specifications, limited as they are.

vlcsnap-2013-01-12-10h49m42s238

The last words heard while drowning will be “Who ordered this? You can bet no one will be responsible, especially not when the bill arrives and when the resulting mess needs to be cleaned up. Data in the cloud will not solve this. Like the hosting business, who serve up massive amount of idle servers, the cloud will host massive amounts of idle data as in both situations it’s providing the service that generates revenue, not the real use of that service by you or it’s economic value to you.