Hiring practices – The Dilbert Life


What’s going on?

For the last 2 years I’ve been in a number of situations where management was pushing to hire extra FTE’s. The numbers on the personnel lists had to be met whether it made sense or not I guess. Sometimes they wanted to give someone a raise or do them a favor with a better contract.

Furthermore in their effort to reach the numbers there was pressure to drop our standards, to lower the bar so to speak. Well, not on my watch. My standards are in place for a reason. To make sure that the people on the job can actually handle that job under all circumstances. Look, when the shit hits the fan and things get tough you’ll need every bit of team spirit and camaraderie that you can get to make sure the team can withstand the stress and get the job done. If you put people in the teams that don’t cut it you’ll already have dysfunctional unit in “normal” times, what do you think will happen when things get tough? They’ll just cut the rope and let the dead weight fall.

People, those standards, the difficulty & complexity in selection, the so called “bar” are there for a reason. If you cannot meet those demands it is not those standards that are the problem. The only real issue is that you’re not capable to meet those standards. So stop complaining about them. Instead start working on yourself so you can meet them. If you can’t do that you’ll need to find job satisfaction somewhere else.

But why this pressure to hire people even if they don’t seize up? Why this obsession with more and more personnel?

Filling the numbers

On paper all is well, you’ve got plenty of warm bodies to fill the cubicles and to meet productivity demands. You can’t produce a baby in one month with 9 women, it doesn’t work that way. Perhaps methodologies are putting so much pressure on having all forms and numbers right that achieving just that is priority one. You might have the best functional team in the business but if those numbers don’t match up to the methodology documents that doesn’t matter. You get no bonus but lots of hassle. So what’s a poor weak manager who wants to climb the ladder to do? Sell out the team to comply. Go ahead. Kiss your team’s respect goodbye. You might as well have flushed their motivation down the toilet.

Other times people simply can’t manage. So when they have underperforming team members the easy way out is getting extra FTE’s. That’s way easier than dealing with the problem. So bloat your FTE needs to the needed numbers + the number of miss hires already in place. And then there is the danger of hiring good people, they are a treat to the peaceful little world management is in. Remember, “A people” hire “A people”, “B people” hire “B people” … and it goes downhill from there.

Sometimes you might need to have x amount of people to legitimize your management position so … bring on the FTE’s :-)

There are lots of reasons why this happens. Managers, especially middle management often suck at management. It is really a very hard, demanding and crucial job. Management at least gets the perks. Middle management is caught between a rock and a hard place. But the dangling carrot and their ambition keeps them from walking out and the keep choosing the easy way to deal with. Perhaps they don’t even have a choice. That’s the best they can do.

Giving a raise / doing a favor

This is bad, very bad. You are inflicting real damage here. This is the equivalent of making me an NCO in the airborne rangers because I love to shoot guns and like to brag in the bar with a ranger patch on my sleeve. It’s pure madness. I’m overweight, I’m as myopic as a mole and I have several medical issues that would put me (and as a consequence my mates) in danger, even on an exercise. What do you think will happen? Everybody will lose and will be unhappy in this situation. Me, because I know I’m fooling myself, I know I’m a burden to my colleagues and endangers myself and them. I will never be accepted let alone be respected. Nope they’ll come to despise me. They have their job to do, plus now they need to do mine, correct my failures, drag me along and rescue me. So basically they’ll resent the hell out of me. They’ll be pissed off at the officers who passed me because they have devalued their skills and show no professional recognition of what they are capable of. The officers won’t be happy because they’ll have to command and work with a dysfunctional unit. It is madness. Is cannot work, it will not work.

If this metaphor doesn’t get the message across than perhaps this one will: imagine your brain surgeon was given the position because nagged long enough to hospital management for a raise

Conclusion

Managers, don’t lower the bar to fill the numbers, to give people a raise or do them a favor. You’ll just create frustration and dysfunctional teams. They won’t function as well as they could and when they need to stand tall to face serious issues you don’t have the number you fool yourself to have on paper. At best you have your good people who’ll  need to work around the bad ones. In the worst case the team is so dysfunctional they won’t get the job done as well and fast as they should or they’ll fail all together. But in reality management is the real failure. They didn’t do their job. Unit operational efficiency might not be on your radar screen but it really should be.

The World Could Be Like Star Trek the Google Way


If Google gets it way the world might become a bit more like Star Trek. For the past 9 years  .NET made a truck load of languages understand each other in the developer world.  Now this bliss could become available for the entire world population. According to this article http://technology.timesonline.co.uk/tol/news/tech_and_web/personal_tech/article7017831.ece

we’ll be able to talk to Chinese, Indian, Russian, etc … colleagues and acquaintances over the phone in our native language and they’ll hear it in their own. If this works well (can’t be easy, far form) it will rock! I can already imagine myself at a conference talking face to face over a phone with a Brazilian IT Pro!

Excuse me now cause I’m of to the Holodeck with a couple of super models to teach them about Hyper-V and how to set up Live Migration. When their looks start to go, they’ll be able to earn a living in IT. These ladies are some smart cookies investing wisely in their future ;-).

The do’s and don’ts when engaging consultants Part II


In part II we take a look at the common mistakes when using consultants. This is all common sense and well documented in literature and on line. So why does it go wrong so often? Politics and just not caring (I just work here dude) often in combination with “fake it until you make it”. Follow this link for Part I

Common Mistakes

Buying advice to settle a score

Now please all acts as grownups! Don’t settle your disagreements this way. One side will feel defeated, the other victorious with a vengeance. Great start for a project that depends on collaboration<\SARCASM>. What if the result of such advice is very mixed with lots of pros and cons on both sides? Then what? You’re doomed. No agreement, no common goals, no purpose or will to cooperate. You are now riding a dead horse. Guess what? You are not going to get anywhere :-) What your team really needs people, is a coach!

Bait & Switch

So the guru walks in, explains with great passion and knowledge the solution and how to achieve it. You make the deal. You are very happy to have such a highly skilled operator to take care of your interests. But soon after the project starts the guru is nowhere to be found. Some pimple faced youths and a washed up senior are handling your case. What the hell happened? Will things work out? Sure the guru will present the finished project which has taken too long, isn’t as trouble free as it should be and was rather hard going the entire time. Well, nothing that can’t be fixed on a follow up consulting engagement, right? This is the oldest trick in the book. They trained their young potentials at your expense and now know which ones can’t hack it. You’ve become the free university for the consulting firm. Avoid this at all cost. Agree on who will do the work. Fix those numbers and names so you get the people you signed up for.

Giving away your business

Well those IT people are a tough crowd. They are opinionated and don’t communicate to well. Hell those guys & gals prefer to work with machines! They are not up to speed with what is politically correct or fashionable, hate faking, can’t stand save asses and don’t tolerate kiss asses. Good IT people live by the sword and die by the sword. It’s all very direct. Either their solution works or it doesn’t. You can’t hide behind reports, gold plated words or lies when you work in IT. The evidence is there every single second of every single day, staring you right in the face. So basically it’s no wonder that weak management and incompetent employees can’t get along with them very well, they are a bit to direct. So why not get rid of that problem? Get some consultants to do all that IT stuff. You’ll be pleasantly surprised by how courteous they are, well willing and facilitating. Sure it cost a bundle but hey, life seems good. Then your best IT people leave (physically or mentally) and your worst just hang around for the paycheck. No worries, no one is indispensible and firm X has more consultants for you! And so, slowly but surely, you become completely dependent on very expensive external staff and what you thought was consulting for your company has turned into a money generating market for a body shop that generates revenue by sending you their juniors to be trained at your expense. Congratulations, you fucked up big time. Your organization is now nothing more than a cash cow. When you no longer produce the gold, you get slaughtered. Game over.

Instead of outsourcing you should have invested in the IT team. Train them, communicate with them, be honest about projects and listen to them when they tell you it won’t work. They don’t do that to piss you off but because they think you’re making a mistake. Get rid of the idiots and assholes (they shouldn’t be there in the first place) and take care of your key IT personnel. They are hard to come by. Get external help from consultants where and when needed, but don’t hand over your business to them. If you do, you lose everything.

Buying the wrong stuff from the wrong people

This one seems so easy to avoid … Don’t buy the advice on what to buy from the company that sells the products. Don’t buy CRM services from a hardware company. Don’t let the marketing guys write software. What kind of switch will Cisco advise you to buy? Think about your interests and those of the vendors. Make sure they align. Buy where you get the best deal for your needs. Get the right expertise from the right experts.

Not knowing what you buy

You’ve seen the possible consulting engagements already.

· Advice

· Implementation

· Coaching

· Skills Acquisition

Now decide what you need and communicate this very clearly. Make sure everyone understands what is expected. Put it in writing. That way there will be no surprises such as getting an advice about e-mail infrastructures instead of a functioning Exchange 2007 system like you taught you’d get .

Buying efforts instead of results

Efforts are commendable, nothing more. They are not the desired outcome. It really doesn’t matter how much time, work, people, and resources you throw against a problem. The only things that matters are the achieved results. Buying efforts is a great way to make a consulting firm very wealthy. It’s usually the result of ill defined scopes, not getting clearly defined deliverables in writing and bad project management. The worst type of this mistake is “consultants” who start churning. Those bastards give true consultants a bad name. In essence this is fraud. They steal time and as such money by doing unnecessary work, prolonging issues or problems to augment the billable time, etc. Never forget that our economy a society is fueled by bull shit. Half of the consultants are selling just that. Hire the other half. Don’t feel bad if you’re a consultant, what do you think permanent staff sells half of the time? Right!

Falling in love with the misuse of “methodologies”

Sometimes both clients and consultant firms fall in love with methodologies or make mistake manuals for methodologies. They focus so much on the methodology everything else becomes secondary to it. This often has a couple reasons. One of them is that a lot of businesses lie when they say their personnel are their biggest capital or most important asset. They hate the fact that they are dependent on specific people and skill sets. It’s way too risky and expensive. They want to modularize people like parts of a car. If it’s broken, replace it with an identical one. The other reason is that talent doesn’t scale very well. This is quite normal, people just don’t scale. Talent needs to be cultivated and that takes time and effort. Performing complex tasks to produce high quality results takes talent. This makes things expensive as availability to talent is limited and thus growth is stalled.

So we’re in trouble here. We can’t attract enough talent and the bean counters insist on reproducible identical drones (“Human Resources”), which is impossible to achieve. So the solution consultancy firms come up with is the misuse of methodologies as a religion. They create guide lines, methodologies, scripts etc. You see, one way people try to scale talent is by creating a cookbook the “less talented” to follow. But books don’t make an excellent cook! This is because talent cannot be methodized completely. This is a farce. In the end it leads to mediocre firms, run by mediocre people producing mediocre products. Talented and motivated people will use methodologies correctly when and where appropriate. But they have also learned to be creative and the use of experience combined with knowledge and guts allows them to produce excellent results. Joel Spolsky did a great write up about this in Big Macs vs. The Naked Chef on his web site (http://www.joelonsoftware.com/articles/fog0000000024.html).

Now methodologies are useful when used correctly! But focusing on them where talent and creativity are needed will only drive the talent away or in hiding. Is this what you hired consultants to do? Is your organization in such a bad shape? Do you really have that many people on the payroll that are not capable of performing their jobs? Next thing you’ll be buying innovation and creativity as process from those same consultants.

Conclusion

Good consultants can and will help you to realize well defined, complex and important goals. However, before engaging consultants, you need to be aware of the pitfalls. Know what consultancy can and cannot do for you given the circumstances of your organization and your needs. Using consultants for mere staff augmentation is not a good idea. Good consultants know this and act upon it. You need to be smart, proactive and involved in selecting and working with consultancy firms. You just can’t expect to hire some consultants, lay back in your easy chair and hope all will be well from that moment on. Far from it, that attitude will only lead to more and very expensive problems.

The do’s and don’ts when engaging consultants Part I


Follow this link to Part II

Introduction

I see so many organizations fail to engage consultants in a rewarding manner that I taught I would share my view on the matter with you. Sometimes management wants to know why working with consultants often fails to deliver the desired result. That question and its answer remind me of the movie “A few good men” where Jack Nicolson’s outburst is legendary “I want the truth! YOU CAN’T HANDLE THE TRUTH!” http://www.youtube.com/watch?v=UXoNE14U_zM&feature=related.

Good consulting can produce very valuable and rewarding results. But next to these merits there are some drawbacks that you need to be aware of. This will ensure that your consulting experience doesn’t turn into a nightmare. The truth is that the use of consultants constitutes a major risk when not used correctly and deployed wisely. Never forget that IT consulting firms have a very different agenda, skills set, view and approach to your business than you do, the owner or manager. Also, do not assume that they only look out for your best interests. The better consulting firms do, even to the level that they will walk away from clients or engagements where they find that the business maturity level is insufficient to achieve value for both parties involved. Why? Because they are true professionals, who know there is more value in a long term, honest, business relationship than there is in short term profit.

Defining Consulting & Consultants

OK, so there are many definitions. I have my points of view. So do you. That’s fine. Perhaps you’ll disagree but I find it practical to define consulting or consultants by way of the results they are supposed to deliver. What all possible results have in common is that the consultants are responsible for achieving the result in the manner they see fit. The moment you are managing anything other than the consulting engagement they become overpriced employees. They are merely temporary or permanent staff, send form a body shop, to augment your head count where and when needed. Make no mistake, I’m not saying that this is not valuable if and when done right, but it isn’t consulting. This might sound esoteric but you’re the one controlling the engagement. So work and manage it. You can always (and should) disengage when things are not working out, not just keep going and complain about it. An American manager I once met put it very bluntly: “When the horse is dead, dismount.”

Let’s define consulting / consultants by the ways or the combination of ways in which they deliver results. There are a couple of ways in which consultants can provide consulting engagements:

  • Advice: they offer their best possible advice given the circumstances.
  • Implementation: they deliver what is needed.
  • Coaching: they coach your own staff to produce the desired result.
  • Skills Acquisition: they transfer skills and knowledge to your staff.

A lot of the time people mistake body shopping and contracting with consulting. As is the case with all distinctions the lines can be blurry. However, in my view, body shopping is not consulting. Contracting, depending on how it is defined and executed, can be one specialized way of consulting. Read on to find out more about this.

Body shopping (the non-consulting form of contracting) is the purchase of warm bodies that show up on site, from nine to five, under your direction and supervision. You will have to be able to manage this extra staff as if it were employees. Please note that these can be very highly skilled, specialized and thus very well paid people. They are just not consulting for you. No disrespect intended.

Now let’s look bit more in detail at what constitutes consulting by the type of results they produce.

Advice

You need advisors when you know the problem well, but need a clear understanding and guidance on what are the best possible solutions in your situation. They need to present their findings and insights to you, with the benefits and drawbacks, to finally come to a result: conclusions & recommendations. To get the best advice you should find real experts in the subject matter. Most likely these engagements are short term or retainer based. In this case hourly or daily rates are the norm.

Implementation

If you can clearly define and communicate the result you need and want, than you might want your consultants to handle the implementation that leads to that result. It doesn’t get any more tangible than this. You can define & control the scope and deliverables. In this situation you might be able to get a fixed price for a known outcome rather than daily or hourly rates. This might be a better deal for both parties involved. Consider a bonus for on time, on budget delivery. Call it contracting if you want, but the rules are that they design and implement the solution. Never, at any time, are they under your supervision and control like employees are. Yes I know some people who do not consider the actual implementation to be consulting, but I disagree. You cannot design stuff you don’t know and are able to implement. As a consultant you might not always implement the solution yourself but you’d better be able to!

Coaching

You have a brilliant staff, very knowledgeable with a solid skill set but they need coaching, focus, guidance and direction to achieve their full potential. They lead your people to success. This is an ideal situation to unlock the value within your organization. Get help from consultants with solid references and put together a plan to fix the deliverables and timing. Coaching is often acquired using negotiated daily rates based on amount of time and the duration of the engagement.

Skills Acquisition

Use this when you need to get your in house skill set elevated. Get some help to train them and bring them up to speed fast on the technologies you definitely want to develop and keep in house. Make sure you have good, hardworking, knowledgably and motivated staff. No use pouring expertise into the unwilling to do the unwanted for the uninterested. Again this form of consulting is often acquired using negotiated daily rates based on amount of time and the duration of the engagement.

You should realize that complex situations will require of two or more of the above types of engagement. Be advised that this requires a substantial management effort on the buyer’s part in order to make the consulting effort a success. Skills Acquisition combined with coaching is absolutely not an effortless commitment!

Using Consultants

In the unfortunate situation where your company is completely a drift and you don’t know the problem, what to do, what your people are capable of or don’t think learning skills are useful, you’re basically in a whole lot of trouble and you don’t have a prayer in hell to save yourselves. You need new top and middle managers to start over. Really! No help can get you out of this mess unless you’re willing to change the people and organization that led to this hell hole. But hey, I know some body shops that will gladly take your money and provide the personnel to help sink the ship. But no, these people are not consultants, nor do they provide consulting services.

If and when you engage consultants know what you are buying and why (Advice, Implementation, Coaching, Skills Acquisition or a combination of them) and communicate this very clearly. Buy the correct service from the right provider. It’s better to have mutually beneficial, long-term relationships with fewer but high quality providers who all have their areas of expertise and know their own limits.

Put it in writing. Define what and when. Follow up on your consulting engagement as it was defined in the contract. Sign off on the deliverables like any other project. Don’t allow “scope creep” without altering the contract officially.

In the next part we’ll look al the common mistakes when using consultants. Why do people make these mistakes so often? Because they don’t know any better or don’t want to know. Especially in larger bureaucratic entities life is easier if you shut up and go with the flow.

Having a Lab & Using It


It’s nice to have a lab, really nice. Trust me. I hear a lot of people complain they don’t have a lab and the time to test out all the possibilities and new features in recent or coming releases of operating systems & server applications. Why don’t they have a lab? Is it the money, lack of space, a shortage of time, the difficulty in setting one up?

Whatever the reason, can you afford not to have one? I think that depends on your area(s) of expertise. But no matter how you look at it we’re making our living in a live long learning business people. Learning isn’t that bad or hard. It is fun and you build on past experience and existing knowledge. It’s not like your starting from scratch every time you start exploring new technologies.

It’s your knowledge that makes you valuable to employers & customers. Even if you are an “architect” and you don’t have to deal with the lowly hands on implementations of your designs you need to know how to do it and be able to do it in collaboration with your team. So yes, you need to keep learning the hands on skills as well.

How long do you think it will take for your “grunts” to start giving you the one fingered salute if you proof to be an empty suit? At that point you can stuff your remarks about how easy it all has become because IT is a commodity that takes no expert knowledge. Your self-promoting sales talk that these facts in combination the excellent products and your outstanding architectures make it a walk the park to implement your solutions will bite you in the proverbial ass. If you don’t stand by your solutions in operational conditions and if you can’t implement and support them, go away. You can’t lead from behind sport!

So techies, system-network-storage admins, designers, analysts and architects get a lab!

Space can be an issue.

@Work: Well get an extra table and use that. Find and make more room. Yes the office will look nerdy but hey, form follows function. Unless of cause you want a designer “office look”, maybe because you’re working hard on climbing the corporate ladder. But even in that world the design of your suit and desk is not the most important characteristic. Those management & sales folks also have real and valuable skills!

@Home: If you’re single well … what do you think the living room is for? If you’re married and have kids it can become a bit harder. Have a spare room to use as study or home office? A free corner where you can set up your lab?

A word of advice: in a lab it pays to have direct access to the hardware during building and testing. How important that is depends on your needs and area(s) of expertise (i.e. networking versus SQL Server development). Perhaps you need to mimic clients their environments a lot for POC’s or trouble shooting and having a lab in a remote place isn’t really practical in that case.

Time

We all have 24 hours in a day. Time is the most valuable resource you have. You cannot buy more hours in a day; you can’t produce time or grow it. Make the most of the time you have. That’ doesn’t mean you can’t have fun and have to become a workaholic. But seriously, becoming competent takes time and a sustained effort on your part. Unless you have a naturaly aquired rare and very valued skill, you’ll have to put in a lot of work. No, you’re not entitled to a great paying job with lots of vacation time, benefits, company car etc. because you went to college or university. Well, not more than any other person on the planet anyway. Now perhaps you say that those efforts might not pay off or help you advance in your career. Perhaps that’s true, but won’t you benefit when you need or want to find another job? Being good at your game and learning skills means that you give yourself some options. It creates at the very least the ability and possibility to try and make something out of opportunities that arises. Opportunity never knocks twice. So be ready.

The real question is what do you do with the hours you have? Sleep for 8 hours. That’s good, you need your rest. Work for 8 hours. Check. That leaves another 8 hours for everything else. What if you carve 3 times 2 or 3 hours a week out of that for lab time and reading? That’s about 24-36 hours a month of solid learning. So where can we get those hours?

Stop watching TV every night. I don’t even own one. If you don’t watch television you have at least a couple of hours extra in every day! That doesn’t mean you have to miss out on the news or great series or movies or be a recluse. Go to the movies once in a while, perhaps a 3D one for the extra experience. Rent or buy your movies & series (on line or off line) and watch them at a time of your choosing. I’ll be dammed if I let a TV guide influence, let alone dictate, my schedule!  And as a bonus to myself I watch my movies on a HD beamer. Now that’s quality time.

Gaming? Sometimes I do that, yes. But I don’t do it very much. I you have time to game till the wee hours of the morning; surely you can’t complain you have no time at all to test & learn anything. There are choices to be made.

Start commuting by train if you can. Chances are your company helps pay for it (anything to reduce the peak traffic hell) and you’ll be able to get some reading done. When your train is late, even more than you bargained for. I plan as much of my travels outside of peak traffic hours. I also telecommute as much as possible. That makes traveling a whole lot less frequent and less stressful. It also saves time, time not spends in transit and traffic jams. Time you have to do something else. Ride your bicycle to work if you can. You get some exercise and avoid traffic jams.

Get out of the house regularly. Read whitepapers or books in the park or in a coffee shop. Go walking, jogging or cycling with an MP3 player with pod casts. And once in a while leave it all tech stuff alone and do other stuff.

Power Bill & the environment

Well I don’t own a car. When I need one is use a rental or barter with friends. Indeed this is “Car as a Service”, CAAS. It’s quite cheap actually compared to owning and running one. For the most part I drive my bicycle and use public transport (train).

So the money saved by not having a car can be used for the lab and the reduction in carbon footprint gives me plenty of surplus CO2 quotas to run a couple of extra PC’s a few times per week. Just turn it off when you don’t need it. In winter it also helps heat the room :-)

Is this unrealistic? How many of you have two cars at home? Home many electrical entertainment & household devices do you own? Two PC’s for 6 or 9 hours a week are going to break the bank and ruin the environment? Get real and get some solar panels.

Get Some Lab Equipment

You don’t have to build the lab feature complete from day one. Make good use of hardware replacement projects. Scrounge for discarded or redundant hardware. An old KVM console, replaced Cisco switches? PC’s or servers left over after a refresh cycle. Just keep an eye open for opportunities to get some good gear. Building such a lab is also fun and is a learning experience in itself. Even developers benefit from knowledge about hardware, networking, operating systems and virtualization.

Getting Access to the software

If you’re in the Microsoft ecosystem then a Tech Net Plus subscription is absolutely one of the best investments you’ll ever make. Developers should look towards MDSN but that’s considerably more expensive. A lot of vendors have free editions and trial versions out there. Use them.

Servers & PC’s

I have 3 small form factor PC’s and one mini tower PC with 4 to 8 GB of RAM and 250 GB of disk space. I also have a laptop for mobile work and a very beefy desktop with dual screens.

The lab is managed using KVM and a couple of extra 21” Flat Screens I got from an office sale. A lot of the time I just RDP into the servers I need to work on.

The mini tower has an extra 500 GB disk that is used as storage for the iSCSI target running on there, which means is can do clustering. Yup I run a Hyper-V R2 Live Migration setup at home.

Can I do with less hardware? Yes, but you can’t go entirely virtual if you want to play with Hyper-V clustering for example.

Networking

Next to an ADSL/VDSL/Cable Modem/Router, get one or two decent 1Gbps layer 2 switches, at least web managed, to test networking scenarios.

Add an extra router to the mix if you want to do sub netting/routing/VLAN’s or get a layer 3 switch (expensive).

Use differently colored Cat 5e cabling to keep things organized: yellow for iSCSI, orange for Hyper-V, green for ordinary LAN, etc.

Each hardware server node has an onboard NIC and I added an Intel 4 port NIC cards. That means they have room for iSCSI, LAN, Hyper-V dedicated NICS, multiple subnets etc.

It looks overkill to some but if you want to test various scenarios, with multiple subnets, routing, VLAN’s (with or without using VLAN id’s) this is very comfortable.

Information

There are many whitepapers, web casts, podcasts and self-study books. Use them with on line documentation, support forums, blogs and articles to get the most out of your lab. The resources out there are vast and varied. You’ll find something that fits your learning style. Don’t just learn about pieces of technology. Learn how they interact with and depend on each other. Get some business insight about the use of technology.

What do I do with all that?

Using this kit and virtualization (VMware Server, Hyper-V) I run AD, DNS, WINS, DHCP, SQL Server, Exchange, MDT 2010, WDS, SCSCOM2007R2, SCVMM2008R2, W2K8R2 IIS etc. Just build your core infrastructure and add or remove virtual machines and environments as needed. This is the lab where I test upgrades, migrations transitions, learn about functionalities, test ideas and reproduce issues to find out how to fix them. This is also the home office from where I telecommute and run my little IT worlds in a couple of cities. Broadband, remote access cards (ILO/DRAC), KVM over IP in combination with secured connectivity make this all possible today.

Now this is all very modest, believe me. You don’t have to be Stefan Didak (http://www.stefandidak.com/office/index.php) to get a great learning & working environment set up.

Selling a lab to your employer & company

For your employer, company and actually also your customers it is a very big benefit to have lab. The better you are in what you do the faster you can deliver better quality results with less risk. The cost isn’t really that high. The moment the cost of becoming and staying competent is higher than hiring others to do the work; it’s time to seriously start thinking about sourcing the work.

At one place we are getting busy to build a test Hyper-V cluster with “recuperated” material from their data center. The servers are recent PE 2950’s from DELL that became available due to a very successful virtualization project. We have access to shared storage, both iSCSI and FC, that they have in place. The thing about this client is that they know that the “cost” in recuperated material is nothing compared having under skilled people working directly on production systems. Their entire lab (servers, network kit) is worth perhaps 20.000,00- to 25.000,00- Euro’s originally, but now it amounts to perhaps 4.000,00- 5.000,00- € in all now on eBay. As an example check out this link: http://tinyurl.com/ybfofb4

If your management thinks training is useless and doesn’t produce results they have multiple serious, issues. One of them might be very bad HRM. Which is even more proof that a lab isn’t that expensive when you really need one I guess? Consider a lousy employee with a master’s degree. One of those cost around 75.000 € / year all overhead & taxes & benefits included. And then you still have to make a profit. Let’s assume you have 500 employees an only 2 of them are really not functioning well to a level that they become a burden to others. That’s 150.000 € right there. Even worse, you need to hire extra employees, contractors or consultants to get results they are incapable of delivering. So actually getting one good employee gets rid of that cost and you still have plenty of cash to buy some lab gear and put a decent savings on your financial report.

Using Windows 2008 (R2) Backups to Go Virtual Part III


Part III

Sanitizing the Disk Configuration & Layout


This series describes a physical to virtual migration of a Windows Server 2008 SP2 Standard x64 on a Dell Power Edge one unit rack server with SATA disks (PE750). In fact that was the first production Windows 2008 server ever at that company, they really wanted to use the RTW FTP over SSL functionality. For this exercise we had no use of commercial P2V tools and the original server had to be kept running during the virtualization exercise. The aim was to achieve this with the free / built in tools as much as possible. The hands on approach also provides for an excellent learning opportunity. We have reconstructed our experiences for this article and will discuss the subject for both Windows 2008 and Windows 2008 R2.

 Part I: http://workinghardinit.wordpress.com/2010/01/27/using-windows-2008-r2-backups-to-go-virtual-2/
Part II:  http://workinghardinit.wordpress.com/2010/02/01/using-windows-2008-r2-backups-to-go-virtual-ii/

ParrtI:

Breaking the Software Mirror


We don’t need the software mirror anymore for redundancy. On the Hyper-V host the disks are LUNs presented from SAN. The SAN provides the redundancy. So the first thing we do is break up the software mirror. You can use the Disk Management GUI to break the data volume(s). Select the member of the mirror you are not booted from, right click and select break mirror. Confirm that you want to do this at the warning message and you’re done. We opt to break the mirror as this gives us to working copies. This comes in handy in case we mess something up. Removing the mirror is also an option but then you’ll end up with only one copy.

Part III – Figure 1: Breaking the data partition mirror.

Now when you try this on the boot/system volumes this will not work at all. You get the following error: “The specified plex is at the current system or boot plex”.

The solution to this is using diskpart to break the mirror as explained in following KB article: Software mirrored boot volumes in Windows Server 2008 cannot be broken from Disk Manage
http://support.microsoft.com/kb/969749/EN-US I guess somewhere Windows GUI admins where deemed not savvy enough to know what they where doing?

Part III – Figure 2: You have to use Diskpart to break the boot partition mirrors.

Sanitizing Disk configuration and layout


The task at hand is to go from dynamically expanding VHD files to fixed vhd files and shrinking them so they don’t waste so much space. While dynamically expanding vhd’s can be compacted to take up less physical storage, the total capacity of the vhd disk itself does not change.

One option would be to convert them but then the issue with the available tools Hyper-v from Microsoft is that they become 1TB fixed vhd’s. We would then still need to shrink them. The Microsoft Hyper-V tools however only allow for extending disks, not shrink them. Another consideration when using the Hyper-V Disk Edit tool to convert to a fixed disk is that you need sufficient disk space to do it. We are dealing with + 1TB disks here and we just could get enough free disk space to do this.

There is a free tool called VhdResizer (http://vmtoolkit.com/files/folders/converters/entry87.aspx, you need to register for the download) that can shrink vhd files. However, it needs free, non-partitioned space to do this. And it cannot handle Windows dynamic disks. This brings us to another major concern and that is that many disk tools can’t handle dynamic disks and only work well with basic disks. To be clear, I’m not talking about dynamically expanding vhd’s here but about dynamic disks in windows. Try working with a vhd (dynamically expanding or fixed) that contains a Windows dynamic disk and you’ll see this error:


Part III – Figure 3: VhdResizer can’t handle Windows dynamic disks.

The Data Partition


Let’s first focus on getting the data from its partition on the boot disk with the operating system to its own disk. That disk has to a fixed vhd disk with a windows basic disk. There are several ways to get there. Not all are fully supported by Microsoft. We list three approached that work.

Copying or Restoring With NTFS Permissions Intact.


Xcopy, robocopy the data from the original data partition or restore the data directly from backup to a newly created fixed vhd of the desired size and which is a windows basic disk in the virtual machine. This method can retain NFTS permissions and is fully supported. Afterwards we can delete the original partition.

Cloning


We could use a disk clone tool that supports dynamic windows disks and clone the partition to a new fixed vhd of the desired size and which is a windows basic disk in the virtual machine. Ghost.exe/Ghost32.exe can do this perfectly. The cloning is supported by Norton in this case. Afterwards we can delete the original partition. Or use Gparted to achieve the same thing al be it a bit more hands on as you’ll see when we start playing with the boot/system disk.

Using VhdResizer/Gparted free tools


Another way which requires also no paid for tool and uses VhdResizer is the following:

Take copy of the vhd from the backup folder that corresponds with the data partition.

Rename it to something sensible if you want.

Add it to the virtual machine.

Start the virtual machine. You’ll see that is a windows basic disk.

You need to create free space on the disk resize the disk with VhdResizer. So shrink the partition to its desired size or a small as you can with Disk Management. This might still leave considerable space (about half of the original partition size) due to unmovable files.

If that’s too much like in this case (we could not shrink the partition in windows beyond 500 GB), gparted live CD comes to the rescue! The disk is already windows basic disk and that is a format this tool can handle. So if you need to shrink more that windows will allow you to do. Turn to gparted live CD. If the disk you’re working with is larger than 127 GB like in this case mount the disk you’ll have to test various versions … for example gparted-live-0.4.1-2.iso can handle this, but the more recent gparted-live-0.4.6-1.iso can’t. Such in live in systems engineering J


Part III – Figure 4: resizing with gparted.


Part III – Figure 5: reading the data


Part III – Figure 6: Copying the data, after this step it’s done! Just exit gparted & shutdown the VM.

Stop the virtual machine

Remove the vhd from the virtual machine

The vhd is a dynamically expanding one but you can convert it to a fixed vhd and resize it (make it smaller in our case) with vhd resizer. This takes a long time and the GUI doesn’t give any decent feedback. Give it time and be patient.


Part III – Figure 7: Using VhdResizer to shrink the vhd and convert it to a fixed one.

Attach the resized, fixed vhd to the virtual machine, start it, make sure it gets the correct drive letter and you’re good to go!

Now VhdResizer.exe is not supported by a vendor, it’s freeware. Gparted Live CD has no warranty but you can get decent information/help on line. Be aware of the fact that between versions (like the example above) certain functionality is sometimes broken. Such is life. The good thing is that Gparted had drivers for keyboard and mouse that work in Hyper-V from the start. A lot of other tools didn’t or don’t. In the end make up your own mind. I have used all methods and tools on several occasions with success.

The Boot/System Disk


Here we want to get rid of the large data partition on the disk. If the above actions where successful we can just delete that partition. We already have the data covered. Now we still want to get to a fixed, decently sized vhd that contains a Windows basic disk. Again we have several options but beware one of them is very risky and not supported by Microsoft. We list three approached that work.

Cloning


Agian we could use a disk clone tool that supports dynamic windows disks and clone the partition to a new fixed vhd of the desired size and which is a windows basic disk in the virtual machine. Ghost.exe/Ghost32.exe can do this perfectly. The cloning is supported by Norton in this case. Since we didn’t provide any details of this above we’ll dig a bit deeper here. Gparted can’t be used directly as it doesn’t handle windows dynamic disks.

Warnings concerning the techniques discussed below:

When you mount the same dynamic disk as a second drive on a virtual server that boots form the original copy of that dynamic drive you’ll get errors in diskpart/disk manager about the second drive being invalid. Trying to manipulate it (bringing it on or off line) makes the system unresponsive and even corrupts that drive. The trick is to mount the bootable dynamic disk (on the dynamically expanding vhd we want to clone) in another virtual OS that boots form a basic disk.

Now that makes you appreciate Windows 2008 R2 where you can just attach a VHD and work with it, right? Or so you would think. Wrong!
THIS DOES NOT WORK WITH WINDOWS DYNAMIC DISKS in a vhd, whether it contains a bootable partition or not. They will also pop up with “Invalid”. WARNING: If you convert it to a basic disk, you’ll lose all data. The good news is that vhdmount from Virtual Server R2 SP1 does the job.

OK now you have been warned lets walk through the cloning process. Create a new fixed VHD file of new required size. Mount both the new and the old vhd file so we can clone them. You can mount these vhd in several ways depending on what you like best or is more suited for your environment.

You can mount the disks in a host using either the native vhd mount capabilities of Windows 2008 R2 if they are NOT Windows dynamic disks (See warnings above). In our case we cannot us this as the original disk is a dynamic disk. If you haven’t got Windows 2008 R2 by using the Virtual Server 2005 R2 SP1 VHD Mount utility which is a free download? Follow the instructions at http://technet.microsoft.com/en-us/library/cc708295(WS.10).aspx to get vhdmount operational. This does work with dynamic disks! You can also attach them to another virtual machine (Do not use the original one, see warnings above) as extra disks and run ghost/ghost32 in that virtual machine. There are lots of options here; you can figure it out it’s not hard.

I’ll just post the commands for vhdmount I used here:

vhdmount /p /f “E:\Disks\DynamicallyExpandingDiskWeWantToResize.vhd”

vhdmount /p /f “E:\Disks\FixedEmpty.vhd”

Note: Mounting the VHD files is possible only if the VHD files are not currently in use by any virtual machine or other program. You should receive a “The Virtual Hard Disk is successfully plugged in as a virtual disk device.” response, followed by a series of “New hardware found” messages.

Initialize the disk to write a MBR to it to avoid possible booting issues you’ll need to fix.

Run GHOST32 utility, which can be found in the directory where you’ve installed Symantec Ghost. Using the GHOST32 utility, you can “resize while copy” any of the local drives (Local → Disk → To Disk) including virtual ones now mounted. Make sure you choose the destination disk drive that represents the new VHD file, as you can easily overwrite any other local disk drive. You can always use GDISK32 utility to determine first which one is the new drive if uncertain.

Once transfer is complete you can safely choose not to restart the computer and just exit the GHOST32.

Don’t forget to detach the vhd’s! When using vhdmount use following commands:

vhdmount /u “E:\Disks\DynamicallyExpandingDiskWeWantToResize.vhd”

vhdmount /u “E:\Disks\FixedEmpty.vhd”

Hxd Hex Editor


This is risky business but worth a try just to give us an option with a free tool. We can convert the dynamic disk to a basic disk with Hxd editor http://mh-nexus.de/en/hxd/. This is totally unsupported but I’ve had great success with it. It doesn’t work for all conversions under all conditions but it’s worth a shot. Make sure to make a backup copy of your virtual machine before truing this! It’s the only way I know to convert a dynamic disk to a basic disk without a commercial tool like Ghost that is one of the few that supports dynamic disks. You can find a good write up of how to use Hxd to convert a dynamic disk to a basic disk here: http://strangelyperfect.tv/6415/how-to-convert-a-dynamic-disk-to-basic-disk-in-windows-7/ & here http://www.dynamic-disk.com/convert-dynamic-disk-to-basic.html

I’ll just report what I did here.

Install the tool in the virtual machine, launch it and select the disk you want to edit. In our case it’s disk 1. In the partition table live in the area between the two arrows. Every 42 you see starting at 1C0 up to must be changed into 07. You have to them all (one for all primary partitions the disk sees) or you’ll mess up the disk.


Part III – Figure 8: The select disk to edit. Note the 42 values in the partition table.


Part III Figure 9: Warning when you save the changes to disk in Hxd.

Save your changes to the disk, these are the values marked in red. Acknowledge the warning, close Hxd and restart your virtual machine. When the servers reboots successfully you can have a look in Disk Management. Sometimes you’ll have to do some disk cleaning afterwards as in the figure below, but the important thing is that the dynamic disk has become a basic disk. Just delete the volumes on the dynamic disk that’s reported as missing and that’s it.


Part III Figure 10: Clean up the dynamic disk that is now reported as missing because it was converted to a basic disk.

Again, I haven’t always been able to get this tool to work in every situation. But it’s worth a try on a COPY of your virtual machine if you have no other options.

You can optimize your disk partition size if you want using Windows Disk Management or Gparted. And finally we can delete the data partition and follow the same methods we described to create data partition to resize the disk to the desired size, using VhdResizer … or not?

Nope no such luck! When I tried to use VhdResizer.exe we got that error again “Invalid VHD file. Please select another”. Now what? Well cloning the disk to a squeaky clean fixed vhd directly and after that see if VhdResizer.exe can handle it. Of cause if you have a disk cloning tool that can handle cloning from a larger disk to a smaller disk you do no not need + 1TB in space and you can simply skip the process of having to reduce its size with VhdResizer.exe. Ghost can handle this but is a paying tool. Gparted is also very handy in achieving this, but more hands on. I tried both. Ghost worked well and without issues. I used ghost/ghost32.exe (11.5). Again we can boot into a virtual machine to do the cloning or mount the vhd’s using VhdMount.exe or directly in the Disk Management GUI (W2K8R2 only) since the disk is already a basic disk! Now to prevent any issues, the ones we warned about above, during the cloning we like to mount the source disk and the target disk in an operating system that is another (not just a copied virtual machine) than the original. That will save you a lot of head ache.

But we really wanted to succeed trying this with a free tool so we took to Gparted once more with the source (dynamically expanding vhd) and target disk (fixed vhd) mounted in a virtual machine.We succeeded to clone the partitions one by one to a new fixed vhd disk. We also marked the correct partition as bootable as judged form looking at the original. But booting from that new disk failed with error 0xc000000e, the boot selection failed because a required device is inaccessible. After rebooting into the Windows 2008 (R2) DVD and opting to repair an existing Windows 2008 installation that it found (0 MB!) I launched the command prompt and ran following commands from there:

bootrec /fixmbr

bootrec /fixboot

boorrec /rebuildbcd


Part III Figure 11: Fixing 0xc000000e the boot selection failed because a required device is inaccessible

For more information see “How to use the Bootrec.exe tool in the Windows Recovery Environment to troubleshoot and repair startup issues in Windows” http://support.microsoft.com/kb/927392.

So than I restarted to try and boot from disk, I was told it needed to run chkdsk which is normal when you used Gparted. After chkdsk ran it booted into the operating system from a fully functional fixed vhd disk that was resized to the correct dimensions. It was even accepted by VhdResizer.exe again as a valid vhd file when I checked.

In the end all was well. Now for the record, this was a demonstration of what we did in one case. It worked for us. The virtualized server is running happily on a live migration Hyper-V cluster as you this is published. Not bad J Just don’t expect Microsoft to support you if you have issues, you’re on your own when you venture into this area. Search and find your own creative solutions, you’re working on VM’s so don’t worry, take copies and you have a lot of changes to try again and again …

The Final Result achieved in production


As a reminder: this is where we came from on the production physical server.

Part III Figure 12: The disk configuration on the original physical machine (W2K8). The bad quality is due to it being taken over a KVM over IP session.

And here we have our final result. This is the layout of our production virtualized server.

Part III Figure 13: Success, the desired disk configuration in the achieved production virtual machine (W2K8).

So we have what we set out to achieve , a virtual machine running on Hyper-V, with 40 GB system drive on a basic disk in a fixed VHD and a 150 GB data drive on a basic disk in a fixed VHD. We succeeded in separating the C: and the D: partitions to their own disk (vhd). The C: and D: disks were also converted from dynamic disks to basic disks. They were also shrunk to a more acceptable size. Finally we converted the disks from dynamically expanding vhd disks to a fixed vhd disks. Mission accomplished.

Conclusions


Some understanding and hands on experience with Windows 2008 (R2) Backup and Complete PC Restore are very valuable to have. Learn this in labs and during testing. Make sure you don’t have to your first restore for real on a critical server.

There is so much to try and experiment with regarding the disk conversions and resizing that it’s almost impossible to discuss them all here. Every situation is different & what I did here might not work for you. So be careful. The good thing is that, once you have virtualized your server, you have the ability to make copies and backups very easily so you can afford to experiment. I suggest you do exactly that and learn from the experience. We all have situations, environments where things are not ideal and where we need to use our creativity to achieve the desired outcome. Whether all of this is worth the time and a good idea in a particular situation is not for me to decide. I’ll just try and get the job done when needed. Advice and consultancy on how to prevent issues, how all their stuff should be in the cloud and such is all very well but people tend to get annoyed with that when they are bleeding from open wounds. So fix their wounds first. And that’s what we did here. Their entire old SATA disk based 1U rack server infrastructure has been virtualized or replaced and is now running happily on a high available Hyper-V cluster.

Well, that’s it. I hope you find some useful tips, pointers and inspiration for both Windows 2008 (R2) Backup and P2V migrations in these notes. The build in tools are good and work well. But you do need to learn about their abilities and limitations. They are free, support and you don’t need agents to backup/restore. Are they the best fit in any situation? No. Are they sometimes the best value for money? Absolutely!

Using Windows 2008 (R2) Backups to Go Virtual II


Part II

Restoring the Backup – P2V

This series describes a physical to virtual migration of a Windows Server 2008 SP2 Standard x64 on a Dell Power Edge one unit rack server with SATA disks (PE750). In fact that was the first production Windows 2008 server ever at that company, they really wanted to use the RTW FTP over SSL functionality. For this exercise we had no use of commercial P2V tools and the original server had to be kept running during the virtualization exercise. The aim was to achieve this with the free / built in tools as much as possible. The hands on approach also provides for an excellent learning opportunity. We have reconstructed our experiences for this article and will discuss the subject for both Windows 2008 and Windows 2008 R2.

Part I: http://workinghardinit.wordpress.com/2010/01/27/using-windows-2008-r2-backups-to-go-virtual-2/
Parrt III: http://workinghardinit.wordpress.com/2010/02/02/using-windows-2008-r2-backups-to-go-virtual-part-iii/

This will probably be the most “boring and redundant” part of the series. A very good post about this process for Windows Server 2008 can be found on the internet @ http://blogs.technet.com/askcore/archive/2009/02/04/windows-server-backup-2008-restore-from-network-location.aspx. They also discuss the command line (wbadmin.exe) way of doing it. So check it out if you need to do that. I did a restore of a Windows Server 2008 R2 machine for this demonstration for completeness. You’ll see that some of the redundant messages during a restore have been fixed and the work flow has been stream lined.

If you did the preparations for the virtual machine mentioned in Part I correctly this part will hopefully be an uneventful activity. Apart from the fact that you do a bare metal recovery from the backup of a physical server to a virtual server which is pretty cool, the result of this restore is the virtualized server with mirrored dynamic disks we’ll have some fun with in Part III. But in the end Part II is a GUI walk through the restore process for illustration purposes as this series is part of a workshop/documentation paper on Windows 2008 (R2) backup and restore for internal/personal use.

Walk Through

We boot the virtual machine we prepared in part I from the Windows 2008 R2 DVD ISO image that we the mounted via the setting for our virtual machine in Hyper-V.

We choose the desired language/keyboard settings and click “Next”.

We’re not going to install Windows but do a restore so we need to select “Repair your computer”.

We’re using a blank virtual machine, so the fact we do not see any installed operating system is no reason, to worry. Select to “Restore your computer using a system image that you created earlier.” and click Next.

At the “Select a system image backup” Click “Next” and your system will be scanned for system image disks

You will get a warning that “A valid backup location could not be found”. Just ignore this and click “Cancel”. It’s perfectly OK.

No latest available and automatically found images appear. This is logical as “a valid backup location could not be found (we have our backups done to a network share) and this is bare metal recovery. It would be spooky if something was found. Select “Restore a different backup” and click “Next”.

Click Advanced. You do not need to install a network driver if you prepared the virtual machine with a legacy network adapter, that driver is included in the Windows Server 2008/R2 DVD so that it just works. The synthetic driver is not included in Windows Server 2008 DVD but it is in Windows Server 2008 R2 DVD.

It will ask “Are you sure you want to connect to the network”, which is what we want and then specify the path of your backup. Remember the remarks if you do not have name resolution you can use the IP address. Also make share you have the credentials needed to access the share, you really need those.

Fill out the UNC path to where you store your backups. That is the path that points to the root where the “WindowsImageBackup” folder resides.

When asked for credentials please provide ‘m.

Select the backup listed and then click “Next”.

Select the desired system image and then click “Next”.

You are presented with the restore options.

We don’t want to exclude disks, we don’t need to install drivers and the default setting under the Advanced button work fine for us, as we need to restart the computer to finish the restore and there is nothing else left to do. Checking for and updating error information is a good thing.

Click Finish to confirm the settings. As we selected “Format and repartition disks” they need you to confirm that this is what you really want. Click “Yes” to start the restore.

You then see a number of dialogs reporting the progress of the restore operation.

The virtual machine restarts and boots into the restored operating system. After verifying all is well with the restored server, making sure we have the integration tools up and running we have accomplished our physical to virtual migration.

The final thing to do is open up a Command Prompt Windows (run as administrator), enter set devmgr_show_nonpresent_devices=1 and then start devmgmt.msc from that same command prompt window. After selecting show hidden devices on the Device Manager View menu you can now see devices that are no longer present in the system, we’ll find our missing hardware NIC’s and delete them. That way we can assign the configuration to the new NIC. See http://support.microsoft.com/kb/241257 for more information. It’s about Windows 2000 but it still applies. That’s it. In the next part (Part III) we’ll deal with the disk sanitization & reconfiguration.