Stupidity Is Not A Great Consulting Skill


Most companies don’t get that body shopping and contracting is not consulting. And today way too many consultants don’t match up to the requirements of being one. So basically these people pay way too much for too little. I have no mercy for fake consultants and companies so messed up in their political crap they fail to recognize quality and value. Just read The do’s and don’ts when engaging consultants Part I and The do’s and don’ts when engaging consultants Part II where I’ve given some rules of fist for optimizing the use of consultants. But here’s some advice to fake consultants. At least try to act professional and be nice. Because one on the main rules of long term happiness while staying employed is the fact that people not want to work with a*holes.

  • Don’t download or synchronize hacking tools while at your employer/customer.
  • If you’re an IT Pro or developer  and you cannot figure out how to expand or mount an ISO with the tools on a modern PC/OS you should realize they want to fire you on the spot for gross incompetence.
  • Playing games like a long term frustrated employee with help desk personnel regarding mundane tasks and leveraging policies or rules as excused not to get work done is annoying, immature & frowned upon.
  • Waiting until you get asked why you didn’t get work done to mention your PC got confiscated due to a security incident an the replacement was missing a tool proves you’re a dead weight.
  • Churning is nothing short of fraud. You deliver real work or get out.
  • It’s rather easy to see that the hours you telecommute are not matching your time sheets. It gets noted. At best they’ll pull a Yahoo on you.
  • When you state that optimizing parallelism in SQL Server and using multiple LUNs and data files is waste of time in an environment that’s on the record for being in the top 5 SQL server operations in the country you might perhaps realize you are being an a* hole. Yes it requires some work, the database is not designed to cater to your laziness, but for performance.
  • It’s designed and build at a great value point by skilled people who earn mediocre wages but are very good and love what they do. Do you even realize the level of your arrogance and the fact that you are now on the “do not hire list”?
  • Realize that not all people who’re working as an employee are clueless sheep that only serve to line your pockets. Most of them are good. Many of of them are great, some even excellent to exceptional. Even when not subject matter experts they’ll see through any fake expertise.

Look, when you’re dead, you don’t realize that, so for you there is no drawback, it’s all good. It’s your surroundings that notice it and have to deal with the loss, the pain. It’s the same when you are stupid. It’s the people around you that suffer. Get it?

Technical Projects, Planning, Skills, Motivation & Psychopaths


When planning a technical project complexity adds ups very fast. Take a virtualization project for example; a lot more things than just the hyper visor installation are coming into play. You’ll need to asses a lot of needs and desires about SANs (snapshots, redundancy, replication, FC, iSCSI, FCoE), network (VLAN, 1/10 Gbps Ethernet, redundancy), disaster recovery/business continuity, hyper visors and there capabilities, management of it all and security. That is a lot of stakes and agendas to take into consideration. And then you haven’t even talked to the business managers, the application owners and developers. Now this isn’t limited to virtualization, but this is just a nice example on how so many stakes come together in one project.

One of the major mistakes, that is made again and again even up until this day in the second decade in the 21st century is the fact that entire important or even critical IT systems are being put into place with a plan that can be paraphrased as follows “We’ll just set it up and sort of see how it evolves and just wing it from there”. I have been forced to do this quite often. This creates many problems some of which I will address below.

The single worst problem is that you create a vacuum. That can be storage space, bandwidth, ample resources for a huge amount of virtual machines or a mixture of all this. The results however are always the same and is one of two possibilities. Either they really don’t want and need it so it will never be used. You can also achieve this by keeping it hidden so they can’t use it. The other option is the most natural one. In nature there is a thing called a “horror vacui”. That means that a vacuum unless protected cannot exist, it has to be filled. Empty LUNSs with data, hyper visor hosts with guest, networks with bandwidth and backup capacity with even more terabytes. You might think the second option is better than the first one as at least the infrastructure is getting used. Unfortunately the reality is that this is creating a very expensive mess to run, support & troubleshoot. The legacy this creates is not a valuable inheritance but a bank breaking, efficiency and effectiveness ruining debt. Stop doing that right now, you are killing your business. You see technology debt is about more than just old hardware and software. It’s about what you build with it or what grows organically with it. Is that a fertile land that sustains the business or a cancer that is killing it?

The way to prevent this is planning done by competent, involved people with experience and context. No plan is perfect, but a plan gives you a framework to achieve the desired result. Even great people make mistakes but they have the skills and attitude to fix them or work around them.

What are some other problems? Wasting money. Take for example a completely oversized server farm. That thing will consume so much money over a three year period in energy and idle capacity that the amount would be sufficient to replace it with new right sized hardware (more bang for the buck, better energy efficiencies in three years) I don’t know about you but those are very disconcerting numbers.

You can also be wasting money and time. And those who know me I loath wasting time. What if the SAN solution you bought doesn’t perform as planned or isn’t the right fit? There goes 500.000 € or you find yourself in the CEO office explaining why you need an extra 400.000 € to get what is really needed. Oh oh! Do you have money and time to do it all over again or will you be living with that expensive mistake until the current solution is end of life? Do you have to wait until the CFO and CEO have recovered enough from the shock to allow a new attempt? Or perhaps you bought a SAN solution that is enough to run NASA’s workload and you’ve invested 4.000.000 € in a rather expensive data room heater.

Getting a virtualization project wrong can wreak havoc on a business and create a sizable financial hemorrhage. You can say that that’s not your problem but I beg to differ. If the project goes south that means you’ll have to find another job. The IT world where I live is rather small so you might even have to switch to another field as you’ll be forever known as the guy that sunk company X with his little “plan”.

The reverse, being rewarded for your hard work and success is not a given. In the end they pay you for getting the job done so results are expected, and to Joe Average manager all ICT is a PC with a software packet to install. So for all you eager beavers who think that with this kind of responsibility and risk management comes big reward when you get it right I suggest you think again. I have witnessed quite the opposite personally. Even when you’re running multiple enterprise SAN’s, networks, infrastructures like SQL Server, Exchange clusters, Hyper-V clusters, geo clusters, load balancers and providing 2nd and 3rd line support for those and taking 24/7 responsibility for the environment the only thing some managers care about is why the PC they never ordered with the software they never ordered can’t be installed tomorrow. “What kind of a chicken shit outfit are you running here” is what they’ll think when you can’t do that. They’ve read the glossy brochure that IT is a commodity and they expect it cheap and always on, much like electricity. In the end some (incompetent) managers act like ungrateful psychopaths. They’ll just abuse you less when you get it right. Don’t expect anything else. Often it’s the ones that are not capable to integrate things they can’t do or don’t understand into their business. They can not value anything that’s beyond their comprehension so they’ll never recognize it. To them, people are, for all practical purposes, resources that are identical, “Full Time Equivalents”. So don’t buy into the hype that there is a skills shortage from that lot and they can’t fill job openings. The volume in which they often waste talent and flush motivation down the drain is shockingly high and indicates that there is no shortage at all or that they can’t recognize skills when they find it and they’ll hire anyone. Surely they didn’t make a mistake so it must be a skills shortage. So you still want to be some hot shot technical architect? Or does a job that only produces open opinions and optional advice on paper sound more attractive. Per hour worked you’ll earn more, run less risk and have a lot less stress. My advice? Don’t switch fields if you enjoy what you’re doing, switch jobs. The best career advice I ever got was “don’t work with or for assholes”.

Well if you don’t agree with your bosses and you dare go against them you’re surely playing with your job, you could get fired! So? Does living in fear of being fired make good employees? Does not being strong and confident enough to tell your managers they are doing certain things totally wrong or that they are mistaken make for good advisors? The worst thing a boss can have are a bunch of “yes men” around him or her. That boss should be smarter than that. It doesn’t work. Having trust in the abilities and loyalty of your employees does not mean you need to agree on everything. As a boss you’ll make the final decisions, yes, but you’d better listen very carefully to your advisors and staff or you might as well have hired some monkeys. You can train them to say yes all the time, all it takes are some bananas. As an employee, don’t let yourself be treated like a monkey and if they fire you for throwing the banana back, good for you!

So you’d better love technology and building solutions because that means you are intrinsically motivated to go the extra miles. When you are, select a small group of people with the same attitude. You’ll be able to drag the devil himself out of hell with such a team at relatively very low cost. Whatever you, do don’t think you can externally motivate or coerce people into achieving this. Charles “Chargin’ Charlie” Beckwith knew that all along when he said “I’d rather go down the river with seven studs than with a hundred shitheads”. And guess what, he wasn’t taught this in some course, by getting a title or by being told this by a manager. He learned it himself by working with the best. These people will keep learning and growing on their own. They don’t need to be told what to do, how to train, what to use, they don’t need nannies & micro management. They need an end state and they’ll get it for you. Frankly that kind of skillset and ability scares the shit out of some bosses as they micro manage actions & items instead of doing their jobs. You can’t use force, treats or authority to make people achievers. In the end you can cut a diamond, but you cannot create it. Trust me. Putting that amount of pressure on someone that isn’t a diamond only turns them into a heap of crushed remains of what used to be a human being or FTE in your typical HR speak.

“Mate you’re not a conformist” my friend said … you’d better believe I’m not Winking smile

Shameless Plug For Mastering Hyper-V Deployment By Aidan Finn


In October 2010 Aidan Finn (MVP) his book “Mastering Hyper-V Deployment” was released and in November three copies of this book landed on my desk. I bought them (pre order) via Amazon. Nope I did not get them as a gift or anything. Why Three? Well that’s the number of people I wanted to get up to speed about Hyper-V and virtualization management and operations in a Microsoft environment.

His book takes you along a journey through a Hyper-V project that will teach you about virtualization in all it’s aspects. It also touches on many supporting technologies and products such as System Center Virtual Machine Manager 2008 R2, System Center Essentials 2010, Data Protection manager 2010 and System Center Operations Manager 2007 R2. No one book can be the only source of knowledge and understanding, but using this book as a start for both new and experienced IT Pros to learn about virtualization with Hyper-V will give you the best possible start. Consider it going to an Ivy league college on a scholarship paid for by Aidan’s experience and hard work. The subsidized tuition fee is the price of the book.

We feel a bit sorry that Aidan only got one copy so we made a group picture of the gang of three on the desk of our newest team member. He got a copy of the book together with 4 recycled PC’s and a TechNet subscription to build a lab.

image

If you know people who want or need to learn about Hyper-V, you’d do well to make sure they get this book and have them set up a lab to play with the technologies. Those efforts will pay off big time when they implement their solutions in the wild. If Ireland is doomed it won’t be because of smart & hardworking Irish IT professionals like Aidan. You see when you design, build and support IT solutions that your customers depend on 24/7 you can not hide behind false promises, you can’t fake away the fact when “stuff” doesn’t work or hide behind vast amounts of papers & documents void of any substance. Nope you are responsible for everything and anything you build. Aidan backed and supported by some very knowledgeable colleagues has made that burden a bit lighter for you to bear with this book. Aidan’s blog lives here: http://www.aidanfinn.com/

Cheap IT Support Requests & The Value of Time


I value my time tremendously. I also accept the fact that you don’t give a rat’s ass about my time. To you every hour I spend not working on your issues is a gigantic waste of time, but to me, it is not. And this is about my time. You cannot get time back, once used it’s gone. You cannot sell memories to get time back. You can’t produce time. You can’t save time. You have what you have and you need to use it when you have it. What does this have to do with IT? The fact than when you’re in IT people almost expect to get advice and support at no cost or on the cheap. This behavior stems from the fact that for some reason they expect that when they buy a server and software all the rest is included for free. With a car they don’t have this mind set. They expect to pay for maintenance, insurance, road taxes and gas. Partially this is the industries fault since they market everything a great, easy, fast, and cheap. Partially it’s the buyers fault for believing commercials and sales men. So how do I deal with the ever returning attempts to get me to work for free and how do I make sure they stop asking. Very simply, I price myself out of “the market”.

One day I discovered this also works outside of IT. Everyone who knows me wouldn’t think of asking me to baby sit but once a female colleague did just that. I guess she was really desperate. Really very desperate I should add. I told her I was not interested. She insisted. I told her again that I was absolutely not interested. She decided to make a case that I should help her out. That’s asking for it. I told her it would be 150 €/hour. I got a speech that babysitting isn’t that hard and worth that much, that it’s unaffordable, that kids, a house mortgage, car payments and life are already expensive enough. All true but not my problem.  You see I do not want to baby sit and my time is very valuable to me. I asked her what day and time she needed a baby sitter, trying to get my point across. She said Saturday night. Oh, on a Saturday to Sunday night in a weekend, after office hours and no retainer for my services. That makes it 300 €/hour and for watching of the most precious and loved human being in your life that’s a bargain! Needless to say I was not hired and luckily never asked again. Mission accomplished.

Think about it, time is the most limited resource the human species has. As I said, you can’t get any more of it. Gone is gone. That makes it more precious to me than anything else.  That means I want to spend it as well as I can. So when it comes to work I try to do things I enjoy and that pay well enough so that I can have enough free time to do other things I also enjoy. This means that when I do work I will not do it at 1 € /hour. Why would I? Even if I can only work 40% of my time at 5 €/hour I’m still way ahead and have more time to myself. With some luck and effort the better paying work is also the type of work I like to do. Cool, two goals achieved in one go.

So why on earth would I baby sit or fix your IT mess (which I dislike) in my spare time (time which is extremely valuable to be)  for some pocket money given the fact that it’s not my job responsibility and I have no financial pressure to do so?  Now I don’t know a thing about babies but IT can get a lot more complicated and involved that the owner of the mess realizes. It takes a lot of time and it just isn’t worth it. So there is your answer. I don’t want to and that’s why I price it so highly. To make sure no one asks or agrees to it. With these of hand support requests, changes are you’re a small shop running a couple of servers & workstations that are mediocre at best. This is probably combined with some older, hopefully legal, operating systems and applications that might suck and have their own issues. The environment was probably not designed, is most likely mismanaged for whatever reason and most of the time you won’t like the recommendations (get Adobe Acrobat and Office of your server and stop surfing on it so you don’t get spyware on the box). You complain about how expensive the hardware is, that the software costs money, that the small business IT shop is expensive and can’t get it right like you want. Perhaps the reason is that they can’t do it for the price your willing to pay, you are asking for things that can’t be done or perhaps they are not very good at their business. Whatever the reason, somehow you think that I should fix all that for a token fee since you already paid all that money to hardware vendors, software vendors, your “IT Guys” and because it won’t take me very long since I good at what I do. Well, it doesn’t work that way. My rate is not determined by how easy it might be for me. It’s determined by my knowledge, expertise and quality of my work. I don’t do the easier work as that won’t get me as much money for the same amount of time and I get bored doing it.

Am I a money hungry capitalist pig? No. I will and do work for free for a good cause, a close friend or a sport club I sympathize with. It’s called voluntarism and you can beat that as a motivation. I will not spend my valuable spare time fixing a mess that I did not create for free or cheaply. Actually I rather have my time to myself even when the money is good. You see, you’re in that mess because you don’t know what you’re doing; you’ve had very bad counseling or services and perhaps want things you can’t afford or are willing to pay for. The effort and cost of fixing all this is probably going to make you shout at me in anger. The impact that will have on your business processes and culture is something you’ll find unacceptable. The cost and needs of a professional IT environment are beyond what you can grasp, are willing or capable to pay. So the best thing for you is use free, cloud based services and make due with what you have or can get from those services. You cannot expect people to feel obligated to fix your problems because you already spent so much money on it. My free time at night and weekends is for studying, reading, hobbies, and friends. Not for fixing other peoples problems. So if you need a good environment hire one or more good IT partners to take care of your infrastructure needs in a professional manner. That’s the only sustainable and workable way of doing it.

The Dilbert® Life Series: Enterprise Architecture Revisited One Year Later


The Dilbert® Life series is a string of post on corporate culture from hell and dysfunctional organizations running wild. This can be quite shocking and sobering. The amount of damage that can be done by "merely" taking solid technology, methodologies, people and organizations, which you then abuse the hell out of, is amazing. A sense of humor will help when reading this. If you need to live in a sugar coated world were all is well and bliss and think all you do is close to godliness, stop reading right now and forget about the blog entries. It’s going to be dark. Pitch black at times actually, with a twist of humor, if you can laugh at yourself that is. And no, there is no light to shine on things, not even when you lite it. You see, pointing a beam in to the vast empty darkness of human nature doesn’t make you see anything. You do realize there is an endless, vast and cold emptiness out there. This is not unlike the cerebral content of way to many people I come across by in this crazy twilight zone called “the workplace”. I believe some US colleagues refer to those bio carbon life forms as “sheeple”.

Last year my very first blog post (http://workinghardinit.wordpress.com/2010/01/16/hello-world/) was about the one and only meeting I ever had with the Enterprise Architecture consultants that came in to help out at place where I do some IT Infrastructure Fu. Now one year, lots of time, money, training and Power Point slide decks after that meeting, the results on the terrain are nowhere to be seen. Sure there were lots of meetings, almost none of which I attended unless they dragged enterprise architecture into an IT related meeting on some other also vague action items like the IT strategy that was never heard of again. They’ve also created some new jobs specifications and lots of lip service and they’ll probably hire some more consultants to help out in 2011. But for now the interaction with and impact of any Enterprise Architecture on their IT infrastructure is nowhere to be found.

We put a good infrastructure plan in place for them. It’s pretty solid for 2011, pretty decent for 2012 and more like a road map for the time span 2013-2014. Meaning it’s flexible as in IT the world can change fast, very fast. But none of all this has come to be due to insights, needs, demands or guidance of any enterprise architecture, IT strategy or business plan.  No, it’s past experience and gut feeling, knowing the culture of the organization etc.  Creating strategies, building architectures is difficult enough in the best of circumstances. Combine this with fact that there is a bunch of higher pay grade roles up for grabs and the politics become very dominant. Higher pay grades baby? What do I need to get one? Skills and expertise in a very critical business area of cause!  Marketing yourself as a trusted business advisor, taught leader and architect becomes extremely important. As you can imagine getting the job done becomes a lot more difficult and not because of technical reasons. My predictions for 2011 are that by the end the year those pay grades will have been assigned. Together with a boatload of freshly minted middle management, who’ll be proud as hell and will need to assert their new found status, they’ll start handing out work to their staff.  Will that extra work materialize into results or only hold them back from making real progress? Well, we’ll need to wait for 2012 to know as 2011 will be about politics.

Basically from the IT infrastructure point of view and experience we have not yet seen an Enterprise Architecture and I don’t think they’ll have one in the next 12 months. Perhaps in 24 to 36 months but by then the game plan in IT infrastructure will be up and running. So realistically, I expect, if it leads anywhere against expectations, the impact of an Enterprise Architecture will be for 2014 and beyond. Which means an entirely new ball game and that will need a revised architecture. The success of the effort will no doubt be that they detected the need to change. This sounds uncomfortable similar to the IT strategy plan they had made. So for now we’ll do for them what’ we’ve always done. We’ll work with one year plans, two to three year roadmaps combined with a vision on how to improve the IT infrastructure. The most important thing is to stay clear of ambition and politics. Too much of that makes for bad technical decisions.

You got to love corporate bull. They don’t lie, no sir, they just sell bull crap. Which is worse, truth or lies don’t even matter, just the personal agendas. Liars at least, by the very fact of lying, acknowledge the value of truth, so much in fact, they’d rather have you not knowing it. Most consultancy firms send out kids that are naïve enough to believe the scripts and don’t even realize they are talking crap. They are told over and over again they are right, the best and they like to believe this so much they really do. It’s a bit like civil servants at the EU. Pay people double their market value, sweet talk their ego’s all day long and they will become prophets for the religion of the day. No, I’m not saying Enterprise Architecture is bull crap. I’m saying that way too many people & companies claiming to do enterprise architecture are turning it into exactly that. IT strategies, architectures that are so empty and void of content that all those binders are thrown in a drawer never to be seen again. A fool with a tool is still but a fool. Agile methodologies or tools don’t make your programmers agile gurus just like owning a race car doesn’t make you a race car pilot. All of this has happened before, and all of this will happen again. Every new, innovative process, methodology or concept falls victim to this. The money grabbing sales crowd gets there paws on it and starts selling it as competitive advantage or even innovation in a bottle to the corporate sheeple & management failures that should know better. They end with less money, loads of wasted time and a shitload of dead trees. As a side node, this whole “* Architect” thing  has runs it’s inflationary course. We need a new professional status currency once more. Take care and keep laughing clip_image001!

Exchange 2010 Public Folder Worries At Customer: No existing ‘PublicFolderProxyInformation’ matches the following Identity


A customers was recently using the EMC GUI in their Exchange 2010 environment, having a look a the public folder properties when they got this error:

—————————
Microsoft Exchange
—————————
Can’t log on to the Exchange Mailbox server ‘DAGMBX.demolab.com’. No existing ‘PublicFolderProxyInformation’ matches the following Identity: ‘\demolab\HeadQuarters\FincanceDepartment\FiscalUnit’. Make sure that you specified the correct ‘PublicFolderProxyInformation’ Identity and that you have the necessary permissions to view ‘PublicFolderProxyInformation’.. It was running the command ‘Get-MailPublicFolder -Identity ”\demolab\HeadQuarters\FincanceDepartment\FiscalUnit” -Server ‘DAGMBX.demolab.com”.
—————————
OK  
—————————

image

Hey … when did this start?  They never complained about this before, but did they ever use it.This probably was actually the first time they tried to look/edit the public folder permissions after doing the following over the past month and in this particular order:

  1. Moving to Exchange 2010 SP1
  2. Removing the last Exchange 2007 servers from the organization.

Now I know about a bug that exist and that was recently blogged about by Dan Rowley in Exchange 2010 get-mailpublicfolder \name returns No existing ‘PublicFolderProxyInformation’. The point is that there should be a mailbox database mounted on the server that has the System Attendant mailbox associated with it.  However, this is not the case here.  The mailbox servers are member of a DAG and all of them host a copy of the PF. The replication runs fine, users can work with them, the remaining Outlook 2003 users report no issues. But there is more in that blog: “Basically the work around is to mount a mailbox store on the server that is generating the error, or if there is a database already mounted – verify the system attendant is properly configured to point to a valid homemdb.” Now that last point is interesting and indeed that was the issue here. On two members of the DAG the homeMDB attribute was not set. Now what could be the root cause of this? I don’t know, certainly not in this case. All things have been done by the book … Ah well, luckily the fix is not very difficult. We need to put a valid entry in the homemdb. In this case we’ll take the value of the DAG member that had it filled in. This seems to be the most recently created database in the DAG. In Exchange 2010 this is done as described below. Note we have a DAG here, so we can work with any database that has a valid copy on the server(s) in question.

How to check the homeMDB attribute value:

  • Start ADSI Edit and navigate to CN=Configuration,DC=,DC=,DC=/Services/Microsoft Exchange//Administrative Groups/Exchange Administrative Group (FYDIBOHF23SPDLT)//Servers/MBXServerWithIssue
  • Right-click Microsoft System Attendant, and then click Properties to display the  Attributes list and find the homeMDB attribute.
  • If the homeMDB attribute has a value make sure  it points to a valid mailbox database. If the value of the homeMDB attribute is empty (not set) or incorrect you need to fix this.

image

How Fix the homeMDB attribute value:

  • In ADSI Edit navigate to Start ADSI Edit and navigate to CN=Configuration,DC=,DC=,DC=/Services/Microsoft Exchange//Administrative Groups/Exchange Administrative Group (FYDIBOHF23SPDLT)/Databases."
  • Right-click a mailbox database that is local (NON DAG) or has a valid copy on the server (DAG) , select Properties and in  the Attributes list, select the distinguishedName, and then click View.
  • Copy the value of the distinguishedName attribute and close the dialogs

image

NOTE in this particular case we can copy the value that was filled in the homeMDB attribute on one of the DAG members. You might not have one set in any.

  • Right-click Microsoft System Attendant, and then click Properties to get to the Attributes list, click homeMDB, and then choose Edit
  • In the Value box, paste the value that you copied form the distinguishedName attribute
  • Close the dialog boxes and exit ADSI Edit

When you’ve don this you’ll find following entry in the application event viewer:

Log Name:      Application

Source:        MSExchangeSA

Date:          11/2/2010 3:25:59 PM

Event ID:      9159

Task Category: General

Level:         Warning

Keywords:      Classic

User:          N/A

Computer:      DAGMBX.demolab.com

Description:

Microsoft Exchange System Attendant has detected that the system attendant object in the DS has been modified. System Attendant needs to restart the Microsoft Exchange Free Busy Publishing Service.

image

After that, I wait 10 minutes to get AD replicated and make sure to close the EMC and start it again and voila, it’s fixed.

No ADSI Edit required to fix “Object is read only because it was created by a future version of Exchange: 0.10 (14.0.100.0). Current supported version is 0.1 (8.0.535.0).”


During the removal of the last Exchange 2007 SP3 Mailbox server after completing the transition of Exchange 2007 to Exchange 2010 SP1 we ran into the following well known error: Object is read only because it was created by a future version of Exchange: 0.10 (14.0.100.0). Current supported version is 0.1 (8.0.535.0).

 

image

The issue is that due to the coexistence of Exchange 2007 & Exchange 2010 we can no longer remove the public folder database with the Exchange 2007 GUI (EMC). But the public folder is not visible in the Exchange 2010 GUI (EMC) as it lives on an Exchange 2007 server. Trying to remove the public folder database manually using the Exchange 2007 GUI confirms this, you’ll get the same error.

This error has been described in some blogs as early as October 2009 on http://www.proexchange.be/blogs/exchange2010/archive/2009/10/28/remove-exchange-2007-mailbox-role-fails-with-error-object-is-read-only-because-it-was-created-by-a-future-version-of-exchange-0-10-14-0-100-0-current-supported-version-is-0-1-8-0-535-0.aspx and later on as recently as October 2010 on http://www.howexchangeworks.com/2010/10/object-is-read-only-because-it-was.html

The described solution/work around in these blogs get the job done perfectly, using ADSI Edit to delete the offending Exchange 2007 public folder database. It wouldn’t be the first time ADSI Edit saves an Exchange Consultants proverbial bacon. But if it can be done without using it I often recommend not to do it.  I’ve seen to many over eager deletions in ADSI Edit get people into trouble (like deleting a public folder database before it could be dumped safely without data loss).

For this problem, it’s not required to use ADSI Edit to get rid of the public folder on the Exchange 2007 Mailbox server. You can just fire up the Exchange Command Shell (EMS) in Exchange 2010 and execute following PowerShell command:


Remove-PublicFolderDatabase "E2K7MBX\SGPublicFolders\StoreSGPublicFolders"

Confirm   4: Are you sure you want to perform this action?5: Removing public folder database "E2K7MBX\SGPublicFolders\StoreSGPublicFolders".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
WARNING: The specified database has been removed. You must remove the database file located in K:\E2K7Data\SGPublicFolders\PublicFolderDatabase.edb from your computer manually if it exists. Specified database: PublicFolderDatabase

This works just fine. I have no objections using ADSIEdit when needed but I don’t advise using it to others unless really necessary.In this case it just isn’t needed to fix the problem. For good measure I also deleted the storage group in which the public folder lived. After that the install went well end without any issues.

EMC Does Not Show All Database Copies After Upgrade To Exchange 2010 SP1– Still Investigating


LATEST UPDATE March 9th 2011: I have installed Exchange 2010 SP1 Rollup 3 at customer and this did indeed fix this issue finally.

Updates to this post are being added as we get them below. Last update was October 13th 2010. The have identified the cause of the issue. It’s a case sensitivity bug. The fix is WILL be contained in Exchange 2010 Sp1 Roll Up 3? But they ARE working on a incremental update in between. See below for more details and the link to the Microsoft blog entry.

At a customer we have a 3 node geographically dispersed DAG. This DAG has two nodes in the main data center and one in the recovery site in another city, but it is in the same AD Site. This works but is not ideal as DAC in Exchange 2010 RTM presumes that the node will be in another Active Directory site. As you can imagine at that location we’re very interested in Exchange 2010 SP1 since that adds support for the DAC to be used with a geographically dispersed DAG node in the same Active Directory site.

We did an upgrade to SP1 following the guidelines as published in http://technet.microsoft.com/en-us/library/bb629560.aspx and we made sure all prerequisites where satisfied. We upgraded the backup software to a version that supported Exchange 2010 SP1 and made sure no services that hold a lock on Exchange resources are running. The entire process went extremely well actually. We did have to reconfigure redirection for OWA as the SP1 installation resets the settings on the Default Web Site on the CAS Servers. But apart from that we had no major issues apart from one very annoying GUI problem. Everything was fully functional, which we verified using EMS and by testing failovers. But in the EMC GUI we had the problem under Organization / Mailbox / Database Management we only see the database copies listed on one server and not on all tree.

clip_image002

When you check the properties of the databases shows all three servers that are hosting copies. We used EMS commands to test for problems but it all checks out and works. Failing over a server works, both in the GUI and in PowerShell, just like activating a database.

The same issue can be seen in Server Configuration /Database Copies as demonstrated in the screenshots below. In the first figure you we selected the mailbox server where the database copies are visible.

clip_image004

But on the other two nodes nothing shows up, just “There are no items to show in this view”.

clip_image006

No errors in the vent logs or installation logs. All is working fine. So what gives? We tried all the usual suspects like throwing away any user related MMC cache information and cleaning out the Exchange specific information in the user profile up to deleting the profile etc. But nothing worked.

Running the script below, which is given to you by Microsoft to check your DAG before upgrading to SP1, confirms all is well.

(Get-DatabaseAvailabilityGroup -Identity (Get-MailboxServer -Identity $env:computername).DatabaseAvailabilityGroup).Servers | Test-MapiConnectivity | Sort Database | Format-Table -AutoSize

Get-MailboxDatabase | Sort Name | Get-MailboxDatabaseCopyStatus | Format-Table -AutoSize

function CopyCount

{

$DatabaseList = Get-MailboxDatabase | Sort Name

$DatabaseList | % {

$Results = $_ | Get-MailboxDatabaseCopyStatus

$Good = $Results | where { ($_.Status -eq "Mounted") -or ($_.Status -eq "Healthy") }

$_ | add-member NoteProperty "CopiesTotal" $Results.Count

$_ | add-member NoteProperty "CopiesFailed" ($Results.Count-$Good.Count)

}

$DatabaseList | sort copiesfailed -Descending | ft name,copiesTotal,copiesFailed -AutoSize

}

CopyCount

Searching the internet we find some folks who have the same problem. Also with a 3 node DAG that is geographically distributed. Is this a coincidence or is this related? http://social.technet.microsoft.com/Forums/en-US/exchange2010/thread/37d96c3d-433e-4447-b696-c0c00e257765/#5071f470-13cb-4256-8aa7-ade05bb4d67d. At first I taught it might have been related to the issue described in the following blog post http://blogs.technet.com/b/timmcmic/archive/2010/08/29/exchange-2010-sp1-error-when-adding-or-removing-a-mailbox-database-copy.aspx but in the lab we could not reproduce this. The only thing we managed to confirm is that you can delete the Dumpsterinfo registry key without any problem or nasty side effects. I’m still looking into this, but I’ll need to get Microsoft involved on this one.

Updates:

  • As an other test we created a new mailbox database and by the time we got the copies set up to the 3 nodes that brand new database and its copies showed the same behavior. For that new database the registry key Dumpsterinfo doesn’t even exist (yet?). So  That’s another nail in the coffin of the idea that behavior being related to the Dumpsterinfo key I guess.
  • Next test was that I added two static IP addresses to the DAG. One for each subnet in use. Until now we had a DCHP address and I noticed it was an address for the subnet of the node that is showing the database copies. I might as well give it a try right? But nope, that didn’t make a difference either. Still waiting for that call back from Microsoft Support.
  • Meanwhile I’m thinking, hey this DAG is only showing the database copies with the lowest preference (3). So I change the preference on a test database to 1 and refresh the EMC. No joy. This must really be just a GUI hiccup or bug. Now what would prevent the EMC GUI from displaying that information?
  • Some one on the newsgroup has the same issue with a 2 node DAG in the same subnet. So not related to a 3 Node geographically dispersed DAG.
  • MS Support got in touch. They have heard it before. But unless it was related to net logon errors they don’t have a cause or solutions. There are other cases and they will escalate my support call.

On September 27th 2010:

  • After a call from an MS support engineer to confirm the issue and pass on more feedback last week, we got an update via e-mail. After completing a code review and analysis they believe to have identified the problem.  They have also been able to reproduce the issue. More information is being gathered with reference customers to confirm the findings. More updates will follow hen they have more information on how to proceed. Indeed all is well with Exchange 2010 SP1 and PowerShell is your friend :-) Well progress is being made. That’s good.

On October 4th 2010:

We requested feedback today and tonight we got an e-mail with a link to a blog post confirming the issue and the cause. When the Exchange Management Console draws the database copies pane, it compares the host server name of a database copy to the server name of a database copy status.  This comparison is case sensitive and if they do not match up like in DAG-SERVER-1 <> Dag-Server-1 the database copies are not shown in the GUI. Again in EMS all works just fine. A fix is still in the make. You can find the Microsoft bug here: http://blogs.technet.com/b/timmcmic/archive/2010/10/04/database-copies-fail-to-display-after-upgrading-to-exchange-2010-service-pack-1.aspx

On October 10th:

I received another mail from Microsoft support just now. They expect this issue to be fully resolved in Exchange 2010 Service Pack 1 Rollup Update 3.  At this time they also intend to release an incremental update that corrects the issue. But this has some caveats.

1)  The incremental update would have to be applied to all servers where administrators would be utilizing the Exchange Management Console.  I think this is expected, like with most updates.

2) The incremental update cannot be applied with other incremental updates – for example if later an issue is encountered that is fixed in a different incremental update one would have to be removed prior to installing the second.  This can be a problem for people in that situation, so pick what is most important to you

3) The incremental update would only be valid for a particular Rollup Update.  For example, if the incremental update is installed for Exchange 2010 SP1 RU1, and you desire to go to Exchange 2010 SP1 RU2, you would have to contact Microsoft to have the incremental update built and released for Exchange 2010 SP1 RU2.  This may inadvertently delay the application of a rollup update.  Nothing new here, we’ve seen this before with interim fixes.

The workaround for customers not desiring to install an incremental update would be to continue using the Exchange Management Shell with the Get-MailboxDatabaseCopyStatus command. Nothing new here Smile

They have also updated their blog: http://blogs.technet.com/b/timmcmic/archive/2010/10/04/database-copies-fail-to-display-after-upgrading-to-exchange-2010-service-pack-1.aspx

I’m planning on keeping the case open in order to get my hands on the fix to test in the lab and have it for customers who so desire.

October 13th:

The fix WILL be included in Exchange 2010 SP1 Roll Up 3. They ARE working on the interim updates but this will take several weeks or longer.

 

Microsoft Project 2007-2010 Default Format When Saving Group Policy


For a roll out of Office 2010 we needed to set the default format when saving to Microsoft Project 2007 and not Microsoft Project 2010. This was the prevent any accidental file format issues with people still using Microsoft Project 2007. The rest of the Office 2007 formats such as Word, Excel, Visio, … have remained the same as in 2007 (in the case of Visio even the same as 2003).

We had already loaded the ADMX GPO templates for Office 2010 some and intend to set the default to Microsoft Project 2007. So we navigate to User Configuration => Policies => Administrative Templates: Policy definitions (ADMX files) retrieved from central store => Microsoft Project 2010

MSProject2010-1 (2)

Where we find following options:

MSProject2010-2

Funny we thought we’d find three options: Project 2010, Project 2007 and Project 2000-2003. Now what mpp format would that “Project” setting refer to? Microsoft Project 2007 or 2010? Well we checked the contents of the ADMX file proj14.admx which can be found, when using the central store that is, at:

\\blog.demo\SYSVOL\blog.demo\Policies\PolicyDefinitions\proj14.admx

There we see the following:

MSProject2010-3

Aha it’s the mpp format of Microsoft Project 2007 as indicated by MSProject.MPP.12. An entry for Microsoft Project 2010 doesn’t even exist. We quickly had a look at the proj12.admx (Microsoft Project 2007) and there the entries are exactly the same for that setting for “Save” policy.

It seems that even when the file formats is different between Microsoft Project 2007 and Microsoft Project 2010 they offer no way to easily identify what mpp format you’re setting it to in the GPO. Why they don’t offer Microsoft Project 2010 I don’t know. I guess their take is that you could leave it disabled in the GPO. Which is true but GPO setting are normally very unambiguous and you can explicitly set every option when you enable it, even the default when disabled. Perhaps they just forgot as they did not need this for the other office products in the Office 2010 release. I don’t know.

Anyway we tested this and indeed it set the default format for saving to Microsoft Project 2007. So all is well. Tip of the day: check the content of the ADMX file when in doubt.

Exchange 2007 & 2010 Event ID’s: 2601, 2604, 2501 & Users Can’t Access Mailboxes / Public Folders On My Day Off


I took the day off as I needed some time to deal with government administration. Good thing this is a blog about IT issues because holey crap what a time eating, confusing and rather pointless mess government administration can be. The process to get to the desired outcome is very tedious, prone to misunderstanding & pretty inefficient . What the entire duration of the process and the number of administrative entities involved contribute to the desired result is a mystery. It’s pure show and window dressing. But OK, we took the day of to finally get it all sorted after 5 months of patiently waiting for this day.

So I sleep until 08:00, get up and head for the kitchen for a jar of coffee. With the only Java I truly like in my hand I make my way to the home office. I check mails/alerts from System Center, Support Requests etc. I’m like a responsible guy dude, even when I need a day off. I do monitor the condition of my projects in production and I do step in when needed and document my findings. It keeps me honest when I design and sell my solutions. Beware of some architects that are not the ones having to deal with the crap architectures they design, they are often empty suits. Anyway, I see an issue that could be a warning of more to come. Someone has a problem with Outlook 2007 which reports the following error (translation from Dutch):

“Unable to expand the folder. The Microsoft Exchange Server computer is not available. Either there are network problems or the Microsoft Exchange Server computer is down for maintenance.(/o=<DOMAIN>/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=servers/cn=<dagmember1>)”

Now I know that user. Smart, diligent and reliable. That user even provides the relevant and necessary information in their support request. Yes they do exist and HRM should hire those exclusively. So in combination with that error we knew we did not have an PEBKAC or ID-10T on our hands but a real issue.

I quickly check that DAG member node Outlook of that user is trying to connect to but I know that due to maintenance their mailboxes currently reside on another member of the DAG. So i could very well be just the public folders. Bingo. A quick test reveals this to be the case. Also the Windows 2008 R2 server and Exchange 2010 itself are running perfectly fine, happy as can be, except on that one node we see the Application Event Log messages:

Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/19/2010 7:12:43 AM
Event ID:      2601
Task Category: General
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      dagmember1.company.blog
Description:
Process MSEXCHANGEADTOPOLOGY (PID=1620). When initializing a remote procedure call (RPC) to the Microsoft Exchange Active Directory Topology service, Exchange could not retrieve the SID for account <WKGUID=XXXXXXXXXXNOREALIDXXXXXXXXXXXXXX,CN=Microsoft Exchange,CN=Services,CN=Configuration,…> – Error code=8007077f. The Microsoft Exchange Active Directory Topology service will continue starting with limited permissions.

Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/19/2010 7:12:43 AM
Event ID:      2604
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      dagmember1.company.demo
Description:
Process MSEXCHANGEADTOPOLOGY (PID=1620). When updating security for a remote procedure call (RPC) access for the Microsoft Exchange Active Directory Topology service, Exchange could not retrieve the security descriptor for Exchange server object DAGMEMBER1 – Error code=8007077f. The Microsoft Exchange Active Directory Topology service will continue starting with limited permissions.

Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/19/2010 7:12:43 AM
Event ID:      2501
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      dagmember1.company.blog
Description:
Process MSEXCHANGEADTOPOLOGY (PID=1620). The site monitor API was unable to verify the site name for this Exchange computer – Call=DsctxGetContext Error code=8007077f. Make sure that Exchange server is correctly registered on the DNS server.

I think I’m OK when I see the possible cause. Why? Because I also know even if that probable cause isn’t the problem, it’s a hiccup I’ve seen before and I know how to fix its one. When you search those errors you can find a TechNet article describing a possible cause: “An inactive network connection is first on the binding list” http://technet.microsoft.com/en-us/library/dd789571(EXCHG.80).aspx. The fix is quite simple. Correct the NIC order and restart the MSExchange ADTopology Service. I had my scare about Active Directory and DNS horrors the first time I ever saw this one. So no gut wrenching panic here :-)

But why do servers ever get in to this state when the NIC ordering is just fine? We did some firmware and upgrade recently after hours but that didn’t affect the NIC binding order. Now I’m pretty weird at times but I still know what I’m doing. Those NIC where OK when I configured those servers. Checking that has become a second nature on multi homed and clustered servers. I also remember happening this to me once before somewhere in February 2010 with another setup of Exchange 2010 on Windows 2008 R2. And in that case the NIC order in the binding list was also OK. I checked back then as well just to make sure. But since I build those Exchange 2010 setups myself I just know they are close to godliness both in design and implementation :-). Back then the issue went away by restarting the server, restarting the MSExchange ADTopology Service will do however, and the problem never came back. For some reason the AD Site information query fails. Now Windows retries and is OK after a while. Exchange, tries to get the AD Site information once, fails and keeps thinking there is an issue. With as a result clients have no connectivity and those errors that initially make you think you could have DNS issues, AD problems etc. But fortunately it’s a lot less serious.

So when the NIC binding order is OK why does this happen? I can’t tell you for sure but I do know that I’m not the only one (not that weird after all) since Microsoft published KB Article “MSExchange ADAccess Event ID’s 2601, 2604, 2501” http://support.microsoft.com/kb/2025528 . This article is a so called FAST PUBLISH from Microsoft Support and states that the issue only occurs on Windows 2008 R2 and that it affect Exchange 2007 and Exchange 2010. The cause? Well this is where they provide only what I already knew:

“During a restart of the server, the operating system queries Active Directory to get its AD Site information.  On a Windows 2008 R2 server, this will sometimes fail.  As the Exchange services are starting, it also will do a query for its AD Site and that too will fail. Windows will continue to try and determine its AD Site name and will eventually succeed.  However, Exchange does not re-try the query and the above errors are logged in the application log every 15 minutes.”

And yes the workaround/fix is also nothing new:

“After the server has been up for a minute or two, run NLTest /DSGetSite to verify that that the proper Active Directory Site is being returned by Windows.  Once that has been verified, restart the MSExchange ADTopology Service.”

Do note that this will also restart a slew of dependant Exchange services so it takes a little while.

  • Microsoft Exchange Transport Log Search
  • Microsoft Exchange Transport Log
  • Microsoft Exchange Service Host
  • Microsoft Exchange Search Indexer
  • Microsoft Exchange Replication Service
  • Microsoft Exchange Mail Submission
  • Microsoft Exchange Mailbox Assistants
  • Microsoft Exchange File Distribution
  • Microsoft Exchange EdgeSync
  • Microsoft Exchange Anti-spam Update

So after some manual intervention we had the users back in business. And all is well for them, as they rise and sleep under the watchful eye of a bunch of good IT Pro’s who’ll protect them form further harm and problems ;-) Now I need to get an auto fix for this I think until Microsoft fixes this one for good. SCOM where are you? No, no, no … It’s my day off for getting that administration done!