New Version of ExFolders that is Exchange 2010 SP1 Compatible


I’ve mentioned the tool Exfolders before in http://workinghardinit.wordpress.com/2010/05/25/exchange-2007-2010-public-folders-issues-the-active-directory-user-wasnt-found/.   It’s a great tool and a worthy successor for PFDAVadmin.Now please note that when you upgrade to Exchange 2010 SP1 you’ll need to update the Exfolders tool as well. You can find the E2K10SP1 compatible version here: http://msexchangeteam.com/files/12/attachments/entry456255.aspx I’m happy to see that the new version of the tool is released in sync with the service pack. It’s a very handy an valuable tool to have. PFDAVadmin users already know this from experience. You can also use it to connect to an Exchange 2007 server but you need to run if on an Exchange 2010 server.

For more information about the tool take a look at this blog post by the Exchange Team: http://msexchangeteam.com/archive/2009/12/04/453399.aspx Don’t forget to read the instructions and follow them, especially regarding the import of the TurnOffSNVerificationForExFolders.reg file or the tool will crash.

Solving USB License Dongle Needs & COM Port Needs In Virtual Machines – RemoteFX USB Support Won’t Replace USB Over Ethernet


The Issue

One of the biggest showstoppers people come across when virtualizing servers is often software license dongles. A lot of software vendors have decided that such measures are needed to protect ‘m from piracy. I don’t think it works. I might prevent the occasional unlicensed use but it has never stopped a determined “pirate” from cracking & hacking the dongles or the software and offering that solutions on the dark side of the internet. The only cookies they have are poisonous ;-)

But short of all the stupid issues paying customers will have with license dongles, when it’s time to virtualize the servers you’re in trouble. There is no USB device on a virtual machine to stick that dongle into. Some quality desktop virtualization solutions like VMware workstation do offer USB support but the heavy duty server production hypervisors do not. So now there is an extra issue to solve. The solution to that problem can be solved USB over Ethernet software. When we start talking about physical COM port mapping (SMS Gateways anyone and other “esoteric” engineering toys …) it’s the same issue. But what’s discussed below is also valid for that problem. The companies have software for that as well.

The Solution

When you need to virtualize server applications that need a locally attached dongle you need to start looking for USB over Ethernet solutions that are reliable, support x64 bit, have signed drivers and support your Operating Systems. Normally you pay for the number of servers and USB device connections for which you get a license key. The client software is free and unlimited.

I’ve used both Fabulatech and KernelPro. But now I exclusively use KernelPro. FabulaTech version 4.1.1 was rock solid but ever since version 4.2/4.3 & 4.4 Beta they have brought us nothing but “Blue Screen of Death” hell either simply in production or during reboots (server grade hardware, DELL, HP). They really dropped the ball in a huge way It also had issues with clean uninstalls for upgrades etc. We just had to cut our losses after many months of frustration and troubleshooting as we were bleeding money and losing valued customers, not to mention our reputation. We now implement KernelPro USB over Ethernet version 2.5.5 as the solution of our choice. It works very well and we just never had an issue with that product. In the Hyper-V guest you install the client software and make sure to allow the needed ports in the client firewall configuration. For a license dongle you still need the dongle drivers to be installed but once that done you’re golden. To make sure server reboots do not interfere with the licensing I configure the USB over Ethernet software to automatically connect the correct USB device to the correct client. To prevent unwanted client connections you can use a IP filter list to allow or disallow client connections and disable USB devices for sharing. When using DNS names be aware of DNS Client Caching etc. when you change the IP address of a client and such. During fail over testing that might trip you over. I your application supports multiple dongles you can even create fault tolerant USB over Ethernet setups. Now try to sell that to a client: “Hello, I’m the architect of you highly available USB over Ethernet License Dongle solution for your Hyper-V farm”. Been there, done that!

Due Diligence

As I’ve written in a previous blog post “Perversions of IT: License Dongles” (http://workinghardinit.wordpress.com/2010/03/29/perversions-of-it-license-dongles ) you need to be aware of the fact that you need to manage the Ethernet over USB environment very carefully. Make sure the versions of the server software and the client software are the same on all nodes. We’ve seen network connectivity loss when the versions don’t match up, even if the software didn’t complain about different versions. Always make sure they are running the same version. Don’t just randomly update. For this reason I always disable the check for updates.

You need to test its stability, have some extra hardware and extra dongles for testing as to confirm your dongles respond well to this type of setup. We can’t afford to bring down production environments with USB over Ethernet software “upgrades of faith”. With some clients the license dongles become a virtualization show stoppers due to such risks. Talking about technology debt a cheap piece of plastic license protection is preventing paying customers from virtualizing. How do you think they feel about you as a software company. You’ll find out when they find an alternative without a dongle ;-)

RemoteFX USB Support to the rescue?

When Windows 2008 R2 SP1 was announced there was also a lot of talk about support for USB devices via RemoteFX USB . I was very happy with that because it’s a very useful and handy feature. Initially I was also optimistic about the fact that it might help with the license dongle issue. But as the name says, it very much geared to offer support for VDI environments. That means you’ll be able to hook up cameras, license dongles for interactive user apps so you can run them on your virtual desktop, USB rocket launchers (http://www.youtube.com/watch?v=EmZ-QKglyrc ) and the works.

Now for server applications there is no option provided for mapping the USB dongle on a parent partition to a child partition permanently. So basically for such needs , and in my neck of the woods I have a bunch of those, there is no built in solution. Software running as a service, License software running a couple of dongles and offering network licenses for applications will still need a USB over Ethernet solution. One little positive thing is that I heard rumors that one software vendor is dropping the need for a dongle. On small step but we’ll call that progress anyway. I hope the virtualization push forces a lot of others to do likewise and forget about license dongles. Typically for such companies that will be very late in the game. By the time they acknowledge the needs and realties in the field most customers have dumped that vendor or work around the problem on their own time and at their cost. So a vendor solution is always late. When they have it the industry progress has already led to another problem with their software due to bad choices they made in the past. Such is life. Still I have one advice for software companies: become good or get out!

EMC Does Not Show All Database Copies After Upgrade To Exchange 2010 SP1– Still Investigating


LATEST UPDATE March 9th 2011: I have installed Exchange 2010 SP1 Rollup 3 at customer and this did indeed fix this issue finally.

Updates to this post are being added as we get them below. Last update was October 13th 2010. The have identified the cause of the issue. It’s a case sensitivity bug. The fix is WILL be contained in Exchange 2010 Sp1 Roll Up 3? But they ARE working on a incremental update in between. See below for more details and the link to the Microsoft blog entry.

At a customer we have a 3 node geographically dispersed DAG. This DAG has two nodes in the main data center and one in the recovery site in another city, but it is in the same AD Site. This works but is not ideal as DAC in Exchange 2010 RTM presumes that the node will be in another Active Directory site. As you can imagine at that location we’re very interested in Exchange 2010 SP1 since that adds support for the DAC to be used with a geographically dispersed DAG node in the same Active Directory site.

We did an upgrade to SP1 following the guidelines as published in http://technet.microsoft.com/en-us/library/bb629560.aspx and we made sure all prerequisites where satisfied. We upgraded the backup software to a version that supported Exchange 2010 SP1 and made sure no services that hold a lock on Exchange resources are running. The entire process went extremely well actually. We did have to reconfigure redirection for OWA as the SP1 installation resets the settings on the Default Web Site on the CAS Servers. But apart from that we had no major issues apart from one very annoying GUI problem. Everything was fully functional, which we verified using EMS and by testing failovers. But in the EMC GUI we had the problem under Organization / Mailbox / Database Management we only see the database copies listed on one server and not on all tree.

clip_image002

When you check the properties of the databases shows all three servers that are hosting copies. We used EMS commands to test for problems but it all checks out and works. Failing over a server works, both in the GUI and in PowerShell, just like activating a database.

The same issue can be seen in Server Configuration /Database Copies as demonstrated in the screenshots below. In the first figure you we selected the mailbox server where the database copies are visible.

clip_image004

But on the other two nodes nothing shows up, just “There are no items to show in this view”.

clip_image006

No errors in the vent logs or installation logs. All is working fine. So what gives? We tried all the usual suspects like throwing away any user related MMC cache information and cleaning out the Exchange specific information in the user profile up to deleting the profile etc. But nothing worked.

Running the script below, which is given to you by Microsoft to check your DAG before upgrading to SP1, confirms all is well.

(Get-DatabaseAvailabilityGroup -Identity (Get-MailboxServer -Identity $env:computername).DatabaseAvailabilityGroup).Servers | Test-MapiConnectivity | Sort Database | Format-Table -AutoSize

Get-MailboxDatabase | Sort Name | Get-MailboxDatabaseCopyStatus | Format-Table -AutoSize

function CopyCount

{

$DatabaseList = Get-MailboxDatabase | Sort Name

$DatabaseList | % {

$Results = $_ | Get-MailboxDatabaseCopyStatus

$Good = $Results | where { ($_.Status -eq "Mounted") -or ($_.Status -eq "Healthy") }

$_ | add-member NoteProperty "CopiesTotal" $Results.Count

$_ | add-member NoteProperty "CopiesFailed" ($Results.Count-$Good.Count)

}

$DatabaseList | sort copiesfailed -Descending | ft name,copiesTotal,copiesFailed -AutoSize

}

CopyCount

Searching the internet we find some folks who have the same problem. Also with a 3 node DAG that is geographically distributed. Is this a coincidence or is this related? http://social.technet.microsoft.com/Forums/en-US/exchange2010/thread/37d96c3d-433e-4447-b696-c0c00e257765/#5071f470-13cb-4256-8aa7-ade05bb4d67d. At first I taught it might have been related to the issue described in the following blog post http://blogs.technet.com/b/timmcmic/archive/2010/08/29/exchange-2010-sp1-error-when-adding-or-removing-a-mailbox-database-copy.aspx but in the lab we could not reproduce this. The only thing we managed to confirm is that you can delete the Dumpsterinfo registry key without any problem or nasty side effects. I’m still looking into this, but I’ll need to get Microsoft involved on this one.

Updates:

  • As an other test we created a new mailbox database and by the time we got the copies set up to the 3 nodes that brand new database and its copies showed the same behavior. For that new database the registry key Dumpsterinfo doesn’t even exist (yet?). So  That’s another nail in the coffin of the idea that behavior being related to the Dumpsterinfo key I guess.
  • Next test was that I added two static IP addresses to the DAG. One for each subnet in use. Until now we had a DCHP address and I noticed it was an address for the subnet of the node that is showing the database copies. I might as well give it a try right? But nope, that didn’t make a difference either. Still waiting for that call back from Microsoft Support.
  • Meanwhile I’m thinking, hey this DAG is only showing the database copies with the lowest preference (3). So I change the preference on a test database to 1 and refresh the EMC. No joy. This must really be just a GUI hiccup or bug. Now what would prevent the EMC GUI from displaying that information?
  • Some one on the newsgroup has the same issue with a 2 node DAG in the same subnet. So not related to a 3 Node geographically dispersed DAG.
  • MS Support got in touch. They have heard it before. But unless it was related to net logon errors they don’t have a cause or solutions. There are other cases and they will escalate my support call.

On September 27th 2010:

  • After a call from an MS support engineer to confirm the issue and pass on more feedback last week, we got an update via e-mail. After completing a code review and analysis they believe to have identified the problem.  They have also been able to reproduce the issue. More information is being gathered with reference customers to confirm the findings. More updates will follow hen they have more information on how to proceed. Indeed all is well with Exchange 2010 SP1 and PowerShell is your friend :-) Well progress is being made. That’s good.

On October 4th 2010:

We requested feedback today and tonight we got an e-mail with a link to a blog post confirming the issue and the cause. When the Exchange Management Console draws the database copies pane, it compares the host server name of a database copy to the server name of a database copy status.  This comparison is case sensitive and if they do not match up like in DAG-SERVER-1 <> Dag-Server-1 the database copies are not shown in the GUI. Again in EMS all works just fine. A fix is still in the make. You can find the Microsoft bug here: http://blogs.technet.com/b/timmcmic/archive/2010/10/04/database-copies-fail-to-display-after-upgrading-to-exchange-2010-service-pack-1.aspx

On October 10th:

I received another mail from Microsoft support just now. They expect this issue to be fully resolved in Exchange 2010 Service Pack 1 Rollup Update 3.  At this time they also intend to release an incremental update that corrects the issue. But this has some caveats.

1)  The incremental update would have to be applied to all servers where administrators would be utilizing the Exchange Management Console.  I think this is expected, like with most updates.

2) The incremental update cannot be applied with other incremental updates – for example if later an issue is encountered that is fixed in a different incremental update one would have to be removed prior to installing the second.  This can be a problem for people in that situation, so pick what is most important to you

3) The incremental update would only be valid for a particular Rollup Update.  For example, if the incremental update is installed for Exchange 2010 SP1 RU1, and you desire to go to Exchange 2010 SP1 RU2, you would have to contact Microsoft to have the incremental update built and released for Exchange 2010 SP1 RU2.  This may inadvertently delay the application of a rollup update.  Nothing new here, we’ve seen this before with interim fixes.

The workaround for customers not desiring to install an incremental update would be to continue using the Exchange Management Shell with the Get-MailboxDatabaseCopyStatus command. Nothing new here Smile

They have also updated their blog: http://blogs.technet.com/b/timmcmic/archive/2010/10/04/database-copies-fail-to-display-after-upgrading-to-exchange-2010-service-pack-1.aspx

I’m planning on keeping the case open in order to get my hands on the fix to test in the lab and have it for customers who so desire.

October 13th:

The fix WILL be included in Exchange 2010 SP1 Roll Up 3. They ARE working on the interim updates but this will take several weeks or longer.

 

Heads Up! Serious Issue With Exchange 2010 SP1 When Using TMG and Edge Transport Role on Same Server = TMG in Mail Protection Role


Just a quick blog post to help spread the word and prevent some problems. http://blogs.technet.com/b/isablog/archive/2010/09/01/problems-when-installing-exchange-2010-service-pack-1-on-a-tmg-configured-for-mail-protection.aspx 

The root cause is the removal of the get-antispamupdates cmdlet in SP1. Microsoft is on the case and working hard to get a fix out as soon as possible. If you are effected be sure that Microsoft support will help you fast and efficient.

So please beware and hold of an upgrade to SP1 when your using this setup. A fix from the TMG team is forthcoming.