DELL Enterprise Forum EMEA 2014 in Frankfurt


As you might have noticed on Twitter I was in Frankfurt last week to attend DELL Enterprise Forum EMEA 2014. It was a great conference and very worthwhile going to. It was a week of multi way communication between vendor, marketing, engineering, partners and customers. I learned a lot. And I gave a lot of feedback. As a Dell TechCenter Rockstar and a Microsoft MVP in Hyper-V I can build bridges to make sure both worlds understand each other better and we, the customers get their needs served better.

Dell Enterprise Forum EMEA 2014 - Frankfurt

I’m happy I managed to go and I have some people to thank for me being able to grab this opportunity:

  • I cleared the time with my employer. This is great, this is a win win situation and I invested weekend time & extra hours for both my employer and myself.
  • I got an invite for the customer storage council where we learned a lot and got ample of opportunity to give honest and constructive feedback directly to the people that need to hear it! Awesome.
  • The DELL TechCenter Rockstar program invited me very generously to come over at zero cost for the Enterprise Forum. Which is great and helped my employer  and myself out. So, thank you so much for helping me attend. Does this color my judgment? 100%  pure objectivity does not exist but the ones who know me also know I communicate openly and directly. Look, I’ve never written positive reviews for money or kickbacks. I do not have sponsoring on my blog, even if that could help pay for conferences, travel expenses or lab equipment. Some say I should but for now I don’t. I speak my mind and I have been a long term DELL customer for some very good reasons. They deliver the best value for money with great support in a better way and model than others out there. I was sharing this info way before I became a Rockstar and they know that I tell the good, the bad and the ugly. They can handle it and know how to leverage feedback better than many out there.
  • Stijn Depril ( @sdepril, http://www.stijnsthoughts.be/), Technical Datacenter Sales at RealDolmen gave me a ride to Frankfurt and back home. Very nice of him and a big thank you for doing so.  He didn’t have to and I’m not a customer of them. Thank buddy, I appreciate it and it was interesting ton learn the partners view on things during the drive there and back. Techies will always be checking out gear …

Dell Enterprise Forum EMEA 2014 - Frankfurt

What did all this result in? Loads of discussion, learning and sharing about storage, networking, compute, cloud, futures and community in IT. It was an 18 hour per day technology fest in a very nice and well arranged fashion.

I was able to meet up with community members, twitter buddies, DELL Employees and peers from all over EMEA and share experiences, learn together, talk shop, provide feedback and left with a better understanding of the complexities and realities they deal with on their side.

Dell Enterprise Forum EMEA 2014 - Frankfurt

It has been time very well spent. I applaud DELL to make their engineers and product managers available for this event. I thank them for allowing us this amount of access to their brains from breakfast till the moment we say goodnight after a night cap. Well done, thank you for listening and I hope to continue the discussion. It’s great to be a DELL TechCenter Rockstar and work in this industry during this interesting times. To all the people I met again or for the first time, it was a great week of many interesting conversations!

For some more pictures and movies visit the Dell Enterprise Forum EMEA 2014 from Germany photo album on Flickr

Copy Cluster Roles Hyper-V Cluster Migration Fails at Final Step with error Virtual Machine Configuration ‘VM01′ failed to register the virtual machine with the virtual machine service


I was working on a migration of a nice two node Windows Server 2012 Hyper-V cluster to Windows Server 2012 R2. The cluster consist out of 2 DELL R610 servers and a DELL  MD3200 shared SAS disk array for the shared storage. It runs all the virtual machines with infrastructure roles etc. It’s a Cluster In A Box like set up. This has been doing just fine for 18 months but the need for features in Windows Server 2012 R2 became too much to resists. As the hardware needs to be recuperated and we have a maintenance windows we use the copy cluster roles scenario that we have used so many times before with great success. It’s the Perform an in-place migration involving only two servers scenario documented on TechNet and as described in one of my previous blogs Migrating a Hyper-V Cluster to Windows 2012 R2 for your convenience.

Virtual Machine Configuration ‘VM01′ failed to register the virtual machine with the virtual machine service

As the source host was running on Windows Server 2012 we could have done the live migration scenario but the down time would be minimal and there is a maintenance window. So we chose this path.

So we performed a good health check. of the source cluster and made sure we had no snapshots left hanging around. Yes it’s supported now for this migration scenario but I like to have as few moving parts as possible during a migration.

It all went smooth like silk. After shutting down the VMs on the source cluster node, bringing the CSV off line (and un-presenting the LUN from the source node for good measure), we present that LUN to the target host. We brought the CSV on line and when that was completed successfully we were ready to bring the virtual machines on line and that failed …

Log Name:      Microsoft-Windows-Hyper-V-High-Availability-Admin
Source:        Microsoft-Windows-Hyper-V-High-Availability
Date:          4/02/2014 19:26:41
Event ID:      21102
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      VM01.domain.be
Description:
‘Virtual Machine Configuration VM01′ failed to register the virtual machine with the virtual machine management service.

image

image

 

Let’s dive into the other event logs. On the host the application security and system event log are squeaky clean. The Hyper-V event logs are pretty empty or clean to except for these events in the Hyper-V-VMMS Admin log.

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Date:          4/02/2014 19:26:40
Event ID:      13000
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      VM01.domain.be
Description:
User ‘NT AUTHORITY\SYSTEM’ failed to create external configuration store at ‘C:\ClusterStorage\HyperVStorage\VM01′: The trust relationship between this workstation and the primary domain failed.. (0x800706FD)

 

image

Bingo. It must be the fact that no domain controller is available. It’s completely self contained cluster and both domain controller virtual machines are highly available and reside on the CSV. Now the CSV does come on line without a DC since Windows Server 2012 so that’s not the issue. it’s the process of registering the VMs that fails without a DC in an Active Directory environment.

Getting passed this issue

There are multiple ways to resolve this and move ahead with our cluster migration. As the environment is still fully functional on the source cluster I just removed a DC virtual machine from high availability on the cluster. I shut it down and exported it. I than copied it over to the node of the new cluster  (we’re going to nuke the source host afterwards and install W2K12R2, so we moved it to the new host where it could stay) where I put it on local storage and imported it. For this is used the “Register the virtual machine in-place option”. I did not make it high available.

image

After verifying that we could ping the DC and it was up and running well we tried the final phase of the migration again. It went as smooth as we have come to expect!

Other options would have been to host the DC virtual machine on a laptop or other server. If you could no longer get to the the DC for export & import or heck even a shared nothing migration depending on your environment can help you out of this pickle. A restore from backup would also work. But here in that 2 node all in one cluster our approach was fast and efficient.

So there you go. Tip to remember. Virtualizing domain controllers is fully supported, no worries there but you need to make sure that if you have a dependency on a DC you don’t have the DC depending on that dependency. It’s chicken an egg thing.

Use Cases For Fluid Cache For SAN With DELL Compellent In High Performance Virtualization With Windows 2012 R2


Fluid Cache For SAN

At Dell World 2013 in Austin Texas I spent some time talking to engineers & managers about Fluid Cache For SAN. The demo in the keynote was enough to grab my distinct attention, especially as a Compellent customer.

What is it?

Dell already has Fluid Cache for DAS available in its PowerEdge servers. Now it’s time to bring this to their best SAN offering, the Compellent, and make Fluid Cache shared storage suitable for shared storage clustering. The way to do that cost effective and high performance is to build on the success of on board (local to the server) high performance storage and make that shared through software in a physical shared nothing replication/sync model. To make this happen they use a 10/40Gbps Ethernet solution leveraging RoCE (RDMA over Converged Ethernet). Yes that very technology I have been investing time & effort in for SMB Direct and which we leverage for CSV & Live Migration traffic and with SOFS in Windows 2012 R2.

Basically the super low latency an high throughput enable the memory to be synced across all nodes in a cluster and as such each node sees all the cluster memory. For redundancy you will need at least 3 nodes in a cluster. Dell will scale Fluid Cache For SAN to 128 nodes. Windows Server 2012 R2 can handle 64 nodes, which some think is ridiculously high, but then again, Dell aims even higher so it’s not as weird as you think. Some people have really huge computing needs. Just remember that 10 years ago you probably found that 16GB of RAM was extravagant.

Why this architecture?

Dell uses server based “shared nothing flash storage” & high speed low, latency synchronization to create a logical cluster wide shared pool of flash memory. This means the achieve stellar low latency as the flash storage is inside of the servers, close to the processors and as such delivers excellent performance for the workloads. Way better that “just” flash only SAN can. For data integrity they commit the data only when it written to the Express Flash drive(s) of one server and then also to another and verified. This needs to happen very fast and that where the RoCE network come sin to play. Later, at less speed critical times the data is pushed out to the Compellent SAN for storage. If that SAN is a flash based setup think about the capabilities this gives you in performance. Likewise data reads of the SAN that are highly active are pushed from the Compellent SAN and cached (also in multiple copies) on the Express Flash modules. While two servers with each a copy of the data on Express Flash modules would suffice DELL requires at least three. This is just a plain common sense N+1 redundancy design to have high availability even when a node fails. A cool think to note is that you can build larger clusters with 3 nodes each having one or more Express Flash modules and additional nodes don’t need it as long as they can read the cache of those 3. So the cost of this can be managed. The drawback is that you don’t read & write to a local Express Flash module on those extra node. If you want that you’ll need to put more $ on the table.

clip_image001

The thing to note here is that the Servers/SAN are connected over RoCE/RDMA. Well this look familiar. What technology can also leverages RDMA? SMB Direct in Windows Server 2012 R2! And where do we use this amongst other things? Storage IO in Scale Out File Server, CSV traffic, Live Migration …

The big benefit of this design it just it takes your SAN to the next level but also, if DELL does this right, they won’t break any of the good stuff like VSS aware snapshot with Replay Manager, Automatic Data Tiering, Live Volumes, Live Migration etc. A lot of the high IOPS/low latency solutions out their based on fast local flash break a lot of the good stuff and reduces centralized storage management. What if you can have your cookie and eat it to?

Demo Time at Dell World

Dell demonstrated an Oracle database load on an eight node cluster of PowerEdge R720 servers with Intel Xeon E5 processors, with Linux (no Windows Server 2012 R2 support yet Sad smile) These servers each used 350GB PCI-Express flash cards (“only” PCI-Express 2.0 capable by the way). This cluster, using a Compellent SAN, managed to get a result of more than 5 million IOPS at 6 millisecond response times, delivering 12,000 tps for 14,000 client connections. This was read only. If they dropped the Fluid Cache for SAN they  could “only” achieve 2,000 clients (6 times less clients due to 4 time less transactions and 99% slower responses). See this movie for more info: http://www.youtube.com/watch?v=uw7UHWWAtig and watch the keynote from Dell World 2013 here

clip_image003

Where would I use this?

Cost will determine use cases and this is unknown for now. We can only look at what Fluid Cache for DAS cost right now and speculate. I for one hope/bet on the fact that DELL won’t price itself out of the marked (they have a lot of competition from big & smalls players in a “good enough is good enough” world with a cloud mindset all around). So make it too expensive and we might be happy with “just” 500.000 IOPS at much less cost. It’s a fine line. Price it right, support it well and you might win the bulk of sales in the storage wars. Based on the DAS solution we’re looking at least 8000 $ per server (license is 3500 for DAS => see http://www.theregister.co.uk/2013/03/05/dell_fluid_cache_server_acceleration/  + cost of PCI-Express flash module (> 5000$ => see http://en.community.dell.com/techcenter/extras/chats/w/wiki/4480.3-5-2013-techchat-fluid-cache-for-das.aspx) &  yearly maintenance fee. Then we need to factor in the cost of the RDMA/RoCE capable NICs & the (dedicated) Force10 switches – 2 for redundancy  that are at least 10Gbps (S4810?) or probably 40Gbps (S6000?) & cabling. So this is not a cheap solution and you won’t just “throw it in” on a quiet afternoon to see what it does for you. Not that there will be a DIY “throw it on kit” I think, it’s a step above plug and play. If they keep it affordable and do some other things for Windows Server 201 R2 / Hyper-V they can be the absolute number one SAN vendor for any Microsoft customer. But that’s another blog topic.

Cost is indeed something that might make it a show stopper for us. I just can’t tell yet. One of the key factors is that if affordable it could give the point solutions we now see pop up more and more in storage. a run for their money. While cheap and workable in good enough is good enough scenarios it takes some of the centrally shared storage advantages away. But if we ever do a state full VDI project in an environment with high end physical desktops (500GB or more local storage, SSD disks, 8 core CPU, 8-32GB DDR3, dual or more screens) that run ArcGis, AutoCad, Visual Studio, SQL Server, Outlook with 5GB mailboxes, large documents & huge files (images) this might be the enabler we need to make VDI happen & works as desired with current all-purpose Compellent SANs. IIf the price is right it could enable VDI in now “NO GO” scenarios.  And those are plentiful, … Another use case I see is a virtualized SQL Server environment on Hyper-V with general purpose shared storage. We’re doing very well but the day might arrive that we need those IOPS in order to take it even further. Don’t laugh but realize how much IOPS an SSD delivers to a workstation today and that’s what your users expect & demand. Want to fail at VDI? Have it outperformed by a 4 year old physical PC where you slapped an SSD into.

Could it help in keeping excessive IOPS away from the SAN, making that capable of doing more over a longer life time? In other words can it play a part in the Storage QoS issue across server/cluster/storage system issue for non workload aware storage solutions?

So I might have some homework to do. For our next SQL Server cluster we’ll look at the next generation of servers & start counting our PCI Express slots. We now already consume 4 PCI-Express slots for 2*FC & 2*Dual Port 10Gbps) in our Hyper-V design. That’s another discussion, but they are built purposely for performance under any condition & to be highly redundant. A health check / improvement track by Microsoft for our SQL Server environment has proven this to be an outstanding setup (nice e-mail to see your bosses get by the way). I digress, free PCI-Slots should not be an issue, as we also don’t need the FC cards in the Fluid Cache Nodes. The storage IO uses the RoCE network, to which the Compellent SAN attaches.

Cost is very important in determining if we’ll ever get to deploy it. The cloud is here, and while that is far from cheap either, it’s a lot easier to sell than internal IT for various reasons. That’s just how the powers that be roll right now & how things are.

What we’ll get in our hands

There was a lot of love between Dell & Samsung at Dell World. Talking to Dell at the server/storage/networking boots I understood that Samsung is going to produce flash modules for this that support PCI-Express 3.0 and the industry backed NVM Express host interface for solid state drives which will reduce latency with 1/3 compared to now. As it seems they will produce higher capacity cards than what was used in the demos (800 GB and 1.6 TB). So capacity will increase & latency will drop even more. They leverage the Force10 10Gpbs or 40Gbps switches for the RoCE network. As Dell & Mellanox are cooperating heavily (Mellanox Collaborates with Dell to Deliver 10/40GbE Solution for Mainstream Servers and Networking Solutions) my bet is on Mellanox for the cards. Broadcom is not there yet for it to happen in time and Intel has no RoCE cards afaik. They seem to be playing the waiting game before they jump in.

Magic Ball Time, Speculation & Questions.

I’m not a DELL Server / Storage designer or architect, and those that are don’t tell me to plaster it all over the internet, so this really is magic ball time …

image

I’ll show my ignorance on what Samsung does under the hood when I hear that the next generation of DELL servers can have 6TB of RAM I can only speculate that with the advent of DDR4 in servers & ever dropping cost the path is open to leverage NV-RAM disk for the read/write cache in Fluid Cache for SAN as well a bit like what IDT did http://us.generation-nt.com/idt-announces-world-first-pci-express-gen-3-nvme-nv-dram-press-3732872.html. The persistence comes from writing the DRAM content to NAND at shutdown, can we do that fast enough at 1.6 TB sized caches? Can we fit enough of  those modules on a card? What would that do for IOPS & latency? Does that even make sense at this moment in time?

What if we could leverage the DDR4 dims in the server itself? This would perhaps cut costs and also save us some valuable PCI-Express 3.0 slots for our 10Gbps or better addiction Smile. Sure there is no persistence than but the content is distributed redundantly over the cluster anyway? Is that safe enough to make it feasible? What if we need to shut down the cluster? I guess it’s not that easy and perhaps we just need to make sure future motherboards have 8 or more PC-Express 3.0 slots & not worry about that. Or move to 40/100Gbps & have less need for NICs. Yeah that’s what was said of 10Gbps in the early days …

Support for Windows?

While it’s not there yet I have absolutely no doubt that they will bring it to Windows Server 2012 R2 and higher. Well Windows is a huge on premise market for native workloads like SQL Server, VDI and Hyper-V. The number of sales opportunities in the Microsoft ecosystem is growing (despite cloud) while others are stagnant or dropping. On top of that the low cost of Hyper-V leaves money to be spent on Fluid Cache for SAN. As Dell is in business to make money, they will not leave that big chunk of cash on the table.

When can we get our hands on this technology?

Timing wise that will be early to late Q2 in 2014, which is my best guestimate. Interesting times people, interesting times

DELL World 2013 – Tour Of the Acoustic & Storage Testing Labs & Presenting at the Dell TechCenter User Group


While at Dell World 2013, a group of us had the opportunity to visit the Dell offices as part of the Trends in Data Center Technology Think Tank. We saw advancements in fresh air cooling, a hot house,

image

the storage lab and, new to me, the acoustic labs. Below is a picture of Chris Peterson, the acoustic Architect (he was involved in the design of the DELL VRTX, which is a unique solution and achievement in the industry). Like wise the also have thermal engineers and both of these expertises are closely related.

I will never look at acoustic / thermal engineering for servers & storage in the same way I used to and I have way more respect for the effort and a better understanding of what efforts go in to this research and why.

For some more information on the acoustic lab read this white paper Dell Enterprise Acoustics and watch these videos:

Dell thermal & acoustic engineers discussing the VRTX
Chris Peterson on Dell PowerEdge Generation 12 Server acoustics

Next to all that I attended briefings, had one to one conversations with network, storage & server managers & engineers. I had a lot of information, questions & request to share from our Microsoft MVP Community in regards to our needs & wishes for the best possible support for Windows Server 2012 R2, Hyper-V, ODX, UNMAP, SMB Direct, SOFS, Management & cloud. I even jumped into an open source breakfast discussion on * cloud computing. Last but not least we joined fellow Rock Stars Jonathan Copeland (@VirtSecurity), Rasmus Haslund (@haslund) & Dell Tech Center’s community manager Jeff Sullivan (@JeffSullivan) to discuss what community & social media means to us.

image

I also shared our experiences with Windows Server 2012 R2, Hyper-V, DVMQ, vRSS & ODX at the Dell Tech Center User Group during Dell World 2013.

image

Want to talk and demo DVMQ & vRSS? Start with the basics: RSS Smile 

To all my community buddies a very festive end of the year and a great 2014! If you want to know even more about how rewarding being part of a community can be, check out this blog Mindset of the community by Marc van Eijk (@_marcvaneijk)

I’m In Austin Texas For Dell World 2013


This is the night time sky line of where I’m at right now. Austin, Texas, USA. That famous “Lone Star State” that until now I only knew from the movies & the media. Austin is an impressive city in an impressive state and, as most US experiences I’ve had, isn’t comparable with anything in my home country Belgium. That works both ways naturally and I’m lucky I get to travel a bit and see a small part of the world.image

Dell World 2013

So why am I here?  Well I’m here to attend DELL World 2013, but you got that already Smile

image

That’s nice Didier but why DELL World? Well, several reasons. For one, I wanted to come and talk to as many product owners & managers, architects & strategists as I can. We’re seeing a lot of interest in new capabilities that Windows Server 2012 (R2) brought to the Microsoft ecosystem. I want to provide all the feedback I can on what I as a customer, a Microsoft MVP and technologist expect from DELL to help us make the most of those. I’m convinced DELL has everything we need but can use some guidance on what to add or enhance. It would be great to get our priorities and those of DELL aligned. Form them I expect to hear their plans, ideas, opinions and see how those match up. Dell has a cost/value leadership position when it comes to servers right now. They have a great line up of economy switches that pack a punch (PowerConnect) & some state of the art network gear with Force10. it would be nice to align these with guidance & capabilities to leverage SMB Direct and NVGRE network virtualization. Dell still has the chance to fill some gaps way better than others have. A decent Hyper-V network virtualization gateway that doesn’t cost your two first born children and can handle dozens to hundreds of virtual networks comes to mind. That and real life guidance on several SMB Direct with DCB configuration guidance. Storage wise, the MD series, Equalogic & Compellent arrays offer great value for money. But we need to address the needs & interest that SMB 3.0, Storage Spaces, RDMA has awoken and how Dell is planning to address those. I also think that OEMs need to pick up pace & change some of their priorities when it comes to providing answers to what their customers in the MSFT ecosystem ask for & need, doing that can put them in a very good position versus their competitors. But I have no illusions about my place in & impact on the universe.

Secondly, I was invited to come. As it turns out DELL has the same keen interest in talking to people who are in the trenches using their equipment to build solutions that address real life needs in a economical feasible way.  No, this is not “just” marketing. A smart vendor today communicates in many ways with existing & potential customers. Social media is a big part of that but also off line at conferences, events and both contributor and sponsor.  Feedback on how that works & is received is valuable as well for both parties. They learn what works &n doesn’t and we get the content we need. Now sure you’ll have the corporate social media types that are bound by legal & marketing constrictions but the real value lies in engaging with your customers & partners about their real technological challenges & needs.

Third is the fact that all these trends & capabilities in both the Microsoft ecosystem and in hardware world are not happening in isolation. They are happening in a world dominated by cloud computing in all it’s forms. This impact everything from the clients, servers, servers to the data centers as well as the people involved. It’s a world in which we need to balance the existing and future needs with a mixture of approaches & where no one size fits all even if the solutions come via commodity products & services. It’s a world where the hardware  & software giants are entering each others turf. That’s bound to cause some sparks Smile. Datacenter abstraction layer, Software Defined “anything” (storage, networking, …), converged infrastructure. Will they collaborate or fight?

So put these three together and here I am. My agenda is full of meetings, think tanks, panels, briefings and some down time to chat to colleagues & DELL employees alike.

Why & How?

Some time ago I was asked why I do this and why I’m even capable to do this. It takes time, money and effort.  Am I some kind of hot shot manager or visionary guru? No, not at all. Believe there’s nothing “hot” about working on a business down issue at zero dark thirty. I’m a technologist. I’m where the buck stops. I need things to work. So I deal in realities not fantasies. I don’t sell methods, processes or services people, I sell results, that’s what pays my bills long term. But I do dream and I try to turn those into realities. That’s different from just fantasy world where reality is an unwelcome guest. I’m no visionary, I’m no guru. I’m a hard working IT Pro (hence the blog title and twitter handle) who realizes all to well he’s standing on the shoulders of not just giants but of all those people who create the ecosystem in which I work. But there’s more. Being a mere technologist only gets you that far. I also give architectural & strategic advice as that’s also needed to make the correct decisions. Solutions don’t exist in isolation and need to be made in relation to trends, strategies and needs. That takes insight & vision. Those don’t come to you by only working in the data center, your desktop or in eternal meetings with the same people in the same place. My peers, employers and clients actively support this for the benefit of our business, customers, networks & communities. That’s the what, why and who that are giving me the opportunities to learn & grow both personally & professionally. People like Arlindo Alves and may others at MSFT, my fellow MVPs (Aidan Finn, Hans Vredevoort, Carsten Rachfahl, …), Florian Klaffenbach & Peter Tsai. As a person you have to grab those opportunities. If you want to be heard you need to communicate. People listen and if the discussions and subjects are interesting it becomes a two way conversation and a great learning experience. As with all networking and community endeavors you need to put in the effort to reap the rewards in the form of information, insights and knowledge you can leverage for both your own needs as well as for those in your network. That means speaking your mind. Being honest and open, even if at times you’re wrong. That’s learning. That, to me, is what being in the DELL TechCenter Rock StarDELL TechCenter Rock Star program is all about.

Learning, growing, sharing. That and a sustained effort in your own development slowly but surely makes you an “expert”. An expert that realizes all to well how much he doesn’t known & cannot possible all learn.  Luckily, to help deal with that fact, you can turn to the community.

Upgrading The DELL PowerVault MD3600 Disk Firmware


In this post I’ll walk you through an disk firmware upgrade of a PowerVault MD3600F with 7 MD1200 extension bays filled with disks. We have several of these attached to R710 that act as Disk2Disk backup media servers with 10Gpbs networking and running W2K12R2 of course. At the moment of writing we’re at PowerVault MD Series Storage HDD (SSD) Firmware A13 and in our case we’ll be updating to Seagate_ST2000MN0001_PS08.

First of all read the readme.txt and such to make sure you’re not missing any special instructions for particular cases that might exist such as being on to old controller firmware that you need to upgrade first. Do your home work & due diligence. Of course I’m not responsible for whatever it is you do to your environment. The standard disclaimer of my blog applies Smile

I always make sure my controller firmware is up to date before I don disk firmware updates unless I’m instructed to act otherwise. In general you should not be applying firmware updates and such constantly and  all over the place to keep busy. It is, however, a good practice to keep an eye on releases and see if they fix any risky bugs you might be vulnerable to.  You also don’t get to far behind on firmware updates as this can complicate thing later in the useful service life or you hardware. Fear, doubt and only doing do something when things have broken down are also not a good practice.

So first of all get the latest version of your PowerVault Modular Disk Storage Manager software if you haven’t already. Often you’ll need to have a least a certain version before you can even do newer firmware updates. So just get this in order before starting, especially if they tell you to, don’t try and outsmart the system. Then download the disk firmware form the DELL support website or via the DELL Storage Community Wiki http://en.community.dell.com/techcenter/storage/w/wiki/4234.dell-powervault-md-downloads.aspx?dgc=SM&cid=257966&lid=4630585 and safe it to disk.

Fire up the PowerVault Modular Disk Storage Manager by clicking the icon …image

Enjoy the splash screen during the launch …image

Right click your storage array and select “Manage Storage Array”image

On the menu click Upgrade and select “Physical Disk Firmware…”image

A wizard pops up with some friendly advice and you are warned to stop all I/O. Which  is something you should really do. If you can un mount the file system(s) – off line it in Windows – you can do this to makes sure no I/On kick of during the upgrade. Depending on your workload this might require a maintenance window or not. For us just setting the backups to not run, stopping the agents/services of the backup product on these media servers and taking the storage LUNS off line is enough.image

When you have done so click Next. As you can see I have only one disk type in my MD3600F and its MD1200 extension disk bays and they are all on the same firmware.

image

Click on Add and browse to the location where you unzipped the disk firmware package. There’s a bunch of them but it should be straight forward to select the correct one. If not perhaps you shouldn’t be doing this Smile

image

In this example you see on disk type all being moved to one new firmware. If you have multiple disk types or brands you can select multiple firmware packages to be transferred. That’s quite handy.

image

When done click Next to continue. Note: Don’t select more that one version of the firmware for the same type of disk. That will just throw an error message at you telling you that’s not you smartest move.image

You can select all hard disk to deploy the firmware. I use this but I tend to test a run on one disk (a hot spare) first before I do this. If that one is successful I choose “Select All” click finish.

image

To make sure that you know what you’re doing and probably keep any lawyers at bay you must type in yes before you can continue. Yes making backups is always prudent. Always have at least one way out.

image

You can follow the firmware update process in the right hand column which displays the progress (Not attempted, in progress,Successful).

image

This process goes faster than you might think. When they are all done, close the wizard. Congratulations you have just installed the new disk firmware to 96 disks Smile.image

If, for 100% safety, you have brought your storage off line, you can it online and resume normal operations. I tend to give the hosts a reboot to see all still works as it should.

Upgrading The DELL MD3600F Controller Firmware Using the Modular Disk Storage Manager


As part of our hardware maintenance we’re deploying the updates of SUU 7.3.1 right now. As part of that effort we’re also making sure the PowerVault disk bays are getting their updates. We’ve got a couple of PowerVault MD3600F with 7 MD1200 extension bays filled with disks attached to R710  Power Edge servers that act as Disk2Disk backup media servers with 10Gpbs networking and running W2K12R2.

In general you should not be applying firmware updates and such constantly and all over the place just to keep busy. It is, however, a good practice to keep an eye on releases and see if they fix any risky bugs you might be vulnerable to. You see that sometimes these fix issues you don’t want to run into and those are often marked urgent. You also don’t get to far behind on firmware updates as this can complicate thing later in the useful service life or you hardware. First of all read the readme.txt and such to make sure you’re not missing any special instructions for particular cases that might exist. The standard disclaimer of my blog applies Smile, you’re responsible for your own actions.

First of all get the latest version of your PowerVault Modular Disk Storage Manager software if you haven’t already. Often you’ll need to have a least a certain version before you can even do newer firmware updates. So just get this in order before starting, especially if they tell you to, don’t try and outsmart the system. Download the disk firmware form the DELL support website or via the DELL Storage Community Wiki http://en.community.dell.com/techcenter/storage/w/wiki/4234.dell-powervault-md-downloads.aspx?dgc=SM&cid=257966&lid=4630585 and safe it to disk.

Here’s an example of a read me file you’d better not ignore. If you do, the Modular Disk Storage Manager will not allow you to upgrade anyway.

*** ATTENTION ***
If your PowerVault MD32xx/MD36xx series storage array is currently running a firmware version lower than 07.75.28.60, you cannot directly install the RAID controller firmware version 07.84.47.60 included in this release.
Instead, you must first install the bridge firmware version 07.75.28.60. After the bridge firmware is successfully installed, you can then install RAID controller firmware 07.84.47.60.

Specifically, follow these steps:
1. *** IMPORTANT *** Install Dell PowerVault MD Storage Manager (MDSM) software from the Resource DVD version 4.1.2.29 or higher.
2. Extract the following files from the Bridge_Firmware_07_75_28_60.zip archive:
* Bridge_Firmware_07_75_28_60.dlp
3. Download and activate RAID controller bridge firmware version 07.75.28.60 (contained in the Bridge_Firmware_07_75_28_60.dlp file)
4. Once the bridge firmware installation is successfully completed, wait at least 30 minutes to allow the attached hosts to re-discover the storage array.
5. Download and activate the RAID controller firmware version 07.84.47.60 (contained in the MD3xxx_MD3xxx_Firmware_07_84_47_60.dlp file) together with the NVSRAM configuration file version N26X0-784890-X04 (contained in the MD3xxx_MD3xxx_NVSRAM_N26X0-784890-X04.dlp file) using the Dell PowerVault MD Storage Manager for your corresponding array type.
* MD3200_MD3220_Firmware_07_84_47_60.dlp and MD3200_MD3220_N26X0_784890_004.dlp (6G Non-Dense SAS)
* MD3200i_MD3220i_Firmware_07_84_47_60.dlp and MD3200i_MD3220i_N26X0_784890_004.dlp (1G Non-Dense iSCSI)

* MD3600i_MD3620i_Firmware_07_84_47_60.dlp and MD3600i_MD3620i_N26X0_784890_904.dlp (10G Non-Dense iSCSI)
* MD3600f_MD3620f_Firmware_07_84_47_60.dlp and MD3600f_MD3620f_N26X0_784890_904.dlp (8G Non-Dense FC)
* MD3260_Firmware_07_84_47_60.dlp and MD3260_N26X0_784890_004.dlp (6G Dense SAS)
* MD3260i_Firmware_07_84_47_60.dlp and MD3260i_N26X0_784890_004.dlp (1G Dense iSCSI)
* MD3660i_Firmware_07_84_47_60.dlp and MD3660i_N26X0_784890_904.dlp (10G Dense iSCSI)
* MD3660f_Firmware_07_84_47_60.dlp and MD3660f_N26X0_784890_904.dlp (8G Dense FC)

*** WARNING ***
If you have a single controller PowerVault MD32/MD36 series storage array you must stop all I/O operations before starting the RAID controller firmware upgrade.

Launch the PowerVault Modular Disk Storage Manager, right click your array and select Upgrade Raid Controller Module Firmware.image

Wait while the status is being refreshed.Image 017

You might run into a an issue if the event log contains to many entries. In that case you’ll be warned you can’t upgrade until you’ve cleared it as shown below. Here are some screenshots from another previous upgrade where I ran into this issue.Image 018

In that case open up the Manage Storage Array  for your array.image just

Navigate to the support tab on your and got to “View Event Log”. image

It’s always wise to have a look to see if you have any issues before upgrading anyway. If you need to save the log for some reason do so otherwise just clear it with the Clear All button.image

The lawyers need assurance you confirm that you know what you’re doing so type yes in the warning form and click OK.

image 

You get an entry telling you the log has been cleared and you can close the form.image

If you launch the Upgrade Raid Controller Module Firmware menu option again you’ll now see that you’re good to go with the upgradeImage 024

Click on Download, Firmware and browse to the  and the NVSRAM. I prefer to do both in one go but you don’t have to. Just make sure that when both need an upgrade you don’t forget to do it when upgrading them separately. You can opt to download the firmware and  NVSRAM but activate them later. I normally do it all in one go (the default option).image

They’ll warn you no to do silly things, meaning you have to make sure that the firmware and NVSRAM versions are compatible. Read the documentation to make sure your OK.

image

If you’re good to go, click Yes and the update kicks of. First it’s the firmware update that runs.

image

When that’s done you’ll see the new firmware version in the Pending Version column. After that the NVSRAM update kicks in automatically. This takes a bit longer.Image 013

Once it’s done you get the green check indicating your firmware has upgraded successfully. We chose to activate in the same run but it’s not visible yet.image

Now to see this reflected I the version columns you‘ll need to close and reopen the Upgrade Raid Controller Module Firmware wizard again.
image

Close the wizard. You’re done upgrading the Raid Controller Firmware. Next things to do would be to upgrade disk firmware and perhaps the EMM (Enclosure Management Module) firmware of the extension disk bays if applicable. They

DELL Server DRAC Card Soft Reset With Racadmin


Sometimes a DRAC goes BOINK

Sometimes a DRAC (Dell Remote Access Card) can give you issues. Sometimes it’s some lingering process or another hiccup that causes this. You can try a reboot but that doesn’t always fix the issue. You can go into the BIOS and cancel any running System Services. A “confused” DRAC card can also be fixed by shutting down the server and cutting power for 5 to 10 minutes. That’s good to know as a last resort but not very feasible a lot of times, bar a maintenance window when you’re on premise.

You can also try to do a local or a remote reset of the DRAC card via OpenManage  (OMSA), racadmin. See RACADM Command Line Interface for DRAC for more information on how and when to use this tool. The racadmin can be used for a lot of remote configuration and administration and one of those is a “soft reset” or basically a powercycle, aka reboot, of the drac card itself. Don’t worry your server stays up Smile.

Local: racadmin racreset soft

Remote: racadm -r <ip address> -u <username> -p <password> racreset soft

Real life example

I was doing routine maintenance on 4 Hyper-V clusters and as part of that DUPs (Dell update packages) were being deployed to upgrade some firmware. This can be automated nicely via Cluster Aware Updating and the logging option will help you pin point the issue. See http://workinghardinit.wordpress.com/2013/01/09/logging-cluster-aware-updating-hotfix-plug-in-installations-to-a-file-share/ for more information on this.

Just like we found that the DRAC upgrade was not succeeding on two nodes.

One it was due to the DUP not being able to access the Virtual USB Device

Software application name: iDRAC6
   Package version: 1.95
   Installed version: 1.92

Executing update…

Device does not impact TPM measurements.

Device: iDRAC6, Application: iDRAC6
  Failed to access Virtual USB Device

==================> Update Result <==================

Update was not applied

================================================

Exit code = 1 (Failure)

and the other was because there was some other lingering DRAC process.

 iDRAC is currently unable to process this request because of another task.
  Please attempt one or more of the following steps to cancel the pending iDRAC task:
  1) Wait 30 minutes and retry your request.
  2) Reboot the system; Press F10; select ‘Exit and Reboot’ from Unified Server Configurator, and retry your request.
  3) Reboot the system; Press Ctrl-E; select ‘System Services’. Then change ‘Cancel System Services’ to YES, which will close the pending task;
      Then press Enter at the warning message. Press ESC twice and select ‘Save Changes and Exit’ and retry your request.

==================> Update Result<==================

Update was not applied

================================================
Exit code = 1 (Failure)

They give some nice suggestions but the racreset is another nice one to have I your toolkit. It’s fast and effective.

Run racadmin racreset soft

image

Wait for a couple of minutes and then run the DUP or the items in SUU that failed. With some luck this will succeed now.

image

Great Hardware Support Equals Fast Windows 2012 R2 Implementation


I love it when a plan comes together

We adopted Windows 2012 when right after it went RTM in august 2012. Today we’re are already running Windows 2012 R2 and ready to step up the pace. If you are a VAR/ISV that does not have fast & good support for Windows Server 2012 R2 consider this your notice. You can’t lead from behind. Get your act together and take an example from Altaro. Small, sure, but good & fast. How do we get our act together so fast? Fast? Yes, but it does take time and effort.

As it turns out, we’re pretty well of with the DELL hardware stack. The generation 11 and 12 servers are supported by DELL and on the Windows Server Catalog for Windows Server 2012 R2.

image

For more information on Dell Server inbox driver support see: Windows Server 2012 R2 RTM Inbox Driver Support on Dell PowerEdge Servers. By the way I can testify that we’ve run Windows Sever 2012 R2 successfully on 9th Generation hardware (PowerEdge 1950/2950).

We’ve been running tests since Windows 2012 R2 Preview on R710/R720 and it has been a blast. We’ve kept them up to date with the latest firmware & drives via SUU. And for our Intel X520 and Mellanox ConnectX-3 we’ve had rapid support as well.

So what more could you want? Well support from your storage array vendor I would think. I’m happy to report that Storage Center 6.4 has been out since October 8th and it supports Windows Server 2012 R2. Dell Compellent Adds MLC SSD Tier – Bests 15K HDDs on Price and Performance. Mind you on a lazy Sunday afternoon 2 quick e-mails to CoPilot got me the answer that Storage Center 6.3.10 also supports Windows Server 2012 R2.  Sweet!

And that’s not just DELL, the Dell Compellent Storage Center 6.4 is fully Windows Server 2012 R2 logo certified! That’s what you want to see from you vendor. Fast & excellent support.

image

Here’s the entire DELL hardware line up with Windows Server 2012 R2 support. Happy upgrading & implementation! If you have Software Assurance you’re set to reap the benefits of that investment today!

To my all employers / clients, you see now, told you so. Now, I have a thing in common with Col. Hannibal Smith, I love it when a plan comes together Open-mouthed smile.

image

I know some of you think that all the testing, breaking, wrecking of Preview bits, RTM & GA versions we do looks like chaos. Especially when you visually add the test server & switch configurations. But that’s what it looks like to YOU. To the initiated this is well executed plan, dropping all assumptions, to establish what works & will hold up. The result is that we’re ready today and by extension, so are you.

ODX Speed Up VHDX Creation Times On Windows Server 2012 (R2)


Some technlogies you just need to see in action instead of reading about it. I have posted a video on Vimeo that shows ODX in action on Windows Server 2012 R2 and a DELL Compellent SAN running Storage Center 6.3.10 firmware that supports UNMAP & ODX. Watch the video here or on Vimeo itself for a better experience. It’s a rerun of the demo scripts used in my TechNet Belux Live Meeting of this week.

We demonstrate the amazing speeds at which we can create VHDX files on both a traditional clustered disk and a Cluster Shared Volume. If you have ever tried to create a lot of fixed VHD/VHDX files, especially larger one, then you really need to check out ODX and its potential. If you have a SAN or think about acquiring one make sure you get this feature and be sure that it works as advertised.

I hope you enjoy it and inspires you to look where you can leverage this technology in your own environments.