Upgrading The DELL MD3600F Controller Firmware Using the Modular Disk Storage Manager


As part of our hardware maintenance we’re deploying the updates of SUU 7.3.1 right now. As part of that effort we’re also making sure the PowerVault disk bays are getting their updates. We’ve got a couple of PowerVault MD3600F with 7 MD1200 extension bays filled with disks attached to R710  Power Edge servers that act as Disk2Disk backup media servers with 10Gpbs networking and running W2K12R2.

In general you should not be applying firmware updates and such constantly and all over the place just to keep busy. It is, however, a good practice to keep an eye on releases and see if they fix any risky bugs you might be vulnerable to. You see that sometimes these fix issues you don’t want to run into and those are often marked urgent. You also don’t get to far behind on firmware updates as this can complicate thing later in the useful service life or you hardware. First of all read the readme.txt and such to make sure you’re not missing any special instructions for particular cases that might exist. The standard disclaimer of my blog applies Smile, you’re responsible for your own actions.

First of all get the latest version of your PowerVault Modular Disk Storage Manager software if you haven’t already. Often you’ll need to have a least a certain version before you can even do newer firmware updates. So just get this in order before starting, especially if they tell you to, don’t try and outsmart the system. Download the disk firmware form the DELL support website or via the DELL Storage Community Wiki http://en.community.dell.com/techcenter/storage/w/wiki/4234.dell-powervault-md-downloads.aspx?dgc=SM&cid=257966&lid=4630585 and safe it to disk.

Here’s an example of a read me file you’d better not ignore. If you do, the Modular Disk Storage Manager will not allow you to upgrade anyway.

*** ATTENTION ***
If your PowerVault MD32xx/MD36xx series storage array is currently running a firmware version lower than 07.75.28.60, you cannot directly install the RAID controller firmware version 07.84.47.60 included in this release.
Instead, you must first install the bridge firmware version 07.75.28.60. After the bridge firmware is successfully installed, you can then install RAID controller firmware 07.84.47.60.

Specifically, follow these steps:
1. *** IMPORTANT *** Install Dell PowerVault MD Storage Manager (MDSM) software from the Resource DVD version 4.1.2.29 or higher.
2. Extract the following files from the Bridge_Firmware_07_75_28_60.zip archive:
* Bridge_Firmware_07_75_28_60.dlp
3. Download and activate RAID controller bridge firmware version 07.75.28.60 (contained in the Bridge_Firmware_07_75_28_60.dlp file)
4. Once the bridge firmware installation is successfully completed, wait at least 30 minutes to allow the attached hosts to re-discover the storage array.
5. Download and activate the RAID controller firmware version 07.84.47.60 (contained in the MD3xxx_MD3xxx_Firmware_07_84_47_60.dlp file) together with the NVSRAM configuration file version N26X0-784890-X04 (contained in the MD3xxx_MD3xxx_NVSRAM_N26X0-784890-X04.dlp file) using the Dell PowerVault MD Storage Manager for your corresponding array type.
* MD3200_MD3220_Firmware_07_84_47_60.dlp and MD3200_MD3220_N26X0_784890_004.dlp (6G Non-Dense SAS)
* MD3200i_MD3220i_Firmware_07_84_47_60.dlp and MD3200i_MD3220i_N26X0_784890_004.dlp (1G Non-Dense iSCSI)

* MD3600i_MD3620i_Firmware_07_84_47_60.dlp and MD3600i_MD3620i_N26X0_784890_904.dlp (10G Non-Dense iSCSI)
* MD3600f_MD3620f_Firmware_07_84_47_60.dlp and MD3600f_MD3620f_N26X0_784890_904.dlp (8G Non-Dense FC)
* MD3260_Firmware_07_84_47_60.dlp and MD3260_N26X0_784890_004.dlp (6G Dense SAS)
* MD3260i_Firmware_07_84_47_60.dlp and MD3260i_N26X0_784890_004.dlp (1G Dense iSCSI)
* MD3660i_Firmware_07_84_47_60.dlp and MD3660i_N26X0_784890_904.dlp (10G Dense iSCSI)
* MD3660f_Firmware_07_84_47_60.dlp and MD3660f_N26X0_784890_904.dlp (8G Dense FC)

*** WARNING ***
If you have a single controller PowerVault MD32/MD36 series storage array you must stop all I/O operations before starting the RAID controller firmware upgrade.

Launch the PowerVault Modular Disk Storage Manager, right click your array and select Upgrade Raid Controller Module Firmware.image

Wait while the status is being refreshed.Image 017

You might run into a an issue if the event log contains to many entries. In that case you’ll be warned you can’t upgrade until you’ve cleared it as shown below. Here are some screenshots from another previous upgrade where I ran into this issue.Image 018

In that case open up the Manage Storage Array  for your array.image just

Navigate to the support tab on your and got to “View Event Log”. image

It’s always wise to have a look to see if you have any issues before upgrading anyway. If you need to save the log for some reason do so otherwise just clear it with the Clear All button.image

The lawyers need assurance you confirm that you know what you’re doing so type yes in the warning form and click OK.

image 

You get an entry telling you the log has been cleared and you can close the form.image

If you launch the Upgrade Raid Controller Module Firmware menu option again you’ll now see that you’re good to go with the upgradeImage 024

Click on Download, Firmware and browse to the  and the NVSRAM. I prefer to do both in one go but you don’t have to. Just make sure that when both need an upgrade you don’t forget to do it when upgrading them separately. You can opt to download the firmware and  NVSRAM but activate them later. I normally do it all in one go (the default option).image

They’ll warn you no to do silly things, meaning you have to make sure that the firmware and NVSRAM versions are compatible. Read the documentation to make sure your OK.

image

If you’re good to go, click Yes and the update kicks of. First it’s the firmware update that runs.

image

When that’s done you’ll see the new firmware version in the Pending Version column. After that the NVSRAM update kicks in automatically. This takes a bit longer.Image 013

Once it’s done you get the green check indicating your firmware has upgraded successfully. We chose to activate in the same run but it’s not visible yet.image

Now to see this reflected I the version columns you‘ll need to close and reopen the Upgrade Raid Controller Module Firmware wizard again.
image

Close the wizard. You’re done upgrading the Raid Controller Firmware. Next things to do would be to upgrade disk firmware and perhaps the EMM (Enclosure Management Module) firmware of the extension disk bays if applicable. They

Advertisements

Live Migration Can Benefit From Jumbo Frames


Does live migration benefit from Jumbo frames? This question always comes back so I’d just blog it hear again even if I have mentioned it as part of other blog posts. Yes it does! How do I know. Because I’ve tested and used it with Windows Server 2008 R2, 2012 & 2012 R2. Why? because I have a couple of mantra’s:

  • Assumption are the mother of all fuckups
  • Assume makes an ASS out of U and ME
  • Trust but verify

What can I say. I have been doing 10Gbps since for Live Migration with Hyper-V. And let me tell you my experiences with an otherwise completely optimized server (mainly BIOS performance settings): It will help you with up to 20% more bandwidth use.

And thanks to Windows Server 2012 R2 supporting SMB for live migration we can very nicely visualize this with 2*10Gbps NICS, not teamed, used by live migration leveraging SMB Multichannel. On one of the 10Gbps we enable Jumbo Frames on the other one we do not. We than live migrate a large memory VM back and forth. Now you tell me which one is which.

image

Now enable Jumbo frames on both 10Gbps NICs and again we live migrate the large memory VM back and forth. More bandwidth used, faster live migration.

image

I can’t make it any more clear. No jumbo frames will not kill your performance unless you have it messed up end to end. Don’t worry if you have a cheaper switch where you can only enable it switch wide instead op port per port. The switch is a pass through. So unless you set messed up sizes on sender/receiving host that the switch in between can’t handle, it will work even without jumbo frames and without heaven falling down on your head Smile. Configure it correctly, test it, and you’ll see.

DELL Server DRAC Card Soft Reset With Racadmin


Sometimes a DRAC goes BOINK

Sometimes a DRAC (Dell Remote Access Card) can give you issues. Sometimes it’s some lingering process or another hiccup that causes this. You can try a reboot but that doesn’t always fix the issue. You can go into the BIOS and cancel any running System Services. A “confused” DRAC card can also be fixed by shutting down the server and cutting power for 5 to 10 minutes. That’s good to know as a last resort but not very feasible a lot of times, bar a maintenance window when you’re on premise.

You can also try to do a local or a remote reset of the DRAC card via OpenManage  (OMSA), racadmin. See RACADM Command Line Interface for DRAC for more information on how and when to use this tool. The racadmin can be used for a lot of remote configuration and administration and one of those is a “soft reset” or basically a powercycle, aka reboot, of the drac card itself. Don’t worry your server stays up Smile.

Local: racadmin racreset soft

Remote: racadm -r <ip address> -u <username> -p <password> racreset soft

Real life example

I was doing routine maintenance on 4 Hyper-V clusters and as part of that DUPs (Dell update packages) were being deployed to upgrade some firmware. This can be automated nicely via Cluster Aware Updating and the logging option will help you pin point the issue. See https://workinghardinit.wordpress.com/2013/01/09/logging-cluster-aware-updating-hotfix-plug-in-installations-to-a-file-share/ for more information on this.

Just like we found that the DRAC upgrade was not succeeding on two nodes.

One it was due to the DUP not being able to access the Virtual USB Device

Software application name: iDRAC6
   Package version: 1.95
   Installed version: 1.92

Executing update…

Device does not impact TPM measurements.

Device: iDRAC6, Application: iDRAC6
  Failed to access Virtual USB Device

==================> Update Result <==================

Update was not applied

================================================

Exit code = 1 (Failure)

and the other was because there was some other lingering DRAC process.

 iDRAC is currently unable to process this request because of another task.
  Please attempt one or more of the following steps to cancel the pending iDRAC task:
  1) Wait 30 minutes and retry your request.
  2) Reboot the system; Press F10; select ‘Exit and Reboot’ from Unified Server Configurator, and retry your request.
  3) Reboot the system; Press Ctrl-E; select ‘System Services’. Then change ‘Cancel System Services’ to YES, which will close the pending task;
      Then press Enter at the warning message. Press ESC twice and select ‘Save Changes and Exit’ and retry your request.

==================> Update Result<==================

Update was not applied

================================================
Exit code = 1 (Failure)

They give some nice suggestions but the racreset is another nice one to have I your toolkit. It’s fast and effective.

Run racadmin racreset soft

image

Wait for a couple of minutes and then run the DUP or the items in SUU that failed. With some luck this will succeed now.

image

Attending The Global MVP Summit 2013 (November 18-21)


We have Windows 8.1 running on our desktops & laptops and meanwhile Windows Server 2012 R2 is crunching numbers in our (virtualized) data centers. So it’s time to grab one of those magnificent British Airways Boeing 747 aircraft seats once again and make my way to SEATAC. No rest for the wicked. BAS7400066

Yup, but for now I’ve parked myself in LHR whilst waiting for my flight. Soon I’ll be in the air again for the long haul to the USA. I’m off to Washington State, Seattle to be exact, and from there to Bellevue/Redmond. You might have guessed where I am going already, indeed to the Microsoft campus. I’m attending the Global MVP Summit 2013, November Edition. image

Apart from that magnificent educational & networking opportunity I will spend a lot of the “free” time discussing technology, visions & strategies with my peers and Microsoft employees. I’d like to thank the latter for their patience with me when bugging them with questions Smile. To my buddies, acquaintances & connections, I’ll see you soon. We have a lot to learn & discuss. That’s one of the reasons I’m off a bit earlier. It helps with the jet lag but it also gives me time to meet up with friends and acquaintances I’ve made in the Puget Sound area and talk shop. This helps to keep in touch with what’s happening over the world and to understand where their priorities are, what’s keeping them occupied. While I’m a firm believer in remote and teleworking there is value in getting your boots on the ground every now and then. It prevents tunnel vision and helps avoiding  teleology in our views while enhancing early detection of small trend changes to whole sale tectonic shifts. This is not to be confused with thinking you have a crystal ball or anything.

To my readers & community members I’d like to extend the invitation to pass along feedback to Microsoft. They do listen. So leave a comment, send me a mail (contact via Blog) or ping me on Twitter.

In case you don’t know, everything discussed at the MVP Summit is under NDA, even for MVPs of another expertise. So basically bar some tweets to find our where other MVPs I’ll be going dark.

How To Save A Company From Death By Meetings


I had a very interesting discussion with a fellow virtualization expert & strategic advisor at E2EVC Rome 2013. We discussed many issues and the topic came up that we see way too many potentially strong organizations sub come to “death by meetings”.  Instead of fixing this they are treating the symptoms and are declaring the symptoms to be “illegal”. It’s almost pandemic. You really need to opt out of this this madness for the sake of the company and getting work done.

I think we need to take this further. Bar reverting to the tactics of a UK manager who locked all meeting rooms and took the keys away whilst telling his staff to stop meeting and start working, we should implement these rules:

  1. There is an absolute maximum of 5 hours meeting per 8 hour workday. Work less than full time? Adjust accordingly. This is non negotiable for anyone. You must decline any meeting that violates this rule. You must not strive for this maximum.
  2. You must decline any meeting that has no agenda.
  3. No meeting can have more than 5 attendees unless a very valid reason and need is motivated in the agenda. If not, you have to decline the meeting.
  4. Meetings have to be planned in advance & cannot be made permanent. That is reserved for councils or meetings of the board.
  5. Everyone, from the lowest pay grade to the top manager hast to abide by these rules
  6. Teleconferencing is a perfectly valid way of meeting and is included in these 5 hours.

meetingsdemotivator[1]

People, really, stop meeting for 8 hours a day. It’s scary so many companies can’t get a grip on this. Your thoughts?

Enabling Jumbo Frames Inside Virtual Machines Enhances Throughput & Reduces CPU Load


Let’s play a bit with a Windows Server 2012 R2 Hyper-V cluster with 2*10Gbps Intel X520 teamed switch independent and in dynamic mode, which is optimal for DVMQ. On these NICs we enabled Jumbo frames (and on the switches of course). That team is used to create a virtual switch for consumption by the virtual machines. The switches used are 2*DELL PowerConnect 8132F. So we have full fault tolerance. But the important thing to note is that this is commodity, quality hardware that we can leverage for great results.

Now we’ll compare 2 scenarios. In both test a sending VM will try to saturate a receiving VMs network bandwidth. Due to how this NIC teaming setup works, that’s about 10Gbps in a two member 2*10Gbps team. We work around this by leaving both VMs on the same host, so the traffic doesn’t need to pass across the wire. The VMs have VMQ & vRSS enabled with the host team members having VMQ enabled.

Jumbo frames disabled inside the VM (Both VMs on same host)

Without Jumbo Frames enabled in the Guest VM and all other things being equal the very best we can achieve non-sustained is 21Gbps (average +/- 17Gbps)  receiving traffic in a VM. Not bad, not bad at all.

image

In the picture you can see the host with DVMQ doing it’s job at the left while vRSS is at work in the VM. Pretty clear.

Jumbo frames enabled inside the VM

Now, let’s enable Jumbo frames in the VM.  Fire up PowerShell or use the GUI.

image

Don’t forget to do this on both the sending and the receiving VM Smile. Here we get +/- 30Gbps receiving traffic inside of the VM. A nice improvement isn’t it? Not just that but we consume less CPU resources as well! Sweet Smile

image

Useful Power at your finger tips or just showing off?

vRSS & DVMQ is one of may scalability & performance improvements in Windows 2012 R2. And yes, Jumbo Frames inside the VM do make a difference but in a 10Gbps environment it’s not  “in your face” that visible. The 10Gbps limit of a single NIC team member makes this your bottle neck. But it DOES help to reduce CPU cycles in that case. Just look at the two screen shots below.

No Jumbo frames in VM (sending & receiving VMs on different hosts)

image

We’re consuming 7% CPU resources on the host and 15% in the VM.

Jumbo Frames in VM (sending & receiving VMs on different hosts)

image

We’ve dropped down to 3% on the host and to 9% inside the VM. All bits help I say!

How far can we push this?

Again if you take the NIC Team bottleneck out of the way you can see some serious differences. Take a look at the screenshot below, that’s 36,2Gbps inside of a VM courtesy of vRSS during some other experiments. Tallyho!

image

So let’s face it, I guess we’ll need some faster memory (DDR4) and multiple 40Gbps/100Gbps Cards to see what the limits of Windows Server 2012 R2 are or to find out if we reached it. Right now the operating system is giving hardware people a run for their money.

Also note that if you have a number of VMs doing a lot of network IO you’ll be using quite a number of CPU cycles. While vRSS & DVMQ make this scale you might want to consider leveraging SMB Direct for the various tastes of live migration as this will definitely help you out on that front as the NIC will do the heavy lifting.

Reality Check

But perhaps we also need little reality check. While 100Gbps and DDR4 is very nice you might not need it for your current needs. When the environment is built right you’ll find that your apps are usually your limiting factor before the hardware, let alone Windows Server 2012 R2.  So why is knowing this important? Well I verify Microsoft claims so I can talk from experience and not just from what I read in a Microsoft presentation. Secondly you can trust that your investment in Windows Server 2012 R2 is going to carry you long and far. It’s future proofed and that’s god for you. Both when your when needs grow exponential and for the longevity of your environment. Third, we can leverage this to virtualize high through put environments and get the best possible results and ROI.

Also, please, please test & find out, verify what settings suit your environment best and just just blindly enable stuff. Good luck!

Presentation & Demos E2EVC Rome 2013


Well my E2EVC presentation has been given and it went well. Sweet experience for the 20th edition of this excellent community conference.

image

Thank you to all those attendees that attended my session. I hope you enjoyed it, learned something and got a taste of experimenting with some of the perhaps lesser know features at your disposal in Windows Server 2012 R2.

A big thanks to Alex and Clare for the splendid organization of E2EVC for the twentieth time!

For more information on VMQ & vRSS go here: https://workinghardinit.wordpress.com/2013/10/23/windows-server-2012-r2-virtual-rss-vrss-in-action/
http://blogs.technet.com/b/networking/archive/2013/09/10/vmq-deep-dive-1-of-3.aspx
http://blogs.technet.com/b/networking/archive/2013/09/24/vmq-deep-dive-2-of-3.aspx
http://blogs.technet.com/b/networking/archive/2013/10/22/vmq-deep-dive-3-of-3.aspx

For a good start on SMB Direct with iWarp or RoCE go here: https://workinghardinit.wordpress.com/2013/08/28/adventures-in-rdma-the-roce-path-to-windows-server-2012-r2-smb-3-0-glory/

You can download the presentation here. The video of the session will be made available later by E2EVC so you can see the demo.

Just to make you all drool for the video to become live here’s a screenshot of what DVMQ/vRSS can achieve. Just pushing it to the limit. No, this is not “Photoshopped” I have witnesses Winking smile. That’s a W2K12R2 VM receiving 37.4Gbps of traffic. That will do or most of you I guess until 100Gbps NICs are the standard LOM on your future computing device.

image

By the way some people really loved some of the drawn art I used in the presentation. For these I owe thanks to Kathy Sierra and her great blog art. It’s a shame she felt the need to go dark.

More stuff is in the work as working hard in IT never fails to deliver more good subjects, findings and results to blog about.

If you’re a managerial type and feel offended, you probably should, as you’re doing it all wrong Smile. Otherwise you would have smiled because you have a sense of humor and nodded at the mistakes of your colleagues.