Attending the Dell Tech Summit EMEA


As you read this I’m preparing to get on my way to the DELL Tech Summit in Lisbon, Portugal for a few days. I’ll be discussing the needs we have from them as customers (and their competition actually for that matter) when it comes to hardware in the Microsoft landscape in the era of Windows Server 2012.

image

I’m very happy and eager to tell them what, in my humble opinion, they are doing wrong and what they are doing right and even what they are not doing at all Smile  I believe in giving feedback and interaction with vendors. Not that I have any illusion of self importance as to the impact of my voice on the grand scheme of things but if I don’t speak up nothing changes either. As Intel and Microsoft are there as well,  this makes for a good selection of the partners involved. So here I go:

  1. More information on storage features, specifications and roadmaps
  2. Faster information on storage features, specifications and roadmaps
    • Some of these are in regards to Windows Server 2012 & System Center 2012 (Storage Pools & Spaces, SMI-S, ODX, UNMAP, RDMA/SMB3.0 …) and some are more generic like easier & better SAN/Cluster failovers capabilities, ease of use, number of SCSI 3 persistent reservations, etc.
  3. How to address the IOPS lag in the technology evolution. Their views versus my ideas on how to tackle them until we get better solutions.
  4. Plans, if any, for Cluster In a Box (CiB) building blocks for Windows Server 2012 Private Cloud solutions.
  5. When does convergence make sense and when not cost/benefit wise (and at what level). I’d like a bit more insight into what DELLs vision is and how they’ll execute that. What will new storage options mean to that converged network, i.e. SMB 3.0, Multichannel & RDMA capable NICs. Now convergence always seems tied to one tech/protocol (VOIP in the past, FCoE at the moment) and it shouldn’t, plenty of other needs for loads of bandwidth (Live migration, Storage Live Migration, Shared Nothing Live Migration, CSV redirected mode, …).

Now while it’s important to listen to you customers, this is not easy if you want to do it right, far from it. For one we’re all over the place as a group. This is always the case unless you cater to a specialized niche market. But DELL serves both consumers and enterprises form 1 person shops to fortune 500 companies in all fields of human endeavor. That makes for nice cocktail of views and opinions I suspect.

Even more importantly than listening is processing what you hear from your customers. Do you ignore, react, or take it away as more or less valuable information. Information on which to act or not, to use in decision making, and perhaps even in executing those decisions. And let’s face it without execution decisions are pretty academic exercises. In the end management is in control and for all the feedback, advise, research that gathered and done, they are at the steering wheel and they are responsible for the results.

One thing that I do know from my fellow MVPs and the community is that for the past 12 months any vendor who would address those questions with a good plan and communications would be a top favorite while selecting hardware at many customers for a lot of projects.

Intel X520 Series NIC on Windows 2012 With Hyper-V Enabled Port Flapping Issue


When you install Windows Server 2012 RTM to a server with X520 series NIC cards you’ll notice that there is a native driver available and the performance of that driver is fantastic. It’s really impressive to see.

image

That’s great news but I’ve noticed an issue in RTM that I already dealt with in the release candidate.

The moment you install Hyper- V some of the X520 NIC ports can start flapping (connected/disconnected).  You’ll see the sequence below endlessly on one port, sometimes more.

image

image

image

As you can imagine this ruins the party in Hyper-V networking an bit too much for comfort Confused smile But it can be fixed. The root cause for this I do not know but it is driver related. The same thing happened in the release candidate. But now things are easier to fix. Navigate to the Intel Site to download their freshly released driver for the X520 series on Windows Server 2012 and install it (you don’t need to install the extra software with Advanced Network Services => native Windows NIC teaming has arrived). After that the flapping will be gone.

image

Hope this helps some folks out!

How To Deploy Windows Server 2012 on DELL UEFI Now–Notes From The field


The most current UEFI OS Deployment on a R810 is a bit finicky when you want to deploy Windows Server 2012 using the normal procedure & selecting “Other OS” as it’s obvious that the entry for Windows Server  2012 is not in there yet. The problem is that the Windows installer doesn’t seem to create the best practice UEFI partitions. It just seems to create a 320MB System Reserved partition and the rest is for your OS installation as Primary partition. In a good (by the book UEFI) install you’d see a layout like this (from Sample: Configure UEFI/GPT-Based Hard Drive Partitions by Using Windows Setup):

image

image

The reason for this seems to be that the firmware is still not 100% up to date for how Windows Server 2012 deals with UEFI installations. This I learned via my very helpful twitter friend Florian Klaffenbach

While an update for the system firmware is in the works and won’t be to long away let me share you how I dealt with this issue. It’s a bit more work but it get’s the job done. At least for me on a R810 with BIOS version 2.7.4.

I’m copying and adapting the step by step from Microsoft Windows Server 2012 Early Adopter Guide – Dell here and adapting it to how I worked around it. It’s “magic” Winking smile.

Installing Using Dell Unified Server Configurator

  1. Connect the keyboard, monitor, mouse, and any additional peripherals to your system
  2. Turn on the system and the attached peripherals.
  3. Press <F10> in the POST to start the System Services. The Initializing UEFI. Please wait… and the Entering System Services…Starting Unified Server Configurator messages are displayed.
  4. In the Unified Server Configurator window, if you want to configure hardware, diagnostics, or set changes, click the appropriate option. If no changes are required, press OS Deployment. => you can opt to start with a cleanly build VDisk. Which is best and should suffice. But is doesn’t. We’ll clean the disk later anyway later on in Step 14.
  5. In the Operating System Deployment window, click Deploy OS. The Configure or Skip RAID window is displayed. If Redundant Array of Independent Disks (RAID) is configured, the window displays the existing RAID configuration details.
  6. Select Go directly to OS Deployment. If RAID is not yet configured, configure it at this time.
  7. Click Next. The Select Operating System window is displayed with a list of compatible operating systems.
  8. Choose Microsoft Windows Server 2012 and click Next.NOTE: If Microsoft Windows Server 2012 is not listed, choose any other operating system
  9. Choose whether you want to deploy the operating system in UEFI or BIOS mode, and click Next => I do not get this choice if UEFI is already on in the BIOS settings
  10. In the Insert OS Media window, insert the Windows Server 2012 media and click Next.
  11. In the Reboot the System screen, follow the instructions on the screen and click Finish. If a Windows operating system is already installed on your system, the following message is displayed: Press any key to boot from the CD/DVD …Press any key to begin the installation. If you used a clean VDisk this is no issue
  12. In the Windows Setup screen, select the appropriate option for Language, Time and Currency Format, and Keyboard or Input Method.
  13. Click Next to continue.
  14. STOP => Select to REPAIR your system and launch a command line. Form there you start diskpart and run following commands on the disk where you want to deploy Windows Server 2012:
    • select disk 0
    • clean
    • convert gpt

      In my case this is Disk 0. This is what the installer should be able to do automatically with a clean disk any way but it doesn’t happen.

      Now DO NOT navigate to the X:\ root and launch setup again. Shut exit the repair console and shutdown the server.

  15. Start the server
  16. Press <F10> in the POST to start the System Services. The Initializing UEFI. Please wait… and the Entering System Services…Starting Unified Server Configurator messages are displayed. => DO NOT TOUCH ANYTHING ANYMORE. It will take longer than expected but you will boot into the installation of Windows 2012 again.
  17. In the Windows Setup screen, select the appropriate option for Language, Time and Currency Format, and Keyboard or Input Method.
  18. Click Next to continue.
  19. On the next page, click Install Now.
  20. In the Operating System Install screen, select the operating system you want to install. Click Next. The License Terms window is displayed, click Next.
  21. In the Which Type of Installation Do You Want screen, click Custom: Install Windows only (advanced), if it is not selected already.
  22. In the Where do you want to install Windows screen, specify the partition on which you want to install the operating system. To create a partition and begin installation:
    1. Click New
    2. Specify the size of the partition in MB, and click Apply. A Windows might create additional partition for system files message is displayed. => NOW THE UEFI partitions on the GPT disk are created Open-mouthed smile.
    3. Click OK.Select the newly-created operating system partition and click Next.
      The Installing Windows screen is displayed and the installation process begins. After the operating system is installed the system reboots. You must set the administrator password before you can log in for the first time
  23. In the Settings screen, enter the password, confirm the password, and click Finish.
    The operating system installation is complete.

image

Now, while this worked for me on the Dell R810 with BIOS 2.7.4,  I give no guarantees whatsoever. You’ll have to test it yourself or wait for the firmware update that is coming soon. Any way, perhaps it helps some of you out there!

The Right Stuff


You all probably know that to get a difficult job done well and fast, you need the right people in the right place at the right moment in time. Those people also need the right tools. This requires people who can think on their feet, people who are resourceful and who will always seek and find opportunities under adverse conditions.

The placement and timing of these resources and assets is more than just management of some table matching names to roles. It’s not enough to have the right resources and skill sets. You need to know who and what is available and what these or they can contribute. Management is often not very good at this, so in a crisis they need to let go and rely on their people. Free Tip: you can’t start building a team when the crisis arrives ;-)

It’s the boots on the ground will have to deal with the issues at hand and take the decisions. In a crisis time is of the essence. There is no place for too many layers of management, let alone micro management, only the ones with the right responsibilities insight and knowledge are needed and helpful. The decisions become tactical and operational within the context of the situation at hand and it its relation to the entire environment. So they have to be made by people who preferably know the environment well and have a very good skillset, drive and motivation. Basically this is what I refer to when I talk about the right stuff.

If you have ever worked or work in that sort of environment you know what I’m talking about. The knowledge that no matter where you are going for whatever reason, you’re doing so with a team of very skilled people who are the very best in the business, at the top of their game and ready to roll with any situation thrown at them. They are capable to react in a moment’s notice and focus entirely on the job at hand. If you’re interested in building such a team I suggest you select your team members very carefully. Head count doesn’t mean jack shit if they are the wrong people for the job and the team. Don’t ever lower the bar, it’s there for a very good reason.

This year I had misfortune of having to respond to two major HVAC disasters at night in a weekend. I had the good fortune of having the right stuff at my disposal. There is no “On Call”, there is no monetary compensation. This team is my crew and they are all volunteers who will do what is needed when it is needed. Why because they have professional pride and know that at these moments the very survival of the business they work for depends on them acting fast and correctly. To them it’s not about “somebody should do this” or ”that’s not my job”. It’s not about “this should be taken care of” or “I never had a template telling me what to do”. No, they step forward and get it done. This weekend, from the very first alert, 4 people were mobilized in 30 minutes and acted at the speed of light. This led to the emergency shutdown of a data center in a city 60 kilometers way to prevent a catastrophic meltdown of millions of euros in hardware (not even trying to put a value on the data loss). Two people were acting remotely and 2 (including me) were heading over there to have boots on the ground. The reason for this is that “the away team” could deal with anything that couldn’t be handled remotely and coordinate with facility management. Having people on site is important to all involved (two is preferable for safety reasons) for assessing the situation and for the sake of speed. More people often becomes less efficient as numbers are not the same as capability.

So to my team, I’m proud of you. I quote Beckwith “I’d rather go down the river with 7 studs than with a 100 shitheads”. You all know you’ve got the right stuff. Be proud of that!

Windows Server 2012 RTM in Production as Backup Media Server


As Brad Anderson suggested a few times this year we are leading our businesses to a better IT future. Here’s a quick screen shot of one of our backups media agent servers hooked up to bunch of NL-SAS disks (150TB). One host (older model, no 10Gbps yet, running Windows 2008 R2) is sending backups over its dedicated 1Gbps interface to this media agent server who has a 10Gbps interface. Performance and traffic looks great Smile. I love it when a plan comes together!

image

 

This is what I looks like at host being backed up

image

Now we’ll see what magic Data Deduplication can do for us with the RTM bits!

Windows Server 2012 KMS Service Activation


Now we have the Windows Server 2012 and Windows 8 RTM bit form our Volume License we need to get some housekeeping done. The first thing we do is setup or update our KMS Service.

In our case it is running on Windows Server 2008 R2 so we need to do a couple of things.

Install the following update: An update is available for Windows 7 and Windows Server 2008 R2 KMS hosts to support Windows 8 and Windows Server 2012 as described in KB2691586. This is also the place where you can request this hotfix.  If you don’t install this hotfix registering a Windows Server 2012  KMS will throw an Error: 0xC004F050 The Software Licensing Service reported that the product key is invalid

So request the hotfix and install it from an elevated command prompt. Just follow the instructions and you’ll be fine Smile

image

Once you’ve clicked OK the installation will start

image

After that’s finished you will be asked to restart the server. Do so. Just restarting the KMS service ("net stop sppsvc" and "net start sppsvc") doesn’t suffice.

image

Now we have that out the way we can start putting our brand new KMS key into action.

Let’s take a look at what is already running:

slmgr.vbs /dlv => clearly the Windows 2008 R KMS key
image

Uninstall the current KMS key using slmgr.vbs /upk, please use an elevated command prompt Winking smile

image

 

Now you can install the new KMS key. The key listed here is obviously a demo one Winking smile If you run in to any issues here, restarting the KMS Service can help. Try that first.

slmgr.vbs /ipk NOPEI-AMNOT-GIVIN-GITTO-YOU!!

image

 

Now activate your brandnew KMS key running slmgr.vbs /ato

image

 

Show what’s up and running now by running slmgr.vbs /dlv again and as you can see we’re in business to activate all our Windows Server 2012 and Windows 8 hosts. Life is good Smile

image

Windows Server 2012 Bits Available for Download for Volume Licensing Customers


A long awaited day has arrived. The bits of Windows Server 2012 RTM are available to us. Ever since the BUILD conference in September 2011 a lot of us have been diving into this version with enthusiasm and amazement for what’s in the product. As a matter of fact I’ve “sold” projects based on Windows Server 2012 internally since October 2011 because we were that impressed with what we saw.

  • Grab the bits on the Microsoft Volume Licensing site (from August 16th onwards). I whish I could tell you it’s also on TechNet or MSDN but no joy there so far.

So we’ve been pouring over the product and the information available, gradually gaining a better understanding of what it can do for us and our businesses. That meant building labs, testing scenarios, presenting on the subject at various occasions.  There was also a lot of thinking, dreaming and discussing ideas and options about what we can do this version of Windows. It has been very busy for the past 11 months but I’m also very happy to have had the opportunity to attend several summits and conferences where I met up with colleague, fellow MVPs, MSFT employees who all shared the enthusiasm for this release and what it means for Hyper-V and the Private/Hybrid Cloud.

 

So to all of them, ladies & gentlemen, my on line community buddies form all over this planet, it’s been a blast Smile. They have been very helpful in all this as have been al the Microsoft employees who’ve answered and discussed all the questions/ideas we threw at them. I would like to thank all of them for their time, their patience and the opportunities given to us. I can offer those guys & galls just one reward: the fact that from day one we are taking this in production and gradually will do so for all our infrastructure systems and so on. It’s a no brainer when you’ve worked with the RC and seen what Windows Server 2012 can do. And no, I’m not forgetting Windows 8. SMB 3.0 & Direct Access and Windows to go alone make that a sweet proposition, but I got those bits already.

image

Well, the downloads are running and the installation of our first production Hyper-V Cluster and infrastructure servers can start as soon as that’s finished (we’ll lead Brad, we’ll lead Winking smile). After some initial tests these will be taken into service and that last feedback will provide us with the go or no go for the rest of our infrastructure. The speed & completeness of our move depends partially on how fast System Center 2012 SP1 brings support for Windows Server 2012.

So future blog posts and my next presentations will spiced with some real life production experience with the RTM bits. May all your roll outs be smooth ones!

Windows Server Backup Benefits from Improvements in Windows Server 2012


Introduction

In certain environments we backup VMs and any remaining physical hosts using Windows Backup. Before you all think this is ridiculous, I advise you to think again. With some automation you can build a very reliable agentless backup solution with the built in functionality. Windows Server 2012 brings good news for the smaller & perhaps low budget environments. Windows Server Backup is now capable of doing host level backups of the Hyper-V guest stored on Clustered Shared Volumes. This was not the case in Windows 2008 R2 and it is a vast improvement.This change is due to the fact that CSV has been changed not to require specialized API capable of dealing with it’s intricacies. All backup products now can backup CSVs without specialized APIs.

This is, linked to huge improvements in how a CSV behaves during a backup. In the past, when you started a backup, the CSV ownership would be moved to the node that runs the backup and all access by other nodes was in redirected mode for the duration of the backup. Unless you used a hardware VSS provider, which were not trouble free either. If your backup software did not understand CSVs and use the CSV APIs you were out of luck. From Windows Server 2012 on you are only in redirected I/O mode for the time it takes to create the VSS snapshot. The rest of the backup duration your nodes access the CDV disk in direct mode. So back to Windows Server Backup. You cannot up the CSV as disk volume but you can select Hyper-V from the items to include in the backup.

image

That will show all VMs running on that host, meaning you cannot backup VMs running on another host. Compared to Windows Server 2008 R2 where using the native Windows backups  with VMs on a CSV LUN meant using in guest backups this a major improvement.

Some Approaches to Using Windows Server Backup

Sometime we run the backups to local disk and regularly copy those off to a file share. This has the benefit of providing the backup versioning you can from using a local disk. The draw back is that the backups can be rather big.

In VMs, that is with backups in the guest, we run those backups to a file share over a 1Gbps management network. Performance is good, but it leaves us with the issue that there is no versioning.

For that reason our backup script copies the entire backup folder for a server to an archive folder on that same JBOD. Depending on how much space you have and need you can can configure the retention time of these older backups. This way you can keep a large number of backups over time.  A script runs every day that deletes the older backups based on the chosen retention time so you don’t run out of space.

There is one way around the lack of versioning when writing backups to a share and that is to mount a VHD on a file share locally to the host where you are running a backup or use pass through disk inside the VM. While you can get away with this becomes rather messy due to management & flexibility drawbacks of pass through disks. Mounting a VHD on a file share inside a VM is also a performance issue. So while possible and viable in certain scenarios I don’t use this for more than a few hosts and those are physical ones.

We had hoped that Windows Server 2012 with its support for VSS snapshots on SMB 3.0 shares would have enabled backup versioning in Windows Backup, just like it can do for backups to local disk. Unfortunately this is not the case. You’ll still get the same warning when backing up from a Windows Server 2012 host to an SMB 3.0 share as you used to get with previous versions:

image

How fast are backups and restores?

The largest environment is a couple of Physical servers, a two node Hyper-V cluster and about 22 virtual machines. That includes some with a larger amount of data. The biggest being 400 GB. Full backups are run weekly at night on all servers over 1Gbps and this works just fine.

We can backup a VM of with a 50GB VHD (about 50% to 75% in use) and copy that backup to the archive folder in 20 minutes. We backup AND copy to archive a VM’s C: Drive (20GB of data) and a D: Drive (190GB of data), 2 separate VHD  in 2.5 hours.

For some statistics. A bare metal restore of a VM or physical host over 1Gbps with a single VHD or volume with the OS, applications and some data takes us 30 tot 35 minutes in real life due to the overhead of setting a new VM up. I you just want to restore individual data you can do that as well. You can even mount the backup VHD and recover them via the Windows Explorer.

This is what a the logging we do and e-mail to the sys admins looks like:

image

Where else do the Windows Server 2012 improvements help?

Data Deduplication

Now there is some good news. We ran ddpeval.exe against the JBOD LUNS where we store these archived backups and got some great results. We also copied such an archive folder to a Windows Server 2012 Host and ran data deduplication against it. In that test we achieved a 84%-85% deduplication rate depending on how many versions of the backups we archive and what the delta is during that time frame.  The latter is important. If we run dedupe only against domain controller backup archives we get up to 94%. Deduplication should not not impact restore performance to much because in 99% of the cases you revert to the last backup which sits in the WindowsBackup Folder. Only if you need older backups you will work against the deduped files, unless the archive folder is on the same LUN as the original backups. I’ don’t have real life info about restores yet. Just a small lab test.

SMB 3.0 & Multichannel

In Windows Server 2012 you also get all benefits of SMB 3.0 and MultiChannel for your backup traffic.

Not your grandfathers ChkDsk

The vastly improved ChkDsk is a comfort for worried minds when it comes to fixing potential corruption on a large LUN. Last but not least, the FLUSH command in NTFS makes using cheaper data disks safer.

VHDX

The VHDX format allows for 64TB. That means your Windows Server backup can now handle more than 2TB LUNs. This should be adequate Smile

Conclusion

With some creativity and automation via scripting you can leverage the Windows Backup to be a nice and flexible solution. Although I feel that providing backup versioning to a file share is an improvement that is missing  the new features help out a lot and all in all, it’s not bad at all! So you see, even smaller organizations can benefits seriously from Windows Server 2012 and get more bang for their bucks.

Disk to Disk Backup Solution with Windows Server 2012 – Part I


Backing Up 100 Plus Terabyte of Data Cheaply

When dealing with large amounts of data to backup you’re going to start bleeding money. Sure people will try to sell you great solutions with deduplication, but in a lot of scenarios this is not a very cost effective solution. The cost of dedupe in either backup hardware or software is very expensive and in some scenarios the cost cannot be justified. It’s also not very portable by the way unless in certain scenarios in which you stick with certain vendors. Once you get into backing up  > 100TB you need to forget about overly expensive hard & software. Just build your own solutions. Now depending on your needs you might want to buy backup software anyway but forget about dedupe licenses. Some of the more profitable hosting companies & cloud providers are not buying appliances or dedupe software either. They make real good money but they rather spend it on SUVs and swimming pools.

What Can You Do?

You can build your own solution. Really. You can put together some building blocks that scale up and out. You’ll a dual socket server with two 8 core CPUs and 24GB of ram, perhaps 32GB. Plug in some 6Gbps SAS controllers, hook those up to a bunch of 3.5” disk bays with 12 *2TB or 3TB disks each and you’re good to go. You can scale out to about 8 disk bays if you don’t cluster. Plug in a dual port 10Gbps card. You’ll need that as you be hammering that server. If you need more than this system, than scale out, put in a second, a third, etc. 3.5TB –4TB of backup capacity per hour in total should be achievable..

image

When you buy the components from super micro and some on line retailers you can do this pretty cheap. Spare parts you say? Buy some cold spares. You can have a dozen disk on the shelf, a SAS controller and even a shelf if you want. You could use hardware redundancy (RAID, hot spares) or use storage pools & spaces if you’re going the Windows Server 2012 route and save some extra money. Disk bay failure? Scale out so that even when you loose a node you still have tree others up and running. Spread backups around. Don’t backup the same data only to the same node. I know it’s not perfect for deduplication with Windows 2012 that way but hey, you win some, you lose some. Checks & balances right?  If you need a bit more support get some DELL PowerVaults or the like. It depends on what you’re comfortable with and how deep your pockets are.

You can by more storage than dedupe will ever save you & still come out with money to pay for the electricity. Okay it’s less good for the penguins but trust me, those companies selling those solutions would fry a penguin for breakfast everyday if it would make them money. Now talking about those penguins, the Windows Server 2012 deduplication feature could be providing me with the tools to save them Smile, but that’s for another post. I hope this works. I’d love to see it work. I bet some would hate to see it work. So much perhaps that they might even consider making their backup format non dedupableDevil?

Tip for users: Don’t use really cheap green SATA disks. They’re pretty environment friendly but the performance sucks. My view on “Green IT” is to right size everything, never to over subscribe and let that infrastructure work hard for you. This will minimize the hardware needed  and the performance is way better than all the power saving settings and green hardware. Which will ruin the environment anyway as you’ll end up buying more gear to compensate for lack of performance unless you’ll just suffer the bad performance. Keep the green disks for the home user’s picture, movie & music collection and use 2TB/3TB SAS/NL-SAS. Remember that when you don’t cluster (shared storage) you can make due nicely without the enterprise NL-SAS disks.

Now I’m not saying you should do what I suggest here, but you might find it useful to test this on your own scale for your own purposes. I did it for the money. For the money? Yup for the money. No not for me personally, I don’t have a swimming pool and I don’t even own a car, let alone an SUV. But saving your company a 100.000 or more in cash isn’t going to get you into trouble now is it? Or perhaps this is the only way you’re going to afford to back up that volume of data. People don’t throw away data and they don’t care about budgets you’d better be able to restore their data. Which reminds me, you will also need some backup software solutions that doesn’t cost an arm and a leg. That’s also a challenge as you need one that can handle large amounts of data and has some intelligence when I comes to virtualization, snapshots etc. It also has to be easy to use, as simple as possible as this helps ensure backups are made and are valid.

Are we trying to replace appliances or other solutions? No, we’re trying to provide lots of cheap and “fast enough” storage. Reading the data & providing it to the backup device can be an issue as well. Why fast enough? Pure speed on the target side is not useful if the sources can’t deliver. We need this backup space for when the shits really hits the fan and all else has failed.That doesn’t have to be a SAN crash or a SAN firmware issue ruing all your nice snapshots. It can also be the business detecting a mistake in a large data set a mere 14 months after the facts when all replicas, snapshots etc. have already expired. I’m sure you’ve got quality assurance that is so rock solid that this would never happen to you but hey, welcome to my world Sarcastic smile.