It’s currently 12:18 AM PST, and we’re sitting at 42 minutes away from shutting down the grid for the big asset conversion. I will be posting regular updates on the OSGrid twitter feed. Today we’ll be doing two important upgrades to OSGrid infrastructure – the first will be moving to new hardware for the asset server, the second will be converting to the new FragstoreFS backend.

The first step will be shutting down the grid so we can export the database onto the new server – based on previous exports, this is estimated to take approximately 2 hours. The next step will be importing the database and adjusting the schema for the conversion – this involves changing the indexes to add a numeric incremental index which allows us to resume conversion from specified points. The import will take around one to two hours, and the index conversion will take a similar period of time.

The next step will be firing up the fragstore converter – this is a specialised piece of software we wrote for this, which will reformat the data into the new CAS-aware schema, and move the BLOB records into the filesystem. Conversion speed varies greatly based on the size of the individual assets being processed, but we’re hoping this will take between 9 and 15 hours to complete – we preconverted a large portion of the database already; but the exact speedup we will get from the preconversion is unknown.

Total estimated downtime is 21 hours – but this will hopefully be the last time we need to shut down the grid for such an extended period of time, as future asset moves will be able to be done transparently to grid users (Murphy willing.).

Update: OSGrid users – If you are reading this due to the twitter/osgrid notice, or the OSGrid news notice – the short of it is, we’ve converting the asset server. It’s going to take 24 hours, and we’re going to need to pull the grid down to do it. Downtime will begin approximately at 1AM Wednesday morning. It will take up to 24 hours to process the conversion – I’ll be posting information here on this blog as we proceed with updates.

For those just joining us on the OSGrid FragStore saga, read parts #1, and part #2 here.

Day 3: Conversion of the 1.5 million test assets has completed – and now we’ve got only the metadata in the table, I’m going to take some attempts at running queries. CAS storage eliminated slightly higher than my predicted 17% – coming in at around disk 23.49% savings on duplicated assets. Unfortunately the first bug has become apparent – creation date wasn’t preserved. I have patched the converter, so this will be filled when we do the final live conversion.

The following is the outputs of some queries I have run on this table, since it doesnt look like these take too long, we should be able to run these off the production server and update them in future. First, there are 15,957 groups of items with more than 5 objects using the same name in the system, of these, the top asset names are -

Returning to a query I was running yesterday on the CAS%s, the following is the updated chart with 1.5 million test assets loaded.

Fig 1. OSGrid CAS Duplicates (Day 3)

And if you have ever been curious about the distribution of assets in terms of asset type – here is a little chart showing the breakdown.

Fig 2. OSGrid Asset Quantities by Type

Full Conversion Date Set

We’ve decided to go ahead with the switchover this coming Wednesday, at 1AM PST. To go ahead with this, we are going to be bringing the grid offline for up to 24 hours. This hopefully will be the last time we need to go offline for more than 15 minutes – and also represents a move from our old asset hardware “Chenobyl” to our new server “Hindenburg” (which has a much larger hard disk capacity).