Underlying disk is a RAID1 SCSI array, so the speed there isnt too much of a problem. The hardware itself is fairly decent — but it’s our experience that MySQL just doesnt do ‘big data’ well.

One of the IBM engineers who used to work on data warehousing has said that Postgres will perform significantly better on GB/TB-sized tables – but ultimately there’s more at play than reliability & speed; decent partial distribution and clustering are going to be must haves this year.

MySQL should be able to handle the load fine, but you’ll have the same issues that you’d have with any other RDBMS.

There’s a few things I’d be curious to know about the current setup. What storage engine are you guys using? What is the underlying storage? How does the database get to the underlying storage (local, nfs, iscsi)?

At the end of the day, in this type of scenario whilst the database is 100% definitively the authorative source for asset information, success will be measured by what caching/expiry you do outside the database.

Your recent posts scaling are making me tempted to put up my hand and offer some help – let me know if you need anything

Postgres apparently isnt too bad here – by MySQL has shown it definetely does not like 200GB tables (we’ve had to recover it multiple times now – any any query against the table that isnt on the primary key will nuke the box for 2 hours). I have a feeling that any problems we have with MySQL right now will be significantly worse if we ever hit the same 300TB+ ranges Linden Lab do.

Re: on-disk performance, I don’t think that will be an issue yet, but it is a long term consideration – the big options I am looking at now are using some FUSE-based “filesystems” designed for distribution and clustering across multiple machines. KosmosFS looks promising, but there are a few others too.

I’d suggest as this dataset is only going to grow, you might be better off creating two seperate tables – one with the metadata (Filename, Checksum, AssetID, etc), and another with only three fields (Primary ID, Relational ID, Data).

I can also forsee using a traditional on-disk filesystem having some potentially devastating effects in this type of scenario also – for example, depending on the activity of the nodes pulling information down, are you going to have enough filehandles available on the operation system to serve up all your requests?

Either way, for this application – it might be beneficial to get yourself a copy of the MySQL query analyser, and enable the slow query log on the server so you guys can pinpoint which queries/tables need optimising for performance.

Thanks for posting and for notifying us in IRC.