Let’s start with the basics – the asset server is a gigantic document storage server. In our cases, documents are things such as primitives, textures, sounds, avatars – the works.

Each document is referred to by a specific unique filename called a UUID – which is a 128-bit number (think 3 with 38 zeros behind it – big number) – a UUID looks something like this “d61c1990-79b9-11de-8a39-0800200c9a66″ when represented in hexadecimal notation. You have probably seen them around a lot – just using OpenSim or Second Life.

A UUID is an excellent choice for a filename; because you can make them up randomly and statistically guaruntee it hasnt been used before (the chance of a UUID duplicate for a good random UUID is about 1 in a very very very very large number). This means you can have multiple asset servers on the one grid – each upload can be given an ID randomly, without the need for a central authority to give them out.

They also optimize extremely well – each “bit” is a single yes or no question; to find out exactly where an asset is located, you can play a game of 20 questions, but in this case, with 128 bits – you can ask 128 questions. For example; finding which server an asset is located on, in a server farm with 4 machines (labelled 1,2,3 & 4) could be found with the following question chain…

1. Is the asset on a server labelled closer to 4 than 1? (yes – servers 3,4 still possibilities)
2. Is the asset on an odd numbered server? (yes – located on server 3)

Asset found on server #3. It’s not very long is it? Even with say 300 servers; you can find the answer within 9 questions. Some further searching is needed to be done to locate the asset on the physical disk itself; but even then you will be asking a very small number of questions – far less than the maximum 128 allowed.

To put this into a diagram, fetching an asset directly by it’s indexed UUID is equivilent to something like this:

Fig 1. How asset UUIDs divide and conquer

Given that computers are capable of asking and answering questions very very quickly (several billion per second); and that none of the above questions require a central server; you can expand your asset server farm in a fairly linear fashion without scaling constraints; providing that you structure your data appropriately.

128 questions also means that you can define very very precise questions; so many that if you created 100 trillion assets every second; the sun would be lifeless before you ran out of address space that can be answered by that many questions. (Although a piece of statistical mathematics called the Birthday Paradox does make the address space significantly less usable the closer you get to that point – but for other reasons)

At this point, we know how assets are stored and retrieved – files are uploaded into the server farm, and given a UUID; that UUID can be used to find out where the asset is located precisely, and return the data later. However, when you access an item off the asset server; there’s only a modest chance you are accessing the asset by it’s UUID directly.

A lot of the time, you will be accessing the asset through a layer of redirection called the Inventory Server – if you have a texture within your inventory; there are actually two components; the first component is the data itself (the asset), and the second component is the inventory shell (uploader, time uploaded, permissions, reference to the asset, etc). When you lose an item of inventory (such as during a transaction); it is often not the asset servers fault, but instead the inventory server.

The inventory server exists so that if you give a copy of a texture to a friend; you and that friend do not make a duplicate copy of the heavier portion (the asset) – both inventory items, yours and your friends will have the same underlying asset ID. This also means that copying an item within your inventory will not prevent it from being saved during a asset glitch (as both point at the same data).

In Second Life®, the inventory server is using MySQL (and probably InnoDB), and the asset server uses IsilonFS (a custom hardware appliance based system.); in OpenSim this will vary a lot depending on the providers configuration – most grids use a SQL backend for storing both assets and inventory. By default, OpenSim will use a small embedded database for both Inventory and Assets – called SQLite. For grids, we recommend something more robust.

In grid mode, MySQL is still an appropriate solution for Inventory; however asset data will often exceed the normal operating tolerances of MySQL and lead to frequent table corruption, table locking issues (which in turn make performance suck) and other nasties. No large grid should be using MySQL for assets (small home and private grids are however fine.)

Grids such as ReactionGrid which use MS-SQL can get a bit further than MySQL when storing asset data (for inventory they are fundementally the same); MS-SQL like most sane database engines, store BLOB data seperately in a BSP-Tree indexed system; this means they can scale up pretty far (although at about the point of clustering, things degrade.); Postgres can do the same – however as best I am aware, Postgres is not yet properly supported in OpenSim.

For bigger grids like OSGrid (where we have hundreds of gigabytes of assets) a dedicated solution is ideal – unlike Second Life, we cant afford to purchase high end NAS-equipment for our storage solutions; so we have built a distributed system called Fragstore which has two big features.

1. A configurable backend – it can use distributed storage systems (such as Project Voldemort or Hadoop), or export to a filesystem (which can be a distributed filesystem via projects like KosmosFS.)
2. A duplicate detection system – so files uploaded twice only get stored once.

The duplicate detection is achieved via a secondary layer of indirection; when we allocate a UUID, we hash the incoming data (256-bit SHA1) and store the UUID and the Hash into a small database table (currently MySQL based); when we recieve future uploads; if the hash matches, the result is only stored once on disk.

The final point I would like to touch on is asset transmission and security – assets are transmitted in both SL and OpenSim over HTTP; the reason this works and is secure is because the UUIDs used in the requests are unguessable. Even by a computer making millions of guesses per second; the chance of hitting a valid address is very very small (same probability as generating a duplicate UUID).

From the asset server to the region server, the request is something in the form of http://assetserver.com/<uuid>/data – which will return the asset in a encoded container (usually packaging a little bit more information about the asset such as the content type, etc.) – using /metadata instead of /data will get you a JSON-encoded package with a bit more information about the asset (creation date, specific asset type, etc.)

From the region servers to the users – this transmission can vary a little. In both SL and OpenSim, the region servers act as proxy caches for the asset server; because assets cannot be updated (and instead are replaced) – this works fairly well. If twenty users in a region see a texture – it only needs to be fetched off the core asset service once; because the region will send it twenty times to the users; rather than twenty users hitting the core server.

In the case of OSGrid, this means we have 2,500+ reverse proxies sitting infront of the asset server (albeit in a somewhat suboptimal layout.); it reduces the bandwidth on the core asset server by approximately 90%+ (you can see our asset stats here); which means we can get away with much lower operating costs (since asset delivery costs are shared with region operators).

This is the problem with SL – the moment something is deleted, or a region is shut down – it’s almost always gone forever. If you wanted to revisit a earlier incarnation of Nexus Prime, or parts of SL’s ancient history – you are out of luck, as there’s a very solid chance those places are simply irrecoverable and lost to the sands of time. As digital data, there should be no good reason for this – disk space is cheap, and sims are small.

The classic case of this is the Bedazzle sims, among them were Gravity Space Station, Chinatown and UnrealSL – but none of them lasted more than between a few weeks or a few months. I’m digressing here slightly; but there are major projects I have been involved in SL that I would love to be able to still access that are gone forever (earlier versions of Aleph, the Atlas Underwater Complex, etc).

The problem isn’t so much that SL doesn’t store ancient rollbacks but that it is simply not possible to save a copy of one; even if you are the rights owner and want to back up your own work. Second Inventory can help here, but it too has flaws – it doesn’t have any kind of mass restore functionality; and it can only save inventory – there’s no chance to save the layout within a region, only the individual contents of it.

It is somewhat sad to see regions shut down by their owners for affordability reasons; knowing full well that the content cannot be ever easily restored later – I personally hate to see it when this happens, because something creative is lost forever.

OpenSim on the other hand, has some real advantages here – I have complete copies of a lot of my builds on OpenSim in varying stages of construction, courtesy of the Region Archive functionality. Every major construction project I have done on any of the grids is sitting somewhere on one of my hard disks as a .tar.gz file containing everything needed to reload it in later. In OpenSim, nothing is ever incapable of being saved – at all times you can dump a copy of the region to a disk, then reload it later somewhere else.

As a creator it’s fairly liberating – and convenient. I can work on a sim locally, export it, then import it into the production environment, and vice versa, take a production environment for local tweaks, edit it, then bring it back again.

Backing up Aleph

I have for a few months been testing an internal tool which allows you to export a OpenSim Archive from a Second Life Region – it was originally developed to export a clients region (their IP); but ended up being handy to preserve some of our workshops and builds from deletion when we closed the sims or rebuilt them. Today, I rewrote it – the previous version was based on the old libomv PrimWorkshop viewer, the new version is now based on the Simian Periscope (Periscope is a kind of multi-user version of GridProxy).

Before anyone asks, the modifications aren’t public – unfortunately for every legitimate user for a tool like this, there’s ten asshats prepared to use it as copybot deluxe, so the source is going to stay private (although I might release a binary version containing creator and permission checks similar to Second Inventory – we’ll see what my schedule looks like in the next few weeks).

This new version is overall a bit more reliable – a number of small bugs and niggles got fixed along the way – but the key factor is it’s now not a 2 hour effort to run, a region can be grabbed with 95%+ accuracy in minutes. You can see here, my personal workshop region ‘Aleph’ in Second Life – it’s a fairly old sim, but it’s gone through a ton of revisions in it’s history. The current revision is a sort of moonbase cross sandbox, complete with orbital lasers.

Aleph Null

Below you can see the same region and contents, but in my personal standalone OpenSim region. This one is located on my personal desktop – but with the same OAR file, I could just as easily reload it on any region running any version of OpenSim since OAR support was added. If I wanted to bring Aleph to OSGrid, it would take only as long as it took to copy and load the file on a region connected to OSGrid.

127.0.0.1:9000

There are limitations, the tool doesn’t copy any form of ‘deep inspection’ – so scripts, etc do not get saved. It’s theoretically possible however to back these up if you are the object owner, something I will be looking at in the future. Estate settings and a few other features aren’t in the v1 OAR format, so those also need to be recreated – but could be something we look at adding in future. It’s also worth noting that it will only backup content with a creator tag in a specified list; while that limit could be removed, it provides an easy way of assuring that you are backing up only content you have rights to.

The Wayback Grid

One of the projects I would like to see would be some kind of opt-in mechanism to subscribe to an automatic backup service similar to Archive.org for the web – the goal of which would be preservation of content in the long run. When someone drops their region, it would be nice to be able to restore it later on if they buy another region, or move to one of the open grids.

Ultimately, this could be taken to an extreme where you can dial back a grid in time, and see it as it was in a previous point in time – however the biggest limitations here are dealing with content prosciption. Most SL content is licensed with one or more restrictions – ethically ‘no copy’ is probably a blocker to performing backups, likewise ‘no transfer’ makes some implications about bringing something out of SL. To do this effectively, you need massive ‘opt-in’ by content creators to approve their content going outside.

The easiest starting point is instead probably to provide some kind of service for creators to voluntarily backup their sims (say, in the case of a creator shutting down their region – but wanting to preserve it). Perhaps there are options here to look at providing some kind of transfer service for people moving regions from say SL to OSGrid [providing they own the copyright]. If anyone is interested in that kind of service, let me know – this might be useful for folks contemplating migrating over to OpenSim/OSGrid and have all their own content.

If a tree falls in a forest and nobody can see it, does it still fire a collision event?

In Second Life(R) it does, and this is one of the reasons that the platform is so expensive – maintaining 20,000 regions requires 20,000 processor cores. Consider the WWW by contrast – a single server can power thousands of websites; because the cost of hosting is virtually free until someone accesses the site. Cost does not equate simulation space – it’s based on simultaneous concurrent accesses and the complexity of processing the requests.

OpenSim falls into the same trap in many ways – while the cost of hosting an empty region is nearly nil; there are two key aspects where processing will always continue even if there is no need or desire to do so. These areas are physics and scripting – arguably two of the larger expenses on the processor bill.

Physics can be limited somewhat – switching away from a persistent simulator (ODE) to something that computes physics ‘on demand’ (POS, BasicPhysics) will remove the expense of physics while no-one is in the region, but you lose the ability to have objects move independently. For those hosting geographical areas this may be perfect – if you don’t have any scripts in the region and compute physics purely on demand, you are looking at a raw cost in just memory – about 20mb per region, and even that can be paged to disk safely.

However, hosting conventional regions is a more tricky prospect – users have an expectation of certain features working, people use scripting and want to physically interact with objects. One of the options worth considering may be simply dialling the simFPS according to the number of viewers in the region. Drop down to 1Hz when there is no users, but dial back up to 45Hz when users appear – all easy enough to do within the codebase, however doesn’t reduce the processor cost to zero while inactive.

If we consider the ultimate goal being to reduce Virtual World running expenses to similar to that of a web-host: costs being per-access, measured in processor time, bandwidth, etc. then we need to go a few steps further. One of the bigger steps is reducing expectations – right now you can safely assume LSL will always be running, so therefor you can write things like “servers” in LSL. Servers are a good example of the problem – they are something that interacts with the world, but aren’t supposed to be part of it; yet they consume the expenses of being in the world all the same.

Servers could be much better replaced by simple web scripts or server daemons on a normal server – which have a decent API that can connect to the world and make their interactions without needing to be part of it. By doing so, you not only make the servers more efficient (Apache+PHP is faster and more powerful than LSL), but you reduce the persistent load on the world itself.

The work I am doing on MRM is addressing part of this problem – redefining the API to reduce the number of scripts you need in the world, but a total solution here includes some kind of “remoting-style” API that lets you send messages and interact with the world, from a completely independent outside authority. The other side of the same coin is reducing ‘background noise’ processing – timers, sensorrepeats and other recurring tasks all place a small but visible background load on the server, all of these events will keep the server processing even if there is nothing to do.

Some things may require this – but encouraging people to think about when they need these events is better. Limitations on the LSL API really prevent this in SL, but in OpenSim would you be willing to use a “TimerIfAvatarInRegion” instead of a plain Timer? If you could – then it would allow the back-end server to attach conditions to your scheduler which provide optimisations when no-one is there. Allowing these kinds of optimizations is going to be key in the long term – because a cost of one processor core per four concurrent users is just simply insane and will never be adopted in the wider marketplace VW operators hope to gain. If webservers could only handle four simultaneous users – Google would require millions of servers to operate instead of just thousands.

While RealXtend have managed this feat a good year ago, it would be nice to get this feature into the mainline standard viewer. Now sure, going by their track record Linden are probably going to manage to cock it up and implement it badly (Can you say “Nonstandard Custom Format” or “Stupid limitations”?), but a feature like that is going to be incredibly useful even in a broken state (Look what’s been achieved with sculpties for example).

It also has some real benefits for content creators – first the largest is it permits real world artists with real world skills to use those skills effectively. Second big reason is it adds the capability to use well developed editing suites with Second Life seamlessly – this is going to dramatically ‘raise the bar’ on content quality and potentially bring SL into line with games only two years old (from the 5-6 it is now).

The first argument I’m going to hear against this no doubt is “By allowing professionals to create content for Second Life effectively, the inworld tools are going to be worthless” – which is basically saying “Let’s keep the tools retarded.” – the fact is the SL building tools are not the best, and by adding complementary support for out world editors, it means that content can be designed both more efficiently and with better visual quality. If the implementation is done similar to the RealXtend implementation, it will end up being like sculpties – you can manipulate them in world, but the core shapes are developed out world.

Really the biggest argument above is a simple case of protectionism – and like real world protectionism, the people hurt the most are going to be the people who don’t adapt and see their countries (platforms) lose economic growth to competitors. Sculpties didn’t kill builders, and neither will meshes, but yes you will probably need to learn a new skill or two.

The next one will be “This will cause additional lag” – the irony here is that it actually will do the inverse. Mesh objects are significantly more efficient computing-wise than primitives. Not only can you cluster texture space significantly more efficiently using UV maps, but you can additionally render less triangles with more detail. In terms of network downloads – it’s not going to be a case of one primitive vs one mesh, you are comparing lots of primitives vs one mesh, and like textures – you can cache them effectively.

As an ammendum to that – people often bring out the idea of million polygon meshes. The answer here is that meshes can be artificially limited on upload (lock em to a certain filesize or number of polygons) – the ultimate irony here is however that primitives are no better – the average second life furry avatar fully rendered is around ~100,000 triangles, or approx 2x that of the average complete Doom 3 scene.