Update: OSGrid users - If you are reading this due to the twitter/osgrid notice, or the OSGrid news notice - the short of it is, we’ve converting the asset server. It’s going to take 24 hours, and we’re going to need to pull the grid down to do it. Downtime will begin approximately at 1AM Wednesday morning. It will take up to 24 hours to process the conversion - I’ll be posting information here on this blog as we proceed with updates.

For those just joining us on the OSGrid FragStore saga, read parts #1, and part #2 here.

Day 3: Conversion of the 1.5 million test assets has completed - and now we’ve got only the metadata in the table, I’m going to take some attempts at running queries. CAS storage eliminated slightly higher than my predicted 17% - coming in at around disk 23.49% savings on duplicated assets. Unfortunately the first bug has become apparent - creation date wasn’t preserved. I have patched the converter, so this will be filled when we do the final live conversion.

The following is the outputs of some queries I have run on this table, since it doesnt look like these take too long, we should be able to run these off the production server and update them in future. First, there are 15,957 groups of items with more than 5 objects using the same name in the system, of these, the top asset names are -

Returning to a query I was running yesterday on the CAS%s, the following is the updated chart with 1.5 million test assets loaded.

Fig 1. OSGrid CAS Duplicates (Day 3)

And if you have ever been curious about the distribution of assets in terms of asset type - here is a little chart showing the breakdown.

Fig 2. OSGrid Asset Quantities by Type

Full Conversion Date Set

We’ve decided to go ahead with the switchover this coming Wednesday, at 1AM PST. To go ahead with this, we are going to be bringing the grid offline for up to 24 hours. This hopefully will be the last time we need to go offline for more than 15 minutes - and also represents a move from our old asset hardware “Chenobyl” to our new server “Hindenburg” (which has a much larger hard disk capacity).

The following is my log of the OSGrid asset conversion saga.

Day 1: Today we’re taking a third attempt at doing the big fragstore conversion for OSGrid, for those not following the Saga of the Asset Server - about two months ago we started noticing major scalibility issues surrounding assets. Right now they are thrown into MySQL as a blob table, resulting in large amounts of waste both in duplicate content and in the fact we’re storing a filesystem inside of a relational database - you can read up on the earlier design decisions leading to FragStore here.

The previous two conversion attempts have suffered “mysterious MySQL glitches” which we assume may be related to various bugs with long running commands. Apparently the proper course of action when running a query that takes more than 60 seconds to process the command, is to freeze up entirely and stop processing requests - for now and evermore.

In an attempt to make this run a bit smoother - we’ve broken up the process into 2,000 batches of 1,000 assets each - previously our batch mechanism was using MySQL LIMIT X, Y which has the side effect of getting slower and slower as you progress down the table (thus causing the above); so we’ve shifted to using a numeric ID on the assets table. Putting the numeric ID on there allows us to at least index sequential accesses - LIMIT unfortunately will not use any form of index hinting.

mysql> ALTER TABLE `assets`
    -> ADD COLUMN `numericID`  int(11) UNSIGNED NOT NULL AUTO_INCREMENT AFTER `access_time`,
    -> DROP PRIMARY KEY,
    -> ADD PRIMARY KEY (`numericID`),
    -> ADD UNIQUE INDEX `assetID` (`id`);

Query OK, 1623826 rows affected (1 hour 11 min 2.54 sec)
Records: 1623826  Duplicates: 0  Warnings: 0

An hour and a bit later, the difference between the speed of processing before and after is pretty astounding

mysql> select id from assets limit 540000,10;
10 rows in set (11.53 sec)

mysql> select id from assets where numericID between 540000 AND 540009;
10 rows in set (0.42 sec)

It doesnt need a whole bunch of explanation to figure the above may help with our situation. Running the revised and simplified “AssetConverterMarkII” appears to go without a hitch - data is stored into the database, the metadata table is being filled correctly - all in all it appears to be functional. With one minor teeny little problem.

Only the first 4096 bytes of data are being written to the backend store. The remaining sectors of data are written - but consist entirely of zeros. Retrieving the data results in a buffer of the same length as the original stored asset - but often half the data is completely missing. An hour later, it looks like the data is being sent to the backend voldestore correctly, but either on the way there or on the way back, it loses something. Unfortunately it looks like the problem is outside the purview of the client adapter and is somewhere in the deep murk of the backend storage server.

Day 2: Rethinking time - after spending quite some time hunting for some alternatives, the simplest solution looks to be the best.

While I am keen to use Project Voldemort in the long term - in the short term debugging our implementation details are just not on my agenda. We use a IKVM cross-compiled connector library from Java, and the problem looks like it is sitting in there somewhere. Unfortunately debugging Java IKVM libraries from within .NET is painful at best, and not something easily fitting into our timescale.

The simplest solution is to throw the asset blobs onto the filesystem - filesystems are after all developed to handle tiny little files. Directories will slow down when there is more than about 3,000 entries within them, so we’re breaking storage up into “/b1/b2/hash.blob” - this means assuming an even distribution, approximately 30 files per directory at current size, scaling us up to a capacity of 100 million assets before we need to rethink the situation.

Distribution and redundancy are both things I am still keen to employ - putting us on the filesystem does allow us to look at things such as KosmosFS which provide transparent distributed filesystems on Linux, and also gives us the opportunity to look at commercial filestores down the road if we ever win the lottery.

Rewriting fragstore to use filesystem components where voldemort was employed took all of an hour and the asset converter was up and running - a lot faster too. Our conversion transfer rate on Voldemort was 66 assets per second. FragstoreFS?

10,000 Assets Processed (102.04 asset(s)/sec): 0 error(s) so far.

The second thing I wanted to test was just how big a savings we were getting from using Content Addressable Storage - with 10,000 processed, we ended up with 613 duplicates eliminated (6.1%). With 20,000 - 1390 (6.9%), with 90,000 - 8962 (9.95%). We’re hoping as the full dataset is processed - the % of duplicates eliminated continues to increase.

Fig 1. CAS Duplicate Savings

The next issue to present itself was a slowdown as the conversion occured - the number above (102.4) held firm for the first 10% of the conversion, then conversion speed began to massively taper off, first down to 71.39/sec, then down to 50.22/sec by 150,000 converted. My fears were a reprise of the situation we thought fixed on day 1 - slowdowns on accessing as we move further down the table.

Nebadon suggested that this infact might be actually because as OSGrid has become more popular the average size of an asset has increased over time - so we skipped a million rows down the table and started converting some of the later rows. Conversion speed? 68.31/sec. This indicates that yes, later assets are more expensive to process - but the conversion speed should still average the 60 or so per second we need to be able to convert the entire database in under 24 hours.

Appearing somewhat happy with the results, conversion on the complete database has started, but we wont know how well it has worked until tommorow.

Day 3: Stay tuned!

I thought it would be interesting to make some pretty charts representing the data behind OSGrid.org’s operations. The following is accurate as of today, but probably wont be tommorow. Some of the data is missing or has gaps in it - this is because during those periods I was not able to get accurate data and decided not to display it at all.

Servers

There are a total of 395 unique IP addresses connected to OSGrid, each roughly corresponds to a unique physical server hosting regions. Of these, 130 were compiled from SVN and give version information. (265 not reporting version).

The most popular version is r9332 with 35 unique servers running this revision. Next is a tie between the official 0.6.4 release and r9313 with 20 unique servers each. The remainder is distributed fairly evenly between r9307 and r9336. It is worth noting that version information was introduced in r9307 - so we cant quite yet see into the 60% that are running behind that version. What is interesting to note is however than 30% of the regions on OSGrid were updated with this revision already and indicates active upgrades and maintainence.

Fig 1. OpenSimulator Versions on OSGrid.org

Regions

At time of writing, there are 2,083 regions connected to OSGrid, owned by 386 unique avatars. (Averaging 5.3 regions per avatar) on 395 unique servers. Each server hosts on average 5.2 regions, the mode is 1, with 35% of servers hosting only a single region. The following chart displays the average number of regions per server over a range of values. Interestingly, this means that 1 in 7.5 users own their own region on OSGrid (compared to 1 in 600 in Second Life®)

Fig 2. Regions per Server on OSGrid

Users

The most interesting statistics are to do with users - currently there are 15,669 users registered on OSGrid.org. 20% of these users have logged in in the last week, 40% in the last month (75% in the last quarter). You can see the proportion of users who have logged in since a certain date on the following chart.

Fig 3. User logins since specified date

User Registrations are also fairly interesting to look at - OSGrid benefited enormously by Linden Lab’s Open Space pricing change announcement back in last November. Since the announcement, registrations on osgrid per day has doubled.

Fig 4. User Registrations per Day

Fig 5. Total User Registrations

It will be interesting to see if history repeats itself again when the next set of price increases occur in June.

Assets

A final note - I cannot make any charts on the asset table because, despite the presence of a creation date - running the query on our 2 million assets results in database meltdown. I can however give some quick figures on the asset table, there are 2,164,534 assets on OSGrid at present occupying a mere 65.4 GB (well less than a percentage of the Linden Lab asset cluster). Part of this is because unless an asset is in inventory, the central asset servers do not care to know about it.

If you are a coder and like the work I have been doing on MRM so far - I’m keen to accept any contributions you may have to the langage - patches on mantis are welcome (here’s how to submit). That being said, I am being somewhat restrictive with API-changing patches to ensure we have a consistent easy to understand language rather than a patchwork quilt of random features. The following is the list of rules I am using for language inclusions;

Coding Standards - The Rules for API Changes

The MRM API is designed to ‘feel’ like the official .NET API in as many ways as possible - this means it should be consistent and easy to understand. The same types and interfaces should be re-used in as many places as possible, duplication is to be strongly shunned.

1. Class Library Design and Naming

For core API (those items distributed with OpenSim), you must follow the Microsoft .NET Coding Standards for naming and class design.

2. Subclassing and Inheritance

Subclass (or subinterface) where possible to make a useful inheritence chain. The following is the current proposed structure for MRM entity inheritance.

Fig 1. IEntity Inheritance

Note that the above proposal is not set in stone - the alternative I am considering is inheriting IPlant from IObject, and making a new IPrimitiveObject - where we move prim related properties (such as Shape) into the subinterface.

3. “Superglobals”

You should insert items into the “Superglobals” appropriately. Superglobals are the [presently three] properties which are inherited from MRMBase. In the user-accessible language these are items such as “Host” or “World”.

3.1 Host Superglobal

Host means “Scripting Host”, items in Host should be directly related to the runtime environment the script is running in, such as performance characteristics, security permissions, etc. - it could almost be seen as analoguous to “System.Environment”.

Items placed into Host should be related to the runtime environment that is hosting the world itself. For instance, a database interface would be appropriate for ‘Host’ since it is unrelated to the world itself and is something that is dependent on the runtime environment.

3.2 World Superglobal

World refers to the SceneGraph - and should contain elements which are reflected inside the visual environment. For instance Avatars, Objects and other items which exist in the ‘World’. This also includes properties of the world itself, such as lighting conditions, sun angle, etc.

4. No Assumed Parameters

MRM API entries should not be designed with an assumed ‘target’ - that means that you should never ever assume you know the data the user will be processing unless it is directly connected to the object the user is calling the method on. Using an LSL function as an example, you should be able to quickly understand this rule.

BAD: llSetPos(vector3 pos); // Sets the position of the host object.
GOOD: llSetPos(IObject target, vector3 pos); // Sets the position of ‘target’

It should be noted that methods inside an object are free to act on themselves (eg IObject.Position is free to operate on the IObject instance), this only applies when referencing an outside object (such as say IAvatar.SitOn should have an IObject parameter and not assume which object to sit the avatar on).

5. Do not use OpenSim Types.

This means do not expose OpenSim internals to the MRM programmer - that means never return SceneObjectGroup, ScenePresence or any other internal representation to a MRM programmer. MRM provides replacements which take these as constructors in the form of IObject/IAvatar/etc’s implementors (SOPObject, SPAvatar, LOParcel, …)

If MRM does not implement your specific internal in an interface already, you must write an interface that can be passed to MRM programmers. The internal data should not implement the MRM “IAbcd” interface but instead a proxy should be written (see SOPObject for an example).

There is two reasons for this - the first is that passing internals means that any internal change will break MRM scripts. The second reason is security - security can be implemented inside the proxy whereas it cannot be implemented in the internal class.

6. Use OpenMetaverse.Types

libOpenmetaverse provides a number of useful types such as UUID, Vector3, etc. Use these whereever possible - for instance when representing U/V coordinates use “Vector2″.

7. Use the most appropriate Type for the job.

Do not use Vector3 to store only two values - use Vector2. There is a lot of types availible to use - make sure to pick the right one for the right job.

8. Do not copy LSL needlessly.

This is a big one - do not assume the LSL function set is an ideal to be upheld. Ask yourself when implementing a MRM replacement, “What is the using developer trying to do?”. Take the llSay, llShout, llWhisper, llRegionSay group - each implements the same underlying task, however the differing methods have a varying ‘range’ depending on which one is called.

For instance, llShout will ’say’ 100m. llSay will ’say’ 20m. A using developer is going to pick between them based on which range they need — but what if they need another range? Wouldnt it be smarter to have Object.Say(string message, int range)? Not only have you reduced the number of functions - but you have given the user more power.

9. Name Consistently

If you use a function name ‘SetFoobar’, the ‘GetFoobar’ should be called exactly that. Do not have ‘SetFoo/FooGet’, or ‘SetFoo/Foo’. (Note in practice, Set/Gets should always be implemented as C# properties.) To use LSL as another example - we should never end up with a pair like ‘llGetTextureOffset’ and ‘llOffsetTexture’.

10. Do not implement abitrary limits.

Copying from earlier, “If you have to, make them configurable. An example of this is the mandatory sleeps in LSL on functions like llEmail. Assume instead that people writing scripts in MRM know what they are doing.”

If you are implementing limits on particular MRM functions, make sure you have a damn good reason for it - you should assume MRM authors are region operators. If you implement specific limits, make sure you load them from OpenSim.ini.

“Safe” Areas to Contribute to

If the above has made you uncertain about contributing - there are a number of areas you can contribute safely towards, and they are pretty easy to find in the codebase.

• Look for “throw new NotImplementedException()” inside the code base - in classes such as SOPObject you can safely implement these functions and properties and nearly guaruntee your patch will get accepted.
• Suggestions - if you dont feel capable of writing a specific function, leave me a comment with something you want to see. Either myself or someone else may take it up and implement it.
• Documentation - I’m using XMLDOC to build the final API documentation, so inserting ///<summary> and other XMLDOC tags into the codebase is always appreciated. Make sure to do the commenting on the interfaces themselves and not their implementations (eg files starting with “I”).
• Tests and Examples - Post MRM scripts and write test scripts, these will eventually be rigged up to our internal testing framework, so the more functionality tests the better.

When running an infinite loop or other computationally expensive function, you will often want to release processing time back to the region to avoid causing undue ‘lag’. For instance - in a sim with 4,000 colour changing scripts, each will be trying to pummel the CPU simultaneously to get their updates onccuring.

What ends up happening is each will run for a fairly long period of time until broken up by a processing expensive “thread switch” at which point the next gets a long run before going to the next and the next and the next, or alternatively - none will finish and only one will be running.

The following diagram explains this in action - script A runs until there is an expensive thread switch, at which point script B will commence operation. You can manually force a thread switch through the use of the “llSleep(0)” statement in LSL, however the cost of switching threads is still present.

Fig 1. Normal Threaded Scripting

Microthreading allows you to say “You can pause and give time to another script” - in a way that is very lightweight and easy for the runtime VM. You manually insert breakpoints into the script which define points between the ‘tasklets’, at these points the VM can switch between runtime contexts without any penalty. As a result you can have thousands of them without problem; when running time is divided between all the running threads evenly - broken up at the tasklet borders.

The following diagram shows the same as above, in a microthreading environment.

Fig 2. Microthreaded Runtime

C# supports a limited form of microthreading via something called the “yield” keyword - and a lot of people have recognised you can use it to build a state machine — the basis of microthreading. Where we apply this to MRM is with some additional keywords.

The basic C# microthreading can be implemented in MRM without any additional work - however there is a significant manual cost in typing the construction of a microthreaded function. I have added two new keywords to MRM scripts today, which map to the C# yield statements.

The first is “microthreaded” - this keyword is inserted between the public/private/protected/internal keywords at the beggining of the function, and the variable type.For example, “private microthreaded void MyFunction(…) {”. Microthreaded functions can only work on functions with a no return type (void) - and internally it works by replacing “microthreaded void” with “IEnumerable”.

The second important keyword is “relax” - this keyword is defined as a breakpoint - an area where the VM can switch into another microthread without a penalty. There is a very small cost in placing these statements into your code, so consider placing them liberally. Place them inside of any loops or other routines which have a total processing time in excess of 2 milliseconds.

The relax keyword can only be placed inside of functions marked microthreaded - and you cannot call microthreaded functions directly (as in “Function(params)”). You must place microthreaded functions into the host scheduler marked as ‘Host.Microthreads.Run(myfunc(params));’

Another important note is that microthreaded functions will run asynchronously to your code, so do not expect “Microthreads.Run(…)” to block, consider it more a ‘Microthreads.ScheduleToRun(…)’. If you take these penalties into consideration however, these Microthreads can provide an excellent way to optimize your scripting and write coroutines relatively easily and quickly.

By default, the MRM VM will run 1,000 microthread tasklets per frame (or optimally 45,000 per second), this can be tweaked from inside of MRMModule.cs, however will eventually move into the OpenSim.ini [MRM] section.

The following is an example script that uses MRM microthreads.

Enjoy.

My first reactions to Google’s new O3D JavaScript API is “Thank God someone finally did it.”, a more detailed examination appears to yield something like Mozilla’s proposed Canvas3D but with practical implementations for all the major browsers and platforms (IE, FF, Win, Lin and Mac) - something Canvas3D was unlikely to get.

Fig 1. O3D Beach Sample

O3D has the advantage of being really the first proper 3D implementation for a browser to market (I won’t count Flash3D here since hardware accelleration is key to doing serious work with 3D.), it’s in ECMA/Javascript which makes it pretty accessible to a wide group of developers - and the API appears to be logical and powerful.

It’s backed by Google which will make overall deployment better - and I would be keen to wager that this endevour goes a lot further than Lively did. The runtime is even availible under the BSDL - which makes it very attractive to commercial users.

Could someone implement a virtual world with it?

I think for singleplayer games O3D is perfect - it’s easy to deploy, easy to write, uses standard file formats. Multiplayer - where VW’s come in is a much tougher nut to crack because there is still a dearth of reasonable communication standards for browsers.

AJAX is by communications standards not very powerful - it relies on the idea of a single thread carrying request/response packets. It can be hacked up into supporting threaded packeted communications - but I believe any virtual worlds using O3D will probably be those that can survive a larger latency (eg There.com(R) versus Second Life(R)) - worlds using it will use a lot more clientside processing to minimise connection loss.

The discussion on #opensim-dev this afternoon about O3D has been promising - it holds a lot of potential, and whether we can utilize it to write a simple browser plugin is going to be an interesting project for the next few days.

Graphically O3D has most of the niceties that a modern 3D engine expects (Shaders, etc) - the keys are going to be in the difficulty of writing scene and state management in Javascript, and making sure the communications systems are as simple and efficient as possible.

The ease of install and the lack of serious memory limits on the 3D scene means this may be a better option than Java3D for an accessible virtual world. Stay tuned.

As of r9240, MRM now has proper support for language extensions - these come in two key flavours - the first is the ability to import outside .NET DLLs which allow you to load foreign libraries into your code, this could take the form of more advanced math libraries or other useful tools. Those libraries will be loaded with the same security context as your MRM - so when CAS is implemented in MRM for security, you will need to be aware that they will be affected too.

The syntax for importing a library is fairly simple - just place within your code, below the //MRM:C# line, a line which reads like this:

//@DEPENDS:MyLibrary.dll

Dependencies are requirements - if the dependent DLL cannot be found, it will throw a compile error, even if no code utilizes this extension.

Accessing Region Modules

The second form of language extension - the preferable one is to connect with other OpenSim region modules. Internally in OpenSim when a region module wishes to communicate with another module, it is done via a shared interface - one module will define an interface it can be accessed by (eg, IMRMModule)

Communicating with an MRM is done in a very similar fashion - you must create an interface that the script can utilize to connect to your module. This is best illustrated with an example, so printed below is a “Hello World” module, it’s structure is pretty simple.

The Module Code

A sample MRM using the above

This code represents a fairly simple example - there are a few considerations to make however when writing MRM interfaces, especially if you desire to get those interfaces into the core OpenSim distribution.

Good practices when writing a module interface

These are based loosely on the rules I am employing in designing the default API.

• Use MRM types where possible - if you need to get an object as a parameter, utilize IObject rather than pass a UUID or localID. You can convert between them pretty easily (SOPObject takes LocalID as a singular constructor, and you can fetch it off via IObject.LocalID).
• Do not pass in internal OpenSim classes. This means do not ever return something like SceneObjectPart - this is because OpenSim internal classes change, and when they change your MRM will break. If you need to use one of these classes and MRM doesnt have a matching interface already, make a new interface and contact me for possible inclusion in the standard distribution.
• Do not require IHost for anything. If you want the user to pass in the host object, ask them to do it - but do not require it. This is because someone may find it useful to utilize your function on a remote object. By using IObject you make your module more useful to programmers.
• If you need a unique ID for the script instance, require a UUID to passed, then request the user uses the global variable “ID”. This is a UUID that corresponds with the internal Inventory Item UUID of the MRM script.
• Do not place abitrary limits on scripts. If you have to, make them configurable. An example of this is the mandatory sleeps in LSL on functions like llEmail. Assume instead that people writing scripts in MRM know what they are doing.
• Follow MSDN C# Coding Guidelines. We use the MSDN Class Library coding standard in OpenSim, MRM by extension uses the same guidelines.

Happy Extending!