When running an infinite loop or other computationally expensive function, you will often want to release processing time back to the region to avoid causing undue ‘lag’. For instance - in a sim with 4,000 colour changing scripts, each will be trying to pummel the CPU simultaneously to get their updates onccuring.

What ends up happening is each will run for a fairly long period of time until broken up by a processing expensive “thread switch” at which point the next gets a long run before going to the next and the next and the next, or alternatively - none will finish and only one will be running.

The following diagram explains this in action - script A runs until there is an expensive thread switch, at which point script B will commence operation. You can manually force a thread switch through the use of the “llSleep(0)” statement in LSL, however the cost of switching threads is still present.

Fig 1. Normal Threaded Scripting

Microthreading allows you to say “You can pause and give time to another script” - in a way that is very lightweight and easy for the runtime VM. You manually insert breakpoints into the script which define points between the ‘tasklets’, at these points the VM can switch between runtime contexts without any penalty. As a result you can have thousands of them without problem; when running time is divided between all the running threads evenly - broken up at the tasklet borders.

The following diagram shows the same as above, in a microthreading environment.

Fig 2. Microthreaded Runtime

C# supports a limited form of microthreading via something called the “yield” keyword - and a lot of people have recognised you can use it to build a state machine — the basis of microthreading. Where we apply this to MRM is with some additional keywords.

The basic C# microthreading can be implemented in MRM without any additional work - however there is a significant manual cost in typing the construction of a microthreaded function. I have added two new keywords to MRM scripts today, which map to the C# yield statements.

The first is “microthreaded” - this keyword is inserted between the public/private/protected/internal keywords at the beggining of the function, and the variable type.For example, “private microthreaded void MyFunction(…) {”. Microthreaded functions can only work on functions with a no return type (void) - and internally it works by replacing “microthreaded void” with “IEnumerable”.

The second important keyword is “relax” - this keyword is defined as a breakpoint - an area where the VM can switch into another microthread without a penalty. There is a very small cost in placing these statements into your code, so consider placing them liberally. Place them inside of any loops or other routines which have a total processing time in excess of 2 milliseconds.

The relax keyword can only be placed inside of functions marked microthreaded - and you cannot call microthreaded functions directly (as in “Function(params)”). You must place microthreaded functions into the host scheduler marked as ‘Host.Microthreads.Run(myfunc(params));’

Another important note is that microthreaded functions will run asynchronously to your code, so do not expect “Microthreads.Run(…)” to block, consider it more a ‘Microthreads.ScheduleToRun(…)’. If you take these penalties into consideration however, these Microthreads can provide an excellent way to optimize your scripting and write coroutines relatively easily and quickly.

By default, the MRM VM will run 1,000 microthread tasklets per frame (or optimally 45,000 per second), this can be tweaked from inside of MRMModule.cs, however will eventually move into the OpenSim.ini [MRM] section.

The following is an example script that uses MRM microthreads.

Enjoy.

An issue becoming increasingly apparent with the OpenSim platform is that user scripting is a hammer - it works fine if you only need a couple of them, but lots of them running in conjunction and the operating system begins to start running poorly. The common key to fixing this is to use something such as Fibres - small lightweight threads that perform better in mass numbers to full blown Operating System threads.

The problem is - there’s no cross platform way to access Fibres, and we have the additional problem that scripts under OpenSim should be transportable from one server to another - meaning the ability to pack up an execution context, move it to another machine, then resume is a feature we want to support.

Linden Lab’s Jim Purbrick wrote in 2006 about their work to use a modified version of Mono to achieve this - unfortunately with OpenSim we do not have the luxury of using modified runtimes, nor do we plan on dropping support for the official .NET runtime either. This means solutions such as Mono Continuations dont work for us very well.

(Update 26/11/08: I had a chat with Jim after showing him this article - apparently Linden is doing something fairly similar to this proposal already, without the aid of a modified runtime.)

The solution, is ideally something we can run on both .NET and Mono, runs reasonably quickly (but obviously some overhead is acceptable) - it needs the ability to save and restore the current execution state, and it needs the ability to suspend and resume. Suspension and resumation should be fairly fast to allow it to be used in a microthreading context.

These requirements unfortunately knock out a very large number of options - the options we considered ranged from using IEnumerator and ‘yield’ to handle threading (no save/restore or return values), to using reflection to emulate an instruction pointer and stack (too slow and complex).

The idea I am currently developing manages to avoid most of these problems, but requires a degree of runtime code mangling - the result however is something that is generally applicable to generic .NET applications that desire save/resume functionality (or microthreading).

The result centers around Mono.Cecil - the runtime analysis library written by one of the Mono developers. It has a very nifty feature however in that it is capable of changing a CIL stream, then writing back out the resulting assembly - meaning we can at the CIL level, begin to manipulate the results and hook in accessory functions to handle our save/restore worries.

Shadowing Variables

The first thing we need to consider trying to save the state of an in-execution script is that unlike Java, C# doesnt have a MethodInfo.LocalVariables.Get() method - to the C# programmer, the current Method’s stack is completely inaccessible. It does however, have the ability to list the local variables within this method.

And that’s where we enter the shadow variables - the goal of the shadow variables, is to dynamically build a Struct that contains a corresponding entry for each local variable within the method, at runtime we use Cecil to swap references to the local variables, to references to the Shadow equivilent - then saving and restoring becomes as simple as serialising and deserialising the shadow container.

Internally, the shadow container should be created before the method call is begun and passed in as an argument - for something that isnt resuming, it would simply be containing null values.

Resuming Execution

The next consideration is the ability to save and resume from an execution point within the code - this is handled through gratuitous use of the CIL equivilent of a ‘GOTO Label’. After each statement, we place a new label - and at the beggining of the method call, we also add a new argument (”InstructionPointerStartPoint”) and a switch statement that will ‘goto’ the appropriate label.

Since we’re passing the shadow container in via the arguments - we do not need to do anything special in terms of copying data back, as all the code references the shadow container directly ‘as is’. This basically emulates the instruction pointer within a standard x86 processor.

Handling Return Values and the Stack

The final consideration we need to make is recursive calls. Suspendable user methods are allowed to call other suspendable user methods - so we need to implement our own call stack manager to make sure we can save the context of a previous call leading to the current one. (Otherwise resumption would effectively fail).

The solution to this is thankfully fairly simple - we create a new ‘StackItem’ class which contains: A reference to the virtual instruction pointer (where we are currently in the execution), a reference to the function call for this method, and a reference to the current shadow container, as well as an optional reference to a decendant stack item for nested calls.

Before proceeding into a nested call, we need to create a new stackitem for it, and launch it within that execution context, but otherwise it falls together fairly simply.

By folding the stackitems together from the initial execution context, we’re able to then package up a copy of the running execution at any single point in time, transport it, then bring it back later. The whole thing ends up looking something like this.

Optimising

Under this situation, we do have some fairly expensive penalties when it comes to calling into a recursive function, but at the same point there are a number of relatively easy fixes. Functions for example can be inlined - only a very tiny number of functions (such as self-recursive functions) need to be handled fully. The vast majority can be inlined, thereby reducing the need to setup nested function calls in a large number of cases.

This one comes up a lot - I hear it in quite a few places “Well, shame it’s in C#.”, it’s usually followed by some nebulous statement about Microsoft, followed by patent threats, embrace extend extinguish, etc etc.

So let’s start with the basics -

• C# is a ECMA and ISO standardized language. It went through the review procedures of ECMA and ISO in a standard fashion (unlike OOXML). Download the ECMA Spec here and the ISO Spec here.
• Yes, the base library is included in that standardization. The primary exceptions are, ADO.NET, ASP.NET, Web Services and Win Forms. Many of these exceptions are however separately covered via Microsoft’s Open Standards Patent program, although we’ve moved to avoiding them entirely in the standard release. (Extensions which use them are however available on the third-party utilities forge)
• The ECMA standard also includes the CIL byte code format.
• If Microsoft decide to add new extensions onto .NET (which they have done with every major release), the OpenSim developers are content to wait until those extensions are available under Mono (which moves fast enough that it isn’t a major problem).
• Microsoft is not known for breaking backwards compatibility to “extinguish” things - the mere fact you can still run Win3.1 applications on Windows Vista should give some assurances there. Microsoft is yet to make any kind of retroactive change to the .NET standard - all .NET 1.1 applications should still run under .NET 3.5 without changes. (and the cost to them of making a change like that would be significant in terms of people using applications under Windows)
• There are two completely F/OSS implementations of C#/.NET and the standard library. Mono (Licensing Concerns Addressed Here) and the FSF’s DotGNU PNET. OpenSim is regularly tested for compatibility under Mono (in fact our automated testing environment uses it). DotGNU is significantly less popular and has not been properly tested with due to not being feature complete yet. There is also a third source-availible implementation from Microsoft called Rotor, however it is not under a OSI-approved license.
• Microsoft maintains a reasonably healthy relationship with the Mono developers and has been known to collaborate in the past (such as for the development of the specialised Moonlight runtime).

The next question usually is “Well, why not write it in Java then?” the answer is multi pronged and highly likely to generate a flame war on the subject - the biggest reason is that coding the same thing in Java would probably take significantly longer to do.

Java is a beast of a language that has had layers of gunk added in every revision resulting in a hodge-podge of inconsistently named items in the standard library that may, or may not address what you want. The second major reason is that the C# standard library is both larger and more functional - the amount of time and effort the Base Class Library has saved is astonishing. Wikipedia has a nice article detailing the differences between the two languages.

However, since both Java and C# share a very similar byte code language - it is possible to do a machine driven cross-compilation so you could run OpenSim under the JVM runtime if you so wish. Source translation between the languages is also reasonably possible however requires a degree of manual work.

Some concluding facts which may actually surprise some people

• One of the largest contributors and users of the project is IBM. One of the first groups inside of IBM to get involved was IBM’s Linux Technology Center. All[?] of the IBM developers are using Linux, Emacs and Mono to develop and test.
• Approximately half the developers working on OpenSim are running under Linux/BSD. As a user base, Linux users represent approximately 80-90% of the casual testers and operators. (Windows is however much better represented in people running commercial operations)
• Our compatibility targets are Mono 1.2.5 (latest is 1.9) and .NET 2.0 - we dont use features which supercede these (although we may raise that to Mono 2.0 once it is out of beta).

Every now and then, this hits me. I fire up OpenSim (or RealXtend), and it crashes instantly with a “BadImageFormatException” - when you see this, you know you have a compatibility problem with an embedded C/C++ binary. In english, 90% of the time this means you are running under a 64-bit environment with a mix of 64 and 32bit code trying to run together.

OpenSim is 64-bit aware

The reason for this happening is that OpenSim by virtue of the .NET platform is quite 64-bit aware, this means natively it can address 16TB of memory just fine if run on a 64-bit operating system. (nb, under Mono you need to make sure Mono is 64bit to get the same benefits).

However unfortunately, some of the libraries we link against have to be compiled ‘one way’ or the ‘other way’ and not both.  When running a 64-bit application, things such as memory pointer lengths are difference, so there is no way to cleanly pass data between a 32-bit and 64bit application - everything must be one or the other.

So how do you fix this?

There’s two routes, the easy route, and the proper route. The easy route is good for 90% of our users - if you dont need to access more than 4GB of memory and dont mind about the slight slowdown when handling double precision numbers, then just run 32-bit.

How? Instead of launching OpenSim.exe launch OpenSim.32BitLaunch.exe instead - this forces OpenSim to run under a 32bit environment (under windows provided by WoW32).

The more detailed fix

The next option is to strip out the sections that are causing problems. OpenSim is modular - so you can swap bits and pieces with their alternatives. The following are components that are not 64-bit compatible by default (however it is possible to fix this, something I will get to in a moment.)

Incompatible Components List

Physics - most problems come from our Physics Engines, since these tend to be written in high performance C/C++ with ASM littered. The following list of physics engines by default are NOT 64-bit compatible.

• OpenDynamicsEngine - The ODE DLL we provide is compiled for a 32bit operating system, however you can fetch the sources for our custom ODE DLL from (SVN) http://opensimulator.org/svn/opensim-libs/ and build it yourself on your native environment.
• AGEIA PhysX - The AGEIA PhysX support (now Nvidia PhysX) relies on a natively compiled 32-bit DLL. At present there is no fix here.

Compatible Engines

• BasicPhysics - This is purely managed but very simple in operation. Collisions are provided only between the avatar and the ground here. This runs without problems regardless of environment
• POSPhysics - POS physics is a more complicated BasicPhysics engine with bounding box collisions between objects. For some tasks this is sufficient.
• BulletX - Our own custom BulletX library is derived from the BulletXNA project, this is a properly developed fully managed physics engine that does proper collisions (however the friction coefficient can be a little low for some users).

Storage Engines - At least one of our storage engines relies on some hard coded components, it’s listed below

• SQLite - Whaaa? SQLite is broken on 64-bit environments? Afraid it’s true. Unfortunately on Windows systems the only solution is to recompile the entire of Mono (including Mono.SQLite.dll) yourself under a 64-bit flag.

Compatible Engines

• MySQL - The DotNetConnector provided by MySQL is 100% compatible and probably the recommended adapter for production installs.
• MSSQL - Windows-only however MSSQL is also free and the adapter is built into .NET itself so is pretty much guarunteed compatible.

Other incompatible components

Perenially we have problems with the following components in addition to the above, this includes libsecondlife’s OpenJPEG version - this can be fixed by compiling libsecondlife and OpenJpeg (using make) on the target system. This may have been fixed recently however as complaints about this DLL have lessened sharply.

Running

Once you have swapped incompatible components out, or recompiled them appropriately, OpenSim should run under a 64-bit environment natively on Windows. As mentioned above, under Linux/Mono you will need to make sure that Mono has been compiled as a 64bit application for this to take effect.

The differences between the two are mostly marginal, however there are some definitve improvements in certain mathematical loops and stability when using large amounts of memory. In production environments, running 64-bit mode can be an attractive option, however will require constant maintainence to compile the above libraries as new releases occur.

There’s been a rush of discussion recently about embedding virtual world technologies in the browser, I have seen two Java3D based implementations (which are very clever and deserve a closer look - especially if either is Open Source).

Tish has written up an interview with Avi who I mentioned earlier, on 3D in the Browser, the technologies and where things should be going - he’s very eloquent and I agree with pretty much everything he has to say. Avi added a companion piece with just a table of definitions and a small commentary on the article, which leads me to add my own comments on things.

First - the best browser is one that we dont need to install much onto the clients system, because the hurdle of making a plugin popular is a big one. XBAP is great for this since you dont need to install it to run it - it runs on a sandbox on it’s own just fine.

The downside to this is that we lose things like Local Caches which dramatically increase the load times when frequenting common areas - it would be nice to be able to optionally “Preview, then install” something we might be able to do via the XBAPs (one mode restricts to sandbox, other presumes it’s installed). Some experimentation in installing easily is going to need to be done.

I’ve been doing some further experimentations with XBAP (and yes, I’ll be posting the latest Xenki code shortly), and discovered some interesting things. First - 3D performance is no worse than a standalone WPF application, Second - it runs on Windows 2000/XP and up - I previously assumed it was Vista only, and was pleasantly suprised to find out this is not the case. Third - getting security certificates needed to automatically launch directly without install actually wont be too painful afterall (as long as libopenmv can get signed, we’re good.).

Cross Platform, Java3D?

The cross platform issue still remains elusive, which is why Java3D has caught my eye recently - given the similarities between C# and Java, it strikes me as potentially useful to be able to take large chunks of Xenki, run it through a special compiler and produce something that will run on Linux and Mac too.

The question becomes: why not use Java directly and skip the C#/WPF bit? Well the answer here is somewhat a personal one, first is that WPF is a lot easier to work with - this is an unfortunate fact of life, but doing things like UI ontop of 3D frames and doing it cleanly and efficiently is something that WPF has done so much better that there is almost no competition between them.

I personally have a preference towards doing things as quickly and cleanly as possible, worrying about making it functional rather than working on structural work that has already been done before - however if someone has already produced a good embeddable engine for Java on 3D pages - I’d be very tempted to at least try and give it a shot.

Java3D does also have the downside that it still sits in a sandbox, and the sandbox is mandatory - there’s no way we can implement a local cache using it to the best of my knowledge, I personally feel that the installable option is a very good way to cross the barrier between ‘hosted’ and ‘installed’ once the user is familiar.

Updated Xenki Sources

As stated above - I’m going to push the new xenki sources out in the next 24-48 hours. Watch this space as I will post the URLs and also the SVN address where I will be keeping changes.