An issue becoming increasingly apparent with the OpenSim platform is that user scripting is a hammer – it works fine if you only need a couple of them, but lots of them running in conjunction and the operating system begins to start running poorly. The common key to fixing this is to use something such as Fibres – small lightweight threads that perform better in mass numbers to full blown Operating System threads.

The problem is – there’s no cross platform way to access Fibres, and we have the additional problem that scripts under OpenSim should be transportable from one server to another – meaning the ability to pack up an execution context, move it to another machine, then resume is a feature we want to support.

Linden Lab’s Jim Purbrick wrote in 2006 about their work to use a modified version of Mono to achieve this – unfortunately with OpenSim we do not have the luxury of using modified runtimes, nor do we plan on dropping support for the official .NET runtime either. This means solutions such as Mono Continuations dont work for us very well.

(Update 26/11/08: I had a chat with Jim after showing him this article – apparently Linden is doing something fairly similar to this proposal already, without the aid of a modified runtime.)

The solution, is ideally something we can run on both .NET and Mono, runs reasonably quickly (but obviously some overhead is acceptable) – it needs the ability to save and restore the current execution state, and it needs the ability to suspend and resume. Suspension and resumation should be fairly fast to allow it to be used in a microthreading context.

These requirements unfortunately knock out a very large number of options – the options we considered ranged from using IEnumerator and ‘yield’ to handle threading (no save/restore or return values), to using reflection to emulate an instruction pointer and stack (too slow and complex).

The idea I am currently developing manages to avoid most of these problems, but requires a degree of runtime code mangling – the result however is something that is generally applicable to generic .NET applications that desire save/resume functionality (or microthreading).

The result centers around Mono.Cecil – the runtime analysis library written by one of the Mono developers. It has a very nifty feature however in that it is capable of changing a CIL stream, then writing back out the resulting assembly – meaning we can at the CIL level, begin to manipulate the results and hook in accessory functions to handle our save/restore worries.

Shadowing Variables

The first thing we need to consider trying to save the state of an in-execution script is that unlike Java, C# doesnt have a MethodInfo.LocalVariables.Get() method – to the C# programmer, the current Method’s stack is completely inaccessible. It does however, have the ability to list the local variables within this method.

And that’s where we enter the shadow variables – the goal of the shadow variables, is to dynamically build a Struct that contains a corresponding entry for each local variable within the method, at runtime we use Cecil to swap references to the local variables, to references to the Shadow equivilent – then saving and restoring becomes as simple as serialising and deserialising the shadow container.

Internally, the shadow container should be created before the method call is begun and passed in as an argument – for something that isnt resuming, it would simply be containing null values.

Resuming Execution

The next consideration is the ability to save and resume from an execution point within the code – this is handled through gratuitous use of the CIL equivilent of a ‘GOTO Label’. After each statement, we place a new label – and at the beggining of the method call, we also add a new argument (”InstructionPointerStartPoint”) and a switch statement that will ‘goto’ the appropriate label.

Since we’re passing the shadow container in via the arguments – we do not need to do anything special in terms of copying data back, as all the code references the shadow container directly ‘as is’. This basically emulates the instruction pointer within a standard x86 processor.

Handling Return Values and the Stack

The final consideration we need to make is recursive calls. Suspendable user methods are allowed to call other suspendable user methods – so we need to implement our own call stack manager to make sure we can save the context of a previous call leading to the current one. (Otherwise resumption would effectively fail).

The solution to this is thankfully fairly simple – we create a new ‘StackItem’ class which contains: A reference to the virtual instruction pointer (where we are currently in the execution), a reference to the function call for this method, and a reference to the current shadow container, as well as an optional reference to a decendant stack item for nested calls.

Before proceeding into a nested call, we need to create a new stackitem for it, and launch it within that execution context, but otherwise it falls together fairly simply.

By folding the stackitems together from the initial execution context, we’re able to then package up a copy of the running execution at any single point in time, transport it, then bring it back later. The whole thing ends up looking something like this.

Optimising

Under this situation, we do have some fairly expensive penalties when it comes to calling into a recursive function, but at the same point there are a number of relatively easy fixes. Functions for example can be inlined – only a very tiny number of functions (such as self-recursive functions) need to be handled fully. The vast majority can be inlined, thereby reducing the need to setup nested function calls in a large number of cases.