We’ve been experimenting with Vivox, so the entire presentation was streamed via the new VivoxVoiceModule; running on the latest trunk code for OpenSimulator; both for the most part performed pretty admirably. A lot of the improvements from IBM and Intel lately into OpenSim’s packeting have been paying dividends. Mono’s previous concurrency “record” was about 40 users (for comparison .NET’s current record is 65).

So, what’s in store in the future and near-future for OSGrid? I had a couple of announcements at the opening presentation.

First – Hello General Store

The general store is the first “webshop” based on OpenSim that integrates directly into the grid itself. Listing of items, setting of permissions, etc are all done from within the viewer – directly integrated with your inventory itself.

This has one major advantage outside of the improved convenience – it means you do not need a region to mediate the transaction. Every transaction can be done directly on the inventory server (infact this is exactly what we do.); but what does this exactly mean?

When you create your general store – a special undeletable folder is created in your inventory called ‘My General Store’, items placed into this folder that are marked ‘Allow anyone to copy’ are listed on our website automatically, providing you are the creator of the item in question (and have copy/trans permissions on it).

Purchasing of items will have them sent to your inventory directly via the inventory server (although you will need to relog after a shopping trip for the items to appear.); saves a bit of time and effort doing the transfers – and you dont lose a transfer because you were in busy mode.

User Achievements

Many of you spotted our new achievements system located on the OSGrid website – it’s mostly a bit of fun right now (and probably will always be), but we are going to be opening achievements up to region operators across the grid. If you have something open to the pubic (some form of competition, game or other activity that is ‘achievement-worthy’) you can offer achievements as rewards.

The exact process is yet to be exactly finalised – but you will recieve a bit of LSL at the end you can include which allows you to award your achievements to specified users. There will be some caps on who is eligable to be able to award achievements (particularly so that people do not just set them up for friends)

“More information soon”

Acknowledgement of Direction

OSGrid has traditionally been a test grid for OpenSimulator – that has been good and bad – in our case however, we’ve definetely moved on from those roots – OSGrid today is now more of a social space than a testing one. As a group, the admin team has decided to acknowledge this as part of OSGrid’s long term direction – and include those aspects in our mission statement (while preserving our current roles too).

While the testing aspects will always be present, we’re going to be focusing on making the social aspects more useful; such as improving our events calendars and website integration features. We would definetely like to see more live events – and will help promote any regular events to our wider communities.

Some of these changes will be more profound, which is actually tied to…

Continents

We are going to be introducing two new continents into the OSGrid ecosystem; so there will be three major continents in total. For lack of a name so far, they will be ‘Terra Alphus’, ‘Terra Betus’ and ‘Terra Gammus’ (subject to change). The current continent centered around 10,000×10,000 is ‘Betus’; and will not be changing.

Alphus will be the most structured of the continents – and subdivided into three zones based on the server geography – US, Europe and Asia. Regions connected to Alphus need to be approved by a small committee, and subject themselves to a few construction limits. Alphus will:

• Consist only of regions on dedicated server hosts (or good quality VPSs)
• Have a reasonable uptime (that is they are significantly likely to be online)
• Follow a master terraforming pattern.

Regions in Alphus will be organised so that servers located in the US are grouped with other US servers; EU with EU and Asia with Asia — meaning border crossing between Alphus regions is a lot more likely to be successful. The goals with the Alphus area is to ensure a more consistent user experience [especially for new users] and can be thought of as a ‘grid within a grid’ of sorts.

Betus is the current continent – it’s not going to be touched (other than perhaps by a few regions moving into Alphus); the current strategy of nearly-anything-goes is not changing here. You can place your own regions in this section without any considerations at all.

Gammus is the final section, and also a new continent – this will be tied to our new highly experimental region launcher. Regions hosted from home using our launcher will end up here. Because these regions are less likely to be online at any given moment, we are conglomerating them together away from other regions – so that the other regions on the grid are not affected by constant neighbour checks.

In summation:

• Your current regions arent changing or moving. If you want to setup a new region on any random coordinates, that’s fine too.
• The new continents will be fairly close to each other in coordinate terms (within 500 coordinates; or within reasonable scroll distance)
• The goals of the introduction of the two other continents are to improve user experience (especially with border crossing, etc.)
• This is not a grading system; Alphus regions are not better than Gammus – only that Alphus voluntarily comply with a tighter set of user experience restrictions.

New Orientation Sims

We’re going to be working on some new orientation island sims with OSGrid; particularly to aquaint people with what OSGrid is and isnt. These are going to be seperate from our welcome area (LBSA), but we are going to look at ways we can integrate them together (such as providing a chat bridge with a helpers group, etc.)

Again, more information coming soon.

The problem is – there’s no cross platform way to access Fibres, and we have the additional problem that scripts under OpenSim should be transportable from one server to another – meaning the ability to pack up an execution context, move it to another machine, then resume is a feature we want to support.

Linden Lab’s Jim Purbrick wrote in 2006 about their work to use a modified version of Mono to achieve this – unfortunately with OpenSim we do not have the luxury of using modified runtimes, nor do we plan on dropping support for the official .NET runtime either. This means solutions such as Mono Continuations dont work for us very well.

(Update 26/11/08: I had a chat with Jim after showing him this article – apparently Linden is doing something fairly similar to this proposal already, without the aid of a modified runtime.)

The solution, is ideally something we can run on both .NET and Mono, runs reasonably quickly (but obviously some overhead is acceptable) – it needs the ability to save and restore the current execution state, and it needs the ability to suspend and resume. Suspension and resumation should be fairly fast to allow it to be used in a microthreading context.

These requirements unfortunately knock out a very large number of options – the options we considered ranged from using IEnumerator and ‘yield’ to handle threading (no save/restore or return values), to using reflection to emulate an instruction pointer and stack (too slow and complex).

The idea I am currently developing manages to avoid most of these problems, but requires a degree of runtime code mangling – the result however is something that is generally applicable to generic .NET applications that desire save/resume functionality (or microthreading).

The result centers around Mono.Cecil – the runtime analysis library written by one of the Mono developers. It has a very nifty feature however in that it is capable of changing a CIL stream, then writing back out the resulting assembly – meaning we can at the CIL level, begin to manipulate the results and hook in accessory functions to handle our save/restore worries.

Shadowing Variables

The first thing we need to consider trying to save the state of an in-execution script is that unlike Java, C# doesnt have a MethodInfo.LocalVariables.Get() method – to the C# programmer, the current Method’s stack is completely inaccessible. It does however, have the ability to list the local variables within this method.

And that’s where we enter the shadow variables – the goal of the shadow variables, is to dynamically build a Struct that contains a corresponding entry for each local variable within the method, at runtime we use Cecil to swap references to the local variables, to references to the Shadow equivilent – then saving and restoring becomes as simple as serialising and deserialising the shadow container.

Internally, the shadow container should be created before the method call is begun and passed in as an argument – for something that isnt resuming, it would simply be containing null values.

Resuming Execution

The next consideration is the ability to save and resume from an execution point within the code – this is handled through gratuitous use of the CIL equivilent of a ‘GOTO Label’. After each statement, we place a new label – and at the beggining of the method call, we also add a new argument (”InstructionPointerStartPoint”) and a switch statement that will ‘goto’ the appropriate label.

Since we’re passing the shadow container in via the arguments – we do not need to do anything special in terms of copying data back, as all the code references the shadow container directly ‘as is’. This basically emulates the instruction pointer within a standard x86 processor.

Handling Return Values and the Stack

The final consideration we need to make is recursive calls. Suspendable user methods are allowed to call other suspendable user methods – so we need to implement our own call stack manager to make sure we can save the context of a previous call leading to the current one. (Otherwise resumption would effectively fail).

The solution to this is thankfully fairly simple – we create a new ‘StackItem’ class which contains: A reference to the virtual instruction pointer (where we are currently in the execution), a reference to the function call for this method, and a reference to the current shadow container, as well as an optional reference to a decendant stack item for nested calls.

Before proceeding into a nested call, we need to create a new stackitem for it, and launch it within that execution context, but otherwise it falls together fairly simply.

By folding the stackitems together from the initial execution context, we’re able to then package up a copy of the running execution at any single point in time, transport it, then bring it back later. The whole thing ends up looking something like this.

Optimising

Under this situation, we do have some fairly expensive penalties when it comes to calling into a recursive function, but at the same point there are a number of relatively easy fixes. Functions for example can be inlined – only a very tiny number of functions (such as self-recursive functions) need to be handled fully. The vast majority can be inlined, thereby reducing the need to setup nested function calls in a large number of cases.

So let’s start with the basics -

• C# is a ECMA and ISO standardized language. It went through the review procedures of ECMA and ISO in a standard fashion (unlike OOXML). Download the ECMA Spec here and the ISO Spec here.
• Yes, the base library is included in that standardization. The primary exceptions are, ADO.NET, ASP.NET, Web Services and Win Forms. Many of these exceptions are however separately covered via Microsoft’s Open Standards Patent program, although we’ve moved to avoiding them entirely in the standard release. (Extensions which use them are however available on the third-party utilities forge)
• The ECMA standard also includes the CIL byte code format.
• If Microsoft decide to add new extensions onto .NET (which they have done with every major release), the OpenSim developers are content to wait until those extensions are available under Mono (which moves fast enough that it isn’t a major problem).
• Microsoft is not known for breaking backwards compatibility to “extinguish” things – the mere fact you can still run Win3.1 applications on Windows Vista should give some assurances there. Microsoft is yet to make any kind of retroactive change to the .NET standard – all .NET 1.1 applications should still run under .NET 3.5 without changes. (and the cost to them of making a change like that would be significant in terms of people using applications under Windows)
• There are two completely F/OSS implementations of C#/.NET and the standard library. Mono (Licensing Concerns Addressed Here) and the FSF’s DotGNU PNET. OpenSim is regularly tested for compatibility under Mono (in fact our automated testing environment uses it). DotGNU is significantly less popular and has not been properly tested with due to not being feature complete yet. There is also a third source-availible implementation from Microsoft called Rotor, however it is not under a OSI-approved license.
• Microsoft maintains a reasonably healthy relationship with the Mono developers and has been known to collaborate in the past (such as for the development of the specialised Moonlight runtime).

The next question usually is “Well, why not write it in Java then?” the answer is multi pronged and highly likely to generate a flame war on the subject – the biggest reason is that coding the same thing in Java would probably take significantly longer to do.

Java is a beast of a language that has had layers of gunk added in every revision resulting in a hodge-podge of inconsistently named items in the standard library that may, or may not address what you want. The second major reason is that the C# standard library is both larger and more functional – the amount of time and effort the Base Class Library has saved is astonishing. Wikipedia has a nice article detailing the differences between the two languages.

However, since both Java and C# share a very similar byte code language – it is possible to do a machine driven cross-compilation so you could run OpenSim under the JVM runtime if you so wish. Source translation between the languages is also reasonably possible however requires a degree of manual work.

Some concluding facts which may actually surprise some people

• One of the largest contributors and users of the project is IBM. One of the first groups inside of IBM to get involved was IBM’s Linux Technology Center. All[?] of the IBM developers are using Linux, Emacs and Mono to develop and test.
• Approximately half the developers working on OpenSim are running under Linux/BSD. As a user base, Linux users represent approximately 80-90% of the casual testers and operators. (Windows is however much better represented in people running commercial operations)
• Our compatibility targets are Mono 1.2.5 (latest is 1.9) and .NET 2.0 – we dont use features which supercede these (although we may raise that to Mono 2.0 once it is out of beta).

OpenSim is 64-bit aware

The reason for this happening is that OpenSim by virtue of the .NET platform is quite 64-bit aware, this means natively it can address 16TB of memory just fine if run on a 64-bit operating system. (nb, under Mono you need to make sure Mono is 64bit to get the same benefits).

However unfortunately, some of the libraries we link against have to be compiled ‘one way’ or the ‘other way’ and not both.  When running a 64-bit application, things such as memory pointer lengths are difference, so there is no way to cleanly pass data between a 32-bit and 64bit application – everything must be one or the other.

So how do you fix this?

There’s two routes, the easy route, and the proper route. The easy route is good for 90% of our users – if you dont need to access more than 4GB of memory and dont mind about the slight slowdown when handling double precision numbers, then just run 32-bit.

How? Instead of launching OpenSim.exe launch OpenSim.32BitLaunch.exe instead – this forces OpenSim to run under a 32bit environment (under windows provided by WoW32).

The more detailed fix

The next option is to strip out the sections that are causing problems. OpenSim is modular – so you can swap bits and pieces with their alternatives. The following are components that are not 64-bit compatible by default (however it is possible to fix this, something I will get to in a moment.)

Incompatible Components List

Physics – most problems come from our Physics Engines, since these tend to be written in high performance C/C++ with ASM littered. The following list of physics engines by default are NOT 64-bit compatible.

• OpenDynamicsEngine – The ODE DLL we provide is compiled for a 32bit operating system, however you can fetch the sources for our custom ODE DLL from (SVN) http://opensimulator.org/svn/opensim-libs/ and build it yourself on your native environment.
• AGEIA PhysX – The AGEIA PhysX support (now Nvidia PhysX) relies on a natively compiled 32-bit DLL. At present there is no fix here.

Compatible Engines

• BasicPhysics - This is purely managed but very simple in operation. Collisions are provided only between the avatar and the ground here. This runs without problems regardless of environment
• POSPhysics - POS physics is a more complicated BasicPhysics engine with bounding box collisions between objects. For some tasks this is sufficient.
• BulletX – Our own custom BulletX library is derived from the BulletXNA project, this is a properly developed fully managed physics engine that does proper collisions (however the friction coefficient can be a little low for some users).

Storage Engines – At least one of our storage engines relies on some hard coded components, it’s listed below

• SQLite – Whaaa? SQLite is broken on 64-bit environments? Afraid it’s true. Unfortunately on Windows systems the only solution is to recompile the entire of Mono (including Mono.SQLite.dll) yourself under a 64-bit flag.

Compatible Engines

• MySQL - The DotNetConnector provided by MySQL is 100% compatible and probably the recommended adapter for production installs.
• MSSQL – Windows-only however MSSQL is also free and the adapter is built into .NET itself so is pretty much guarunteed compatible.

Other incompatible components

Perenially we have problems with the following components in addition to the above, this includes libsecondlife’s OpenJPEG version – this can be fixed by compiling libsecondlife and OpenJpeg (using make) on the target system. This may have been fixed recently however as complaints about this DLL have lessened sharply.

Running

Once you have swapped incompatible components out, or recompiled them appropriately, OpenSim should run under a 64-bit environment natively on Windows. As mentioned above, under Linux/Mono you will need to make sure that Mono has been compiled as a 64bit application for this to take effect.

The differences between the two are mostly marginal, however there are some definitve improvements in certain mathematical loops and stability when using large amounts of memory. In production environments, running 64-bit mode can be an attractive option, however will require constant maintainence to compile the above libraries as new releases occur.