With May coming to a close, I would like to start the first of a monthly feature - some explanations of what’s going on under the surface of OpenSim. But wait, you may ask; doesn’t Justin already provide excellent weekly reports on what’s been committed? Of course - where this differs is in the level of detail, particularly on infrastructure projects. Justin does a weekly feature which means he only has time and space to report on features in bullet form. Where I plan to differentiate with these posts is in the details - not only what has been committed, but also where it is going.

There has been a lot of progress this month - starting at revision 9358 there have been several major areas of focus.

Networking: LLUDP Improvements

We now support the Multiple Object Update packet format for Linden UDP clients (most notably Second Life & realXtend), this builds on work that Mikko of realXtend provided late in April that did the same for Avatar Updates. The work done by Melanie reduces the total number of packets that need to be tracked by packing multiple updates into each packet - resulting in a significantly lower workload for our packeting code. The Avatar packet aggregation with 100 Avatars in the region results in a massive 71% reduction in the number of packets transmitted, and a 29% drop in CPU requirements, according to tests done by Intel for ScienceSim. Object aggregation produces similar results (although hasn’t been measured precisely.)

Packet Pooling - the next major improvement is in ‘packet pooling’ - one of the drawbacks to managed languages is very little control over optimising garbage collection. OpenSim handles potentially thousands of packets each second, in our previous packet system for each of these packets, we created several classes representing the internal data structures. (eg, Byte[] Packet becomes MyPacketObject) - while the processing is fairly quick, creating these packets results in the Garbage Collector handling tens of thousands of tiny unused classes at each collection. On .NET this runs so-so, but on Mono it runs atrociously.

Packet Pooling solves this problem by recycling classes - rather than leaving a dead class to be garbage collected, we stuff new data into it and re-use the class instance. We previously did this work back in 2007 with the help of 3Di developers, however it was at the time proven unworkable due to problems with old data persisting into new data blocks. Melanie managed to solve this issue, and nearly two years later we have it re-enabled. Early pooling code did not show much promise past 40 avatars (although up to that point was fairly dramatically better) - however more profiling done by intel managed to locate the problem as being due to a finite sized pool - a patch fixed this problem and the overall improvement was a further 40% reduction in CPU requirements for 100 avatars (total reduction for this and aggregation is a whopping 57% reduction in CPU requirements for a busy sim).

Overall, the OpenSim networking code has been improved very significantly (especially with April’s texture resend fixes), and I would appreciate seeing more testing with large userbases - the Wright Plaza weekly meetings have been running better than they have for a long time, but only represent a peak of 35 concurrent users.

Some early bugs did present themselves with this networking code - most notably running into several MTU limits for international users (apparently most international trunk lines will simply drop rather than fragment big UDP packets), myself, Nebadon and Melanie managed to identify and fix this problem.

I introduced, then disabled support for NAT Loopback - this allows you to emulate a loop-back compatible router for those hosting from home and unable to get a public internet IP. This is currently commented out due to issues with compatibility in Grid Mode (however Standalone mode can be made to work effectively) - further improvements will be in the realm of adding UPnP support allowing you to host a region from home reliably. Finally I also introduced the capability to use IP and DNS based bans (including wildcards), Dr Scofield improved this with some stability checks.

Groups Support

mcortez has been working on adding a fully functional implementation of groups into OpenSim this month - previously located as part of the flotsam project, the groups implementation has hit core. Early revisions ran into a problem with a permissions exploit and group deeding (allowing anonymous users to group deed a parcel), however these have been fixed and it is ready for further testing. My initial feedback is that it looks good - things such as group chat are behaving well and responsively, although it is not yet fully functionally complete yet (particularly around the area of group permissions).

New Region Modules Framework

I’m not entirely sure if this was introduced this month - however a large amount of work was done porting from IRegionModule to the new ISharedRegionModule interface[s]. This new version is syntactically nearly identical, however has better support for removing regions at runtime, and also utilizes the Mono.Addins framework, which will be essential as we begin to add PECL/CPAN-style module repositories to OpenSim (although exactly when this happens is an unknown). The older region module format is going to remain working for quite some time yet - however if you are feeling adventurous you can investigate the new format and look at porting modules across.

New Services Framework

As part of a general improvement of OpenSim code quality, Diva and Melanie improved the services framework this month, deleting the old OGS1 ‘UGAIM’ in favor of a new ‘OpenSim.Services’ binary - this also means that the Grid mode and Standalone mode now share all the major infrastructure internally (Standalone runs the grid services in ‘private’ now) - this reduces code complexity and makes it easier for us to add and adjust the services framework.

This framework has also made it easier to edit and adjust module points for Grid Services, meaning it is now possible to write Grid Service plugins - such as asset caches, external login systems, etc. Diva has been utilizing these to improve the functionality and quality of the Hypergrid technology. A new cache was implemented using GlynnTucker cache - after it was discovered the previous asset cache did not infact, cache anything, early testing indicates this one works.

Finally, Arthur implemented some improvements to the User Service framework which prevents people from seeing the ‘You are already logged in’ message’ in grid mode.

Remote Control, Administration and Web Integration

BlueWall introduced and improved his JSON Stats plugin system for OpenSim which allows remote monitoring via AJAX applications of a region. Melanie introduced a ‘remote console’ system that utilizes REST, and I introduced some basic capacity for Grid Owners to be able to send grid-wide announcements for maintainence purposes. Dr Scofield improved the XMLRPC-based Remote Admin module to handle additional formats. All of these will no doubt go towards improving administrator remote tools and web interfaces.

Inventory Archives

Justin continued work on Inventory Archives - these allow you to save a copy of your entire OpenSim inventory to a file on disk, which you can then restore on any other compatible OpenSimulator instance. For those who work on multiple seperate OpenSim instances (such as a personal grid, public grid, etc) being able to synchronise your inventory between the two is a big help. IAR archives are still highly experimental and should not be relied on yet for any production purposes.

Justin also added the ability to store creator IDs as a URL rather than a UUID - this allows you to reference foreign grid avatars as a creator.

Prim Meshing

Dahlia continued work on the internal ‘Prim Mesher’ used for physics. Improving sculpty physics shells to be more accurate and support properties such as Mirror, improving the overall meshing performance and adding the capability to disk-cache converted meshes for faster startups.

XML Serialisation Formats and Region Archives

Sean and Justin made several improvements to our XML Serialisation support including adding unit tests to the serialisers. This will improve overall reliability and support for OAR files as we move forward (preventing OARs from becoming version-incompatible with each other.), OARs were also improved with support for a wider variety of .tar formats courtesy of Dr Scofield. Finally, OARs were renamed from .tar.gz to .oar as the default filename.

Scripting

I improved MRM scripting, both the Engine and the API. The engine has been improved to support real-time loading of external binaries, including debugger support. This when compiled with the MRM Loader project (located on the forge) allows you to write (and debug!) MRM scripts from within Visual Studio (very cool.), MRMs were also improved with additional security checks, fixing of certain properties (such as Avatar.Position), events (World.OnNewUser) and introduction of an Audio API (World.Audio and IObject.Sound).

LSL was improved with support for the LSL Webserver API going into production on the Second Life(tm) Beta Grid. Also implemented was llAttachToAvatar, llDetachFromAvatar and a host of other improvements.

IRC Clientstack

As mentioned earlier, OpenSim now has a ClientView which supports a primitive text-only IRC protocol support. This allows you to connect to a region for chat facilities using only an IRC client - this may be an advantage over the previous IRC Bridge module where IRC users were not represented inworld with an avatar.

Final Words

If I missed anything - let me know, Justin’s notes are likely to be slightly more comprehensive on smaller non-architectural changes.

It’s currently 12:18 AM PST, and we’re sitting at 42 minutes away from shutting down the grid for the big asset conversion. I will be posting regular updates on the OSGrid twitter feed. Today we’ll be doing two important upgrades to OSGrid infrastructure - the first will be moving to new hardware for the asset server, the second will be converting to the new FragstoreFS backend.

The first step will be shutting down the grid so we can export the database onto the new server - based on previous exports, this is estimated to take approximately 2 hours. The next step will be importing the database and adjusting the schema for the conversion - this involves changing the indexes to add a numeric incremental index which allows us to resume conversion from specified points. The import will take around one to two hours, and the index conversion will take a similar period of time.

The next step will be firing up the fragstore converter - this is a specialised piece of software we wrote for this, which will reformat the data into the new CAS-aware schema, and move the BLOB records into the filesystem. Conversion speed varies greatly based on the size of the individual assets being processed, but we’re hoping this will take between 9 and 15 hours to complete - we preconverted a large portion of the database already; but the exact speedup we will get from the preconversion is unknown.

Total estimated downtime is 21 hours - but this will hopefully be the last time we need to shut down the grid for such an extended period of time, as future asset moves will be able to be done transparently to grid users (Murphy willing.).

Update: OSGrid users - If you are reading this due to the twitter/osgrid notice, or the OSGrid news notice - the short of it is, we’ve converting the asset server. It’s going to take 24 hours, and we’re going to need to pull the grid down to do it. Downtime will begin approximately at 1AM Wednesday morning. It will take up to 24 hours to process the conversion - I’ll be posting information here on this blog as we proceed with updates.

For those just joining us on the OSGrid FragStore saga, read parts #1, and part #2 here.

Day 3: Conversion of the 1.5 million test assets has completed - and now we’ve got only the metadata in the table, I’m going to take some attempts at running queries. CAS storage eliminated slightly higher than my predicted 17% - coming in at around disk 23.49% savings on duplicated assets. Unfortunately the first bug has become apparent - creation date wasn’t preserved. I have patched the converter, so this will be filled when we do the final live conversion.

The following is the outputs of some queries I have run on this table, since it doesnt look like these take too long, we should be able to run these off the production server and update them in future. First, there are 15,957 groups of items with more than 5 objects using the same name in the system, of these, the top asset names are -

Returning to a query I was running yesterday on the CAS%s, the following is the updated chart with 1.5 million test assets loaded.

Fig 1. OSGrid CAS Duplicates (Day 3)

And if you have ever been curious about the distribution of assets in terms of asset type - here is a little chart showing the breakdown.

Fig 2. OSGrid Asset Quantities by Type

Full Conversion Date Set

We’ve decided to go ahead with the switchover this coming Wednesday, at 1AM PST. To go ahead with this, we are going to be bringing the grid offline for up to 24 hours. This hopefully will be the last time we need to go offline for more than 15 minutes - and also represents a move from our old asset hardware “Chenobyl” to our new server “Hindenburg” (which has a much larger hard disk capacity).

The following is my log of the OSGrid asset conversion saga.

Day 1: Today we’re taking a third attempt at doing the big fragstore conversion for OSGrid, for those not following the Saga of the Asset Server - about two months ago we started noticing major scalibility issues surrounding assets. Right now they are thrown into MySQL as a blob table, resulting in large amounts of waste both in duplicate content and in the fact we’re storing a filesystem inside of a relational database - you can read up on the earlier design decisions leading to FragStore here.

The previous two conversion attempts have suffered “mysterious MySQL glitches” which we assume may be related to various bugs with long running commands. Apparently the proper course of action when running a query that takes more than 60 seconds to process the command, is to freeze up entirely and stop processing requests - for now and evermore.

In an attempt to make this run a bit smoother - we’ve broken up the process into 2,000 batches of 1,000 assets each - previously our batch mechanism was using MySQL LIMIT X, Y which has the side effect of getting slower and slower as you progress down the table (thus causing the above); so we’ve shifted to using a numeric ID on the assets table. Putting the numeric ID on there allows us to at least index sequential accesses - LIMIT unfortunately will not use any form of index hinting.

mysql> ALTER TABLE `assets`
    -> ADD COLUMN `numericID`  int(11) UNSIGNED NOT NULL AUTO_INCREMENT AFTER `access_time`,
    -> DROP PRIMARY KEY,
    -> ADD PRIMARY KEY (`numericID`),
    -> ADD UNIQUE INDEX `assetID` (`id`);

Query OK, 1623826 rows affected (1 hour 11 min 2.54 sec)
Records: 1623826  Duplicates: 0  Warnings: 0

An hour and a bit later, the difference between the speed of processing before and after is pretty astounding

mysql> select id from assets limit 540000,10;
10 rows in set (11.53 sec)

mysql> select id from assets where numericID between 540000 AND 540009;
10 rows in set (0.42 sec)

It doesnt need a whole bunch of explanation to figure the above may help with our situation. Running the revised and simplified “AssetConverterMarkII” appears to go without a hitch - data is stored into the database, the metadata table is being filled correctly - all in all it appears to be functional. With one minor teeny little problem.

Only the first 4096 bytes of data are being written to the backend store. The remaining sectors of data are written - but consist entirely of zeros. Retrieving the data results in a buffer of the same length as the original stored asset - but often half the data is completely missing. An hour later, it looks like the data is being sent to the backend voldestore correctly, but either on the way there or on the way back, it loses something. Unfortunately it looks like the problem is outside the purview of the client adapter and is somewhere in the deep murk of the backend storage server.

Day 2: Rethinking time - after spending quite some time hunting for some alternatives, the simplest solution looks to be the best.

While I am keen to use Project Voldemort in the long term - in the short term debugging our implementation details are just not on my agenda. We use a IKVM cross-compiled connector library from Java, and the problem looks like it is sitting in there somewhere. Unfortunately debugging Java IKVM libraries from within .NET is painful at best, and not something easily fitting into our timescale.

The simplest solution is to throw the asset blobs onto the filesystem - filesystems are after all developed to handle tiny little files. Directories will slow down when there is more than about 3,000 entries within them, so we’re breaking storage up into “/b1/b2/hash.blob” - this means assuming an even distribution, approximately 30 files per directory at current size, scaling us up to a capacity of 100 million assets before we need to rethink the situation.

Distribution and redundancy are both things I am still keen to employ - putting us on the filesystem does allow us to look at things such as KosmosFS which provide transparent distributed filesystems on Linux, and also gives us the opportunity to look at commercial filestores down the road if we ever win the lottery.

Rewriting fragstore to use filesystem components where voldemort was employed took all of an hour and the asset converter was up and running - a lot faster too. Our conversion transfer rate on Voldemort was 66 assets per second. FragstoreFS?

10,000 Assets Processed (102.04 asset(s)/sec): 0 error(s) so far.

The second thing I wanted to test was just how big a savings we were getting from using Content Addressable Storage - with 10,000 processed, we ended up with 613 duplicates eliminated (6.1%). With 20,000 - 1390 (6.9%), with 90,000 - 8962 (9.95%). We’re hoping as the full dataset is processed - the % of duplicates eliminated continues to increase.

Fig 1. CAS Duplicate Savings

The next issue to present itself was a slowdown as the conversion occured - the number above (102.4) held firm for the first 10% of the conversion, then conversion speed began to massively taper off, first down to 71.39/sec, then down to 50.22/sec by 150,000 converted. My fears were a reprise of the situation we thought fixed on day 1 - slowdowns on accessing as we move further down the table.

Nebadon suggested that this infact might be actually because as OSGrid has become more popular the average size of an asset has increased over time - so we skipped a million rows down the table and started converting some of the later rows. Conversion speed? 68.31/sec. This indicates that yes, later assets are more expensive to process - but the conversion speed should still average the 60 or so per second we need to be able to convert the entire database in under 24 hours.

Appearing somewhat happy with the results, conversion on the complete database has started, but we wont know how well it has worked until tommorow.

Day 3: Stay tuned!

Last week it came to my attention that the server currently acting as the asset server for OSGrid.org is nearly full on disk space, and at the time, moving to an alternate server was impossible due to certain DNS requirements. The ability to move the asset server onto another machine is no longer a blocking problem, however there are a number of serious problems with the way they are currently being stored.

To enumerate the problems at present, the current architecture needs a little description.

Current Setup

At the moment, Assets are stored and retrieved through an interface called IAssetStore, where it gets stored to is up to the module that implements this interface, although the majority of providers use some kind of relational database as a backend. This works well for small grids and standalone regions where the number and size of assets is small, however OSGrid.org (using the MySQL adapter) has several million asset entries and a 75GB table file. Because of the primary key asset requests are still very fast - however performing operations over the entire table requires a 75GB table scan, resulting in the simplest of queries taking several hours.

It should be noted that there is no duplicate detection, and two copies of the same texture will take up the full disk space and table space, given that there is a very high number of duplicates in the database this accounts for a very significant amount of waste.

OSGrid.org’s Profile

OSGrid’s asset server is currently running on a high performance machine donated by the Electric Sheep Company. It’s disk is a ~140GB SCSI RAID-1 array, which fine for the metadata (maybe even overkill) becomes an issue when raw storage capacity is questioned. The current assets table is around 75GB or so, and growing at a rate of approximately 10% per week. At the current rate of growth, the disk will be full in about 6 weeks time, which means migration to something more scalable in the long term is nessecary.

In the long term however, moving to a new server every 6 months is not an ideal situation, especially so since osgrid is a nonprofit enterprise and hence relies on donations. This creates a nessecity to switch to something in which capacity can be more easily added without bringing the grid down, or relying on moving to bigger and better hardware constantly, especially so if it ever hits the kind of scale where it simply becomes impossible to buy hardware big enough. Ignoring any associated problems with RDMS’s and table size.

The second large concern is the inability to run queries against the table due to it’s size - while this will always be a concern, it can be limited by application of appropriate indexes and limiting the table size where possible. Running things such as garbage collection obviously becomes impossible when the table is too large.

Candidates for a Solution

For the last week I have been investigating some possibilities for scaling the asset cluster, without spending anything other than development time (see ‘relies on donations’ above). Some of the commercial solutions, such as the one Linden Lab employ - Isilon OneFS appear to fit most of the requirements, but the cost is simply too prohibitive, there are also some limitations on them - such as difficulty in garbage collection, duplicate linking, etc. Thus our requirements look something like this:

• Partial Replication - Scaling should be achievable by adding new machines into the cluster, rather than hopping from one to another.
• Inexpensive - the total cost for setting this up should not exceed the donations osgrid recieves to keep running. Ideally it shouldnt any new monthly expenses.
• Handles large datasets - while OSGrid only has 75gb of assets, the Second Life grid has 200tb+. This means whatever solution is chosen, it needs to have good scaling prospects and remain useful. Moving from one provider to another is a time consuming and difficult process - so it should not need to be done too often.
• CAS - The DB should utilize content addressable storage to eliminate problems associated with duplicate files - ideally these should be handled completely transparently.

Optional requirements (’would be nice if..’)

• Fault Tolerance - if an individual system goes down, the system should be able to compensate automatically without downtime.
• Caching - Frequently accessed items should be cached for more speedy access in future.
• Read Replication - assets should be requestable from multiple seperate machines to increase scaling properties.
• Very speedy access on a primary key - KeyValue stores tend to be ideal here (eg, BDB) since data access is well controlled.
• In Production Already - Ultimately there’s a lot more confidence in a system if it’s been proven to scale elsewhere already.

There’s a number of solutions which fit this bill - there’s the hosted variety, Amazon S3 / SimpleDB, Bitcache, Custom Hackery of the Routing Asset Server variety, Custom Hackery of the write-your-own-DB variety, etc. Evaluating these options through the above criteria, most of the options were invalidated.

Amazon, or (the hosted option).

Amazon’s S3 is a well known option for storing large  numbers of images for popular websites - it effectively says “Dont do this, we’ll do it for you.”, and if the price was right, it’d definetely be my selection. Unfortunately the math doesnt quite add up for OSGrid to utilize S3 in the long run, at the current rate of bandwidth and size, OSGrid would be paying around $300.00/month - growing 10% per week and running the resulting 7100 GB is a minimum $1,100/month without any transfers or requests. By comparison, the same amount of space on plain hard disks is a mere $2,500 once-off ($10,700 savings over a year is enough justification to skip this one).

Bitcache

After mentioning the problem on IRC, Bitcache was suggested as an option. For those who havent seen it before - it’s a storage engine that operates on REST principles - the main limitation is it requires an additional database setup to store the URLs of where assets are stored if you wish to start moving assets over multiple machines. Not a huge limitation and it’s definetely easier to integrate into the current solution than writing one yourself. The other downsides is it doesnt appear to be used in production anywhere ‘large’ and the backend is written in Ruby which isn’t well known for scalability.

Custom Hackery of the Routing Asset Server variety

This option involves writing a special version of the asset server which instead of processing the request, does a UUID lookup, then sends the asset over to another server which has that asset stored. This allows assets to be shifted around using standard database migration tools. It’s an option, however fault tolerance is an issue as it’s possible to lose a section of the database and suffer the consequences thereof. This was the option I was considering until only yesterday, albeit using a new custom BDB-based adapter for the final asset store to increase speed and scalabilty.

For those who like pretty pictures -

Custom Hackery of the Write-your-own-DB variety

Bad. No.

Enter Fragstore

Fragstore is similar in principle to how a number of the above systems work. Fragstore utilizes a pair of databases at it’s core - one contains information such as access times, hash information, etc. The other contains the data itself - I use the term ‘database’ here rather loosely, since the data datastore is actually a piece of technology called (ugh) Project Voldemort. PV is a Java project used by LinkedIn for their own internal database - it’s similar in principle to Amazon’s SimpleDB and Google’s BigTable - and has similar properties to them in terms of scaling. It has a number of useful properties which match up to the above - including fault tolerence, partial replication, high speed access, caching and more - plus it’s been put through it’s paces at LinkedIn already.

The downside however is that it running queries on say access time would be ridiculously expensive. Likewise it lacks CAS-support so duplicate files would be difficult to track down and link together. It’s also written in Java and lacks a .NET connector (something thankfully IKVMC was able to fix with the help of dmiles).

Fragstore builds upon the strength of PV by utilizing it as a raw data store, but then integrates in a traditional DBMS for handling of metadata and querying the tables. We also connect in access to the old database so assets uploaded to the old system can still be accessed while the conversion is taking place. The result looks something like this:

There’s a couple of key changes to note - first is, we’ve switched from a UUID-based primary key to a Integer one, this is done because of MySQL’s internal lack of a UUID datatype and the speed at which string based primary keys are searched (several orders of magnitude slower). While not used in normal operations - it will allow us to run the garbage collector on a slaved machine and produce a list of redundant ID’s which can be eliminated significantly faster. Likewise, we can eliminate all copies of an asset at once by eliminating the hash value - useful for instance in handling DMCA takedown requests.

When will this be done? I’m aiming for the next few days to have a beta version availible, if it performs adequetly, we will run a test by converting the entire osgrid asset database over to it. Keep your eyes peeled for more information soon.

Disclaimer: This is probably a bad idea in motion, I make no claims as to how well this could hold up in the long run or real world - I have some personal concerns about how Amazon is able to scale their S3 service effectively, and the recent downtime should give cause to think first before putting this into production.

If you want to be a cool geek these days, you absolutely have to insert the words ‘cloud computing’ into every other sentence. It’s no suprise then that one of the more adopted and interesting toys in the last year or two has been the Amazon Simple Storage Service (S3 for short).

It’s a “cloud” based storage solution, where you can upload files onto the cloud relatively cheaply. It’s being used in places Akamai used to be used for mirroring and proxying of frequently accessed files. (Even Linden Lab uses it for distrobuting their updates.)

The price is right, and the solution is fully programmable via a convenient web API, so it’s snuck itself into all sorts of places. What I’d like to experiment here is using it as a complete grid asset server, for those who havnt toyed much with the OpenSim asset server under the hood - it’s effectively a webserver with two supported operations: “PUT” and “GET” - funnily enough this overlaps nicely with any network accessible storage system, S3 included.

Building a new asset storage module from scratch

First thing we need to do is go hunting for another asset storage module in OpenSim so we can take a look at the interface we are required to implement. After a quick hunt for “asset_database” (the OpenSim.ini setting) it turns up that we need to create a class that derives from AssetServerBase which in turn derives from IAssetServer.

This looks relatively painless so far, we need to make a class which implements all five of these functions. The next step is to go take a look around and see if there is a decent .NET library for interfacing with S3. Turns out there is - .NET S3. (Post-authoring note: Apparently Amazon have an official library which looks easier to use, for one it doesn’t require the delegate hackery seen in the GetAsset function)

This library, while not fantastically documented does seem to do the job quickly and easily.

Implementing the asset client

Now we are at the point where we can actually write our class, using the above library, I’ve skipped ahead and written the class as presented below:

As you can see, the meat of the function sits in the StoreAsset and GetAsset functions, in this case we’ve overridden them to save to an EC2 Bucket (”directory”). We’ve made the directory a parameter of the constructor, as long as each region on the “grid” uses the same directory, they will act as one giant asset database.

Final Steps: Hooking it up

If we head back to OpenSimBase.cs where we first went looking for “m_assetStorage” originally, on line 412 we see:

Below this, let’s add a new statement

Now, if we change the ini setting to read “s3″ our brand spanking new S3 client should fire up.

In Conclusion

I need to make some notes here, first hard coding classes like the above is a bad idea. We shouldnt be doing it for the grid asset server (and instead should be loading a class via reflection). Second we shouldnt be hard-coding the login keys - moving that up to an ini setting is preffered (and not too difficult to do properly). Third this is a untested bit of code, use it at your own risk, this was done as an experiment in approximately 45 minutes.

However, for all the above warts - this shows off the power of OpenSim for rapid prototyping, one of the great features of OpenSim is just how flexible it can be, and how little work is needed to do something completely crazy with it quickly and relatively easily.