Somewhere in the deep dark recesses of my backup server, is a little directory called ‘AWSERVER’, last modified circa 1999. Inside that directory is my world database and the content path associated with my old ActiveWorlds regions. With a little (or maybe a lot) of work, I could probably re-load it today and log in.Yet, if I tried to revisit some of my full-sim creations in Second Life from 2004, I’d have a lot of trouble trying - I’d need to recover what I could from my inventory and do painstaking rebuilding work, if it was possible at all.

This is the problem with SL - the moment something is deleted, or a region is shut down - it’s almost always gone forever. If you wanted to revisit a earlier incarnation of Nexus Prime, or parts of SL’s ancient history - you are out of luck, as there’s a very solid chance those places are simply irrecoverable and lost to the sands of time. As digital data, there should be no good reason for this - disk space is cheap, and sims are small.

The classic case of this is the Bedazzle sims, among them were Gravity Space Station, Chinatown and UnrealSL - but none of them lasted more than between a few weeks or a few months. I’m digressing here slightly; but there are major projects I have been involved in SL that I would love to be able to still access that are gone forever (earlier versions of Aleph, the Atlas Underwater Complex, etc).

The problem isn’t so much that SL doesn’t store ancient rollbacks but that it is simply not possible to save a copy of one; even if you are the rights owner and want to back up your own work. Second Inventory can help here, but it too has flaws - it doesn’t have any kind of mass restore functionality; and it can only save inventory - there’s no chance to save the layout within a region, only the individual contents of it.

It is somewhat sad to see regions shut down by their owners for affordability reasons; knowing full well that the content cannot be ever easily restored later - I personally hate to see it when this happens, because something creative is lost forever.

OpenSim on the other hand, has some real advantages here - I have complete copies of a lot of my builds on OpenSim in varying stages of construction, courtesy of the Region Archive functionality. Every major construction project I have done on any of the grids is sitting somewhere on one of my hard disks as a .tar.gz file containing everything needed to reload it in later. In OpenSim, nothing is ever incapable of being saved - at all times you can dump a copy of the region to a disk, then reload it later somewhere else.

As a creator it’s fairly liberating - and convenient. I can work on a sim locally, export it, then import it into the production environment, and vice versa, take a production environment for local tweaks, edit it, then bring it back again.

Backing up Aleph

I have for a few months been testing an internal tool which allows you to export a OpenSim Archive from a Second Life Region - it was originally developed to export a clients region (their IP); but ended up being handy to preserve some of our workshops and builds from deletion when we closed the sims or rebuilt them. Today, I rewrote it - the previous version was based on the old libomv PrimWorkshop viewer, the new version is now based on the Simian Periscope (Periscope is a kind of multi-user version of GridProxy).

Before anyone asks, the modifications aren’t public - unfortunately for every legitimate user for a tool like this, there’s ten asshats prepared to use it as copybot deluxe, so the source is going to stay private (although I might release a binary version containing creator and permission checks similar to Second Inventory - we’ll see what my schedule looks like in the next few weeks).

This new version is overall a bit more reliable - a number of small bugs and niggles got fixed along the way - but the key factor is it’s now not a 2 hour effort to run, a region can be grabbed with 95%+ accuracy in minutes. You can see here, my personal workshop region ‘Aleph’ in Second Life - it’s a fairly old sim, but it’s gone through a ton of revisions in it’s history. The current revision is a sort of moonbase cross sandbox, complete with orbital lasers.

Aleph Null

Below you can see the same region and contents, but in my personal standalone OpenSim region. This one is located on my personal desktop - but with the same OAR file, I could just as easily reload it on any region running any version of OpenSim since OAR support was added. If I wanted to bring Aleph to OSGrid, it would take only as long as it took to copy and load the file on a region connected to OSGrid.

127.0.0.1:9000

There are limitations, the tool doesn’t copy any form of ‘deep inspection’ - so scripts, etc do not get saved. It’s theoretically possible however to back these up if you are the object owner, something I will be looking at in the future. Estate settings and a few other features aren’t in the v1 OAR format, so those also need to be recreated - but could be something we look at adding in future. It’s also worth noting that it will only backup content with a creator tag in a specified list; while that limit could be removed, it provides an easy way of assuring that you are backing up only content you have rights to.

The Wayback Grid

One of the projects I would like to see would be some kind of opt-in mechanism to subscribe to an automatic backup service similar to Archive.org for the web - the goal of which would be preservation of content in the long run. When someone drops their region, it would be nice to be able to restore it later on if they buy another region, or move to one of the open grids.

Ultimately, this could be taken to an extreme where you can dial back a grid in time, and see it as it was in a previous point in time - however the biggest limitations here are dealing with content prosciption. Most SL content is licensed with one or more restrictions - ethically ‘no copy’ is probably a blocker to performing backups, likewise ‘no transfer’ makes some implications about bringing something out of SL. To do this effectively, you need massive ‘opt-in’ by content creators to approve their content going outside.

The easiest starting point is instead probably to provide some kind of service for creators to voluntarily backup their sims (say, in the case of a creator shutting down their region - but wanting to preserve it). Perhaps there are options here to look at providing some kind of transfer service for people moving regions from say SL to OSGrid [providing they own the copyright]. If anyone is interested in that kind of service, let me know - this might be useful for folks contemplating migrating over to OpenSim/OSGrid and have all their own content.

With May coming to a close, I would like to start the first of a monthly feature - some explanations of what’s going on under the surface of OpenSim. But wait, you may ask; doesn’t Justin already provide excellent weekly reports on what’s been committed? Of course - where this differs is in the level of detail, particularly on infrastructure projects. Justin does a weekly feature which means he only has time and space to report on features in bullet form. Where I plan to differentiate with these posts is in the details - not only what has been committed, but also where it is going.

There has been a lot of progress this month - starting at revision 9358 there have been several major areas of focus.

Networking: LLUDP Improvements

We now support the Multiple Object Update packet format for Linden UDP clients (most notably Second Life & realXtend), this builds on work that Mikko of realXtend provided late in April that did the same for Avatar Updates. The work done by Melanie reduces the total number of packets that need to be tracked by packing multiple updates into each packet - resulting in a significantly lower workload for our packeting code. The Avatar packet aggregation with 100 Avatars in the region results in a massive 71% reduction in the number of packets transmitted, and a 29% drop in CPU requirements, according to tests done by Intel for ScienceSim. Object aggregation produces similar results (although hasn’t been measured precisely.)

Packet Pooling - the next major improvement is in ‘packet pooling’ - one of the drawbacks to managed languages is very little control over optimising garbage collection. OpenSim handles potentially thousands of packets each second, in our previous packet system for each of these packets, we created several classes representing the internal data structures. (eg, Byte[] Packet becomes MyPacketObject) - while the processing is fairly quick, creating these packets results in the Garbage Collector handling tens of thousands of tiny unused classes at each collection. On .NET this runs so-so, but on Mono it runs atrociously.

Packet Pooling solves this problem by recycling classes - rather than leaving a dead class to be garbage collected, we stuff new data into it and re-use the class instance. We previously did this work back in 2007 with the help of 3Di developers, however it was at the time proven unworkable due to problems with old data persisting into new data blocks. Melanie managed to solve this issue, and nearly two years later we have it re-enabled. Early pooling code did not show much promise past 40 avatars (although up to that point was fairly dramatically better) - however more profiling done by intel managed to locate the problem as being due to a finite sized pool - a patch fixed this problem and the overall improvement was a further 40% reduction in CPU requirements for 100 avatars (total reduction for this and aggregation is a whopping 57% reduction in CPU requirements for a busy sim).

Overall, the OpenSim networking code has been improved very significantly (especially with April’s texture resend fixes), and I would appreciate seeing more testing with large userbases - the Wright Plaza weekly meetings have been running better than they have for a long time, but only represent a peak of 35 concurrent users.

Some early bugs did present themselves with this networking code - most notably running into several MTU limits for international users (apparently most international trunk lines will simply drop rather than fragment big UDP packets), myself, Nebadon and Melanie managed to identify and fix this problem.

I introduced, then disabled support for NAT Loopback - this allows you to emulate a loop-back compatible router for those hosting from home and unable to get a public internet IP. This is currently commented out due to issues with compatibility in Grid Mode (however Standalone mode can be made to work effectively) - further improvements will be in the realm of adding UPnP support allowing you to host a region from home reliably. Finally I also introduced the capability to use IP and DNS based bans (including wildcards), Dr Scofield improved this with some stability checks.

Groups Support

mcortez has been working on adding a fully functional implementation of groups into OpenSim this month - previously located as part of the flotsam project, the groups implementation has hit core. Early revisions ran into a problem with a permissions exploit and group deeding (allowing anonymous users to group deed a parcel), however these have been fixed and it is ready for further testing. My initial feedback is that it looks good - things such as group chat are behaving well and responsively, although it is not yet fully functionally complete yet (particularly around the area of group permissions).

New Region Modules Framework

I’m not entirely sure if this was introduced this month - however a large amount of work was done porting from IRegionModule to the new ISharedRegionModule interface[s]. This new version is syntactically nearly identical, however has better support for removing regions at runtime, and also utilizes the Mono.Addins framework, which will be essential as we begin to add PECL/CPAN-style module repositories to OpenSim (although exactly when this happens is an unknown). The older region module format is going to remain working for quite some time yet - however if you are feeling adventurous you can investigate the new format and look at porting modules across.

New Services Framework

As part of a general improvement of OpenSim code quality, Diva and Melanie improved the services framework this month, deleting the old OGS1 ‘UGAIM’ in favor of a new ‘OpenSim.Services’ binary - this also means that the Grid mode and Standalone mode now share all the major infrastructure internally (Standalone runs the grid services in ‘private’ now) - this reduces code complexity and makes it easier for us to add and adjust the services framework.

This framework has also made it easier to edit and adjust module points for Grid Services, meaning it is now possible to write Grid Service plugins - such as asset caches, external login systems, etc. Diva has been utilizing these to improve the functionality and quality of the Hypergrid technology. A new cache was implemented using GlynnTucker cache - after it was discovered the previous asset cache did not infact, cache anything, early testing indicates this one works.

Finally, Arthur implemented some improvements to the User Service framework which prevents people from seeing the ‘You are already logged in’ message’ in grid mode.

Remote Control, Administration and Web Integration

BlueWall introduced and improved his JSON Stats plugin system for OpenSim which allows remote monitoring via AJAX applications of a region. Melanie introduced a ‘remote console’ system that utilizes REST, and I introduced some basic capacity for Grid Owners to be able to send grid-wide announcements for maintainence purposes. Dr Scofield improved the XMLRPC-based Remote Admin module to handle additional formats. All of these will no doubt go towards improving administrator remote tools and web interfaces.

Inventory Archives

Justin continued work on Inventory Archives - these allow you to save a copy of your entire OpenSim inventory to a file on disk, which you can then restore on any other compatible OpenSimulator instance. For those who work on multiple seperate OpenSim instances (such as a personal grid, public grid, etc) being able to synchronise your inventory between the two is a big help. IAR archives are still highly experimental and should not be relied on yet for any production purposes.

Justin also added the ability to store creator IDs as a URL rather than a UUID - this allows you to reference foreign grid avatars as a creator.

Prim Meshing

Dahlia continued work on the internal ‘Prim Mesher’ used for physics. Improving sculpty physics shells to be more accurate and support properties such as Mirror, improving the overall meshing performance and adding the capability to disk-cache converted meshes for faster startups.

XML Serialisation Formats and Region Archives

Sean and Justin made several improvements to our XML Serialisation support including adding unit tests to the serialisers. This will improve overall reliability and support for OAR files as we move forward (preventing OARs from becoming version-incompatible with each other.), OARs were also improved with support for a wider variety of .tar formats courtesy of Dr Scofield. Finally, OARs were renamed from .tar.gz to .oar as the default filename.

Scripting

I improved MRM scripting, both the Engine and the API. The engine has been improved to support real-time loading of external binaries, including debugger support. This when compiled with the MRM Loader project (located on the forge) allows you to write (and debug!) MRM scripts from within Visual Studio (very cool.), MRMs were also improved with additional security checks, fixing of certain properties (such as Avatar.Position), events (World.OnNewUser) and introduction of an Audio API (World.Audio and IObject.Sound).

LSL was improved with support for the LSL Webserver API going into production on the Second Life(tm) Beta Grid. Also implemented was llAttachToAvatar, llDetachFromAvatar and a host of other improvements.

IRC Clientstack

As mentioned earlier, OpenSim now has a ClientView which supports a primitive text-only IRC protocol support. This allows you to connect to a region for chat facilities using only an IRC client - this may be an advantage over the previous IRC Bridge module where IRC users were not represented inworld with an avatar.

Final Words

If I missed anything - let me know, Justin’s notes are likely to be slightly more comprehensive on smaller non-architectural changes.

OpenSim is now a semi-RFC1459 compliant IRCd. IRC clients are treated as full members of the virtual world, gaining a presence with a horrible default avatar, as the screenshots below picture. Each IRC client has an implementation of IClient* which means we could in theory insert text descriptions of the world and any events in. (Although in practice it’s unlikely to be useful for anything beyond basic text & IM messaging with an IRC client).

The advantage of this over the IRC Bridge module is that each presence is visible in both environments (users show up in the ‘who’ list on IRC, and have an avatar ‘in world’), and we will be able to handle a number of events more directly and appropriately - such as private messaging or group discussions. The IRCd framework I have written is extremely basic and probably not 100% to spec - if anyone with IRCd experience wants to get into the code and fix it up, please do.

In the mean time, enjoy. You can enable this by using the [IRCd] Enabled=true option in OpenSim.ini and connecting on port 6666.

Fig 1. XChat

Figure 2. Hippo Viewer

This weekend, I have been experimenting with replacing OpenSimWI with open source social networking software - Elgg. For those who haven’t used or seen Elgg before, it is very similar to things like Facebook - user profiles, groups, etc.

The key with what we are aiming to do however, is eliminate the duplicated data. That is, if you have a group “inworld”, the exact same data should be used on the website. Join a group on the website - it should show up in world. The way we have done this has been to rewrite the Elgg database layer so that it reads and writes from the same databases the OpenSim Grid Servers do.

The result is you can participate in the grid, without needing to be logged in. Grab an RSS feed of your group notices or send someone an offline IM from the OSGrid website.

Custom Profile Pages (Editing)

A Sample Profile

Friends List (Detail)

Group Listings (Popularity Tab)

Group View

When?

There is no deadline for when this will be launched, but we’re hoping to do so within the month.

If you have taken a look at the OpenSimulator mailing lists or IRC channels in the last 24 hours, you will have found a link towards a new ‘feature tracker‘. It works kind of like Mantis for bug tracking, but it is optimized for quick and simple feature request posting. I should however make some notes on how this works.

First, this is not a roadmap. Putting a feature up and getting a high number of votes won’t get it implemented - however a developer looking at the idea and saying ‘Hrrm, I could do this’ will. High votes and high views will make it more likely that a developer looks at your idea (and may influence priorities), but it is not binding.

Second, this replaces mantis for feature requests - our mantis install has traditionally been the place feature requests go, but a lot of developers havent wanted them there since it is officially a bug tracker only. This tracker seperates the two out and recognizes features needing different tools.

It’s a work in progress and fairly experimental (I wrote the whole thing from scratch in an afternoon), but go take a look and post your OpenSim suggestions.

This is modelled somewhat after the same guide for wordpress. Running an OpenSimulator instance is a challenging prospect for many - there are a number of important criteria you should be looking at before purchasing, but the single most important question is “Will I be managing this myself?”.

“Do it yourself”

DIY Hosting OpenSim while relatively easy by comparison to most hosting software, is still not something that should be done by the technically uninitiated. OpenSim requires a healthy amount of maintainence in the form of updates, backups and patching.

Virtual or Dedicated?

For DIY hosting you have two choices in terms of hosting - Virtual Private Servers (VPS) or Dedicated Servers. Virtual Servers are a percentage of a physical server provided inside of a virtual machine. Virtual servers are often significantly cheaper than a dedicated server, however come at a heavy cost in performance.

Virtual server pricing starts at approximately US$15.00 per month. Dedicated servers will start at approximately US$50.00 per month. Virtual servers are appropriate for low performance private sandboxes, Dedicated servers are a must for public or regular usage.

A final note on dedicated servers - the most inexpensive dedicated server will often outperform the most expensive VPS. If you are on the price border between the two, I recommend switching to the lower performance, but dedicated server - than the higher performance but shared VPS.

Linux or Windows?

Unless you have significant investments in using Linux for your hosting, I recommend using Windows 2003 or 2008, especially if you are not familiar with running hosting environments. The reasons for this are fairly simple - Mono has memory management problems, memory requirements under Mono are often doubled or quadrupled.

Unfortunately Windows server is not always availible - especially at the lower price ranges. Microsoft charges hosting companies a $10/month per cpu fee for Windows Server, so this will always be passed on in the form of higher prices. In the highly competitive VPS industry this price increase makes good Windows VPSs a rare niche.

What processor?

OpenSim will very rarely peg your processor - I would not bother investing too much into the processing capacity of your instance or server. (Whether a VPS or Dedicated machine).

My own experience says that “off the shelf” desktop processors are more than adequate. Here at DeepThink we have standardised on Xeon 3220 based machines (Quad Core, 2.4Ghz) and we find these are more than suitable for our higher-end clients. For an average user, a off-the-shelf Core2Duo is more than sufficient.

How much memory?

This is a much more relevant question for OpenSim. If you are running on Mono, I recommend no less than 1GB of memory per region (512MB if it is mostly empty). If you are running Windows/.NET you can use a rule of thumb at approximately 384 to 512mb per region.

A more precise metric is to total up the following: A base cost of 200mb, total size of all textures in the region + 30% in megabytes (average region is around 50mb), 30mb per 1,000 primitives, 50mb per concurrent user and double the sum if you are running under Mono.

So a region with 5,000 primitives, an average number of textures (50mb) and ten concurrent users you should allocate approximately 900MB of memory for .NET and 1.8GB for Mono. Leave at least 100mb on top of this sum for the operating system itself; if the machine runs out of memory page file access will cripple the performance of your region.

Memory should be as fast as possible - the faster you can get the better.

How much hard disk?

None. OpenSim rarely uses more than a few gigabytes of space unless you are operating a large grid (OSGrid.org is only about 120GB of assets currently). Anything over 10GB is sufficient.

How much bandwidth?

This is a slightly tougher question to answer - it depends a lot on your sim, the complexity and texture detail, the number of visitors. We at DeepThink sponsor the servers running the OSGrid.org Plaza’s, which I believe are the six most visited OpenSim instances running. The Plazas are broken up over two physical machines - each uses about 240GB of bandwidth each month - or approximately 80GB per region per month.

In pratice your region should come in significantly under this - but I recommend making sure you have at least 100GB of traffic per region in the event that you create a popular space. Most hosts provide at least 1TB of transfer on dedicated servers, so this should in most cases be a non-concern.

The other side of bandwidth is connectivity - how fast your server can speak with the outside world. I recommend having at least 200kbit per concurrent user. So for a standard ‘10 concurrent’ situation, you should have at least 2mbit of connectivity.

Backup Space

Make sure to have some kind of external backup system - most hosting providers will sell you SFTP access to a backup server, however make sure that they in turn provide good backup policies on your backups (offsite backup, etc.). OpenSim can be very fickle software, so make sure you have a good backup routine setup.

Latency

If you are a non-american user or cater to a foreign crowd, Latency is a very important issue. Latency is the “lag” between when you do an action and when it occurs on the simulation server - unfortunately latency is often best fixed by changing the geographical location of your servers.

If you cater to an audience in a specific country - I would recommend searching for servers in countries nearby. Often localised hosting is several times more expensive than in the US due to higher prices for local bandwidth, however most of the time there is a cheaper alternative nearby. For example, Japan has very high local server costs - but Korean or Singaporean servers are cheaper and have similar latency characteristics.

Who?

Dedicated Servers

CariNet (San Diego, CA, USA)

CariNet have been a popular choice for people running regions on OSGrid thanks to great pricing and astonishgly quick setup times. Their live support is pretty good and tends to fix most issues within 30 minutes of reporting. Cari have a highly flexible pricing system - so make sure to remember to include additional RAM and the Windows operating system with your purchases.

CariNet are also an official sponsor of OSGrid.org, providing regular free upgrades to the official servers that are hosted there (Currently the two plaza machines and the one welcome machine). They have also provided some special pricing for OpenSim users - see the italics below.

Pricing: Minimum $50.00 per month (special offer at time of writing).

I spoke with CariNet’s sales department, and we have worked out a few “OpenSim Optimized” configurations -

• High Performance - Windows, 4 Core Xeon, 4GB configuration is US$200.00/mo with no setup fee.
  This configuration is ideal for a “SL-replacement” style region host, you could probably get a couple of regions off this hardware, but the exact number will vary as above. I would estimate approximately 2 “Full Regions” could be run on this hardware configuration.
  If you want this server for only $160/month (but with a $120 setup), contact Cari.net sales via their Live Chat with the discount code “OSGRID”. (Speak with Mike or Shawn)
• Mid-Range - Windows, Core2Duo, 2GB configuration is US$140/mo with no setup fee.
  Expect to get one heavy region out of this configuration (or quite a few “homestead-esque” regions) - as always, this varies based on consumption.
  As above, if you want this server for only $120/month (but with a $60 setup), contact Cari.net sales via their Live Chat with the discount code “OSGRID”. (Speak with Mike or Shawn)
• Low-End - Windows, Celeron 2 Core, 2GB configuration is US$125/mo with no setup fee
  Consider this a homestead replacement, you should be able to get a small number of “homestead-equivilents” running off of this - obviously you dont have primitive limits or things like that, but you will notice degrading performance if you rely on a lot of scripted functionality.
  As above, if you want this server for only $105/month (but with a $60 setup), contact Cari.net sales via their Live Chat with the discount code “OSGRID”. (Speak with Mike or Shawn)

Annual and quarterly discounts are availible for paying more than one month at a time.

Website: http://www.cari.net/

OVH (Paris, France, EU)

OVH are the largest dedicated server host in Europe and provide unbeatable pricing on European servers. English pricing is expensive, however the same servers ordered from their French or German language sites are 10-20% cheaper.

OVH provide large amounts of memory and disk space by default, so very little customisation is required on their default servers. Windows 2003/2008 is an additional EUR 15.00 per month. Their RPS low-end machines may be a suitable mid-way point between a VPS and a full machine, but I haven’t tested one myself.

Pricing: Minimum EUR 49.95 per month. An OpenSim optimized configuration will start at EUR 114.00 per month (fully equipped).

Website: http://www.ovh.co.uk

Virtual Servers

Tektonic (Dallas, TX, USA)

Highly recommended by one of our employees, Tektonic provide good quality Linux VPS solutions - while they only offer Linux based VPS instances, their pricing is affordable and do not appear to oversell their servers in any noticable capacity.

Pricing: Minimum $15.00 per month. A “workshop” quality VPS is US$28.00 per month.

Website: http://www.tektonic.net

Slicehost (St Louis, MO, USA)

Slicehost guaruntee a minimum level of performance on their virtual servers which makes them more expensive than other hosts, however have a solid customer service reputation and a high quality control panel.

Pricing: Minimum $20.00 per month. A “workshop” quality VPS is US$38.00 per month.

Website: http://www.slicehost.com

FsckVPS (Atlanta, LA or Texas)

FsckVPS is often recommend on the OSGrid forums as a potential host - their slices are of a reasonable configuration and have plenty of memory availible. Processing time is more limited and some users have complained about running into processor limits, so their higher end packages are recommended.

Pricing: Minimum $9.95 per month. A “workshop” quality VPS is $19.90 per month, however I recommend the $34.90 plan if you plan on concurrent users.

UPDATE 9 June 2009: These guys dropped the ball badly on an exploit, while it does look like their upstream provider gets the lions share of the blame, they had an unacceptable amount of data loss that could have been avoided by doing some simple backup precautions when an exploit becomes known.

Website: http://www.fsckvps.com

“Someone host it for me”

As an alternative to setting up and managing your own OpenSim instance, you can pay for someone else to manage it for you. Often this is simply bundled with a grid - however a number of companies (including DeepThink) offer managed hosting services for OpenSim.

On average, an OpenSim instance will take around four hours of maintanence each month - at an average industry salary this is around $80.00 per month to the hosting company in engineering costs. You should expect the margin per-server to be about this size.

OpenSim hosting companies will charge based on one of two methods - the first method is Second Life(r) style limits - Primitive, Avatar and Script limitations. The companies that charge this way are generally those providing regions within a grid.

The second method is charging for the underlying service used - Processor time, RAM usage and Bandwidth. The companies offering this tend to be fairly agnostic as to where your regions will connect.

An important distinction between the two methods is that you will often get more ‘bang for buck’ on the underlying capacity. For instance if you need multiple low performance regions - you will come out significantly ahead by purchasing server time rather than regions. Likewise if you need a single high performance region you can tailor your requirements more specifically.

A final note - many simulators are sold as ’supporting 45,000 prims’. This is not reflective of actual usage - the 45,000 number is the largest number the Second Life(R) client will display, however on the backend there is often no limit. That however does not mean the region is capable of supporting 45,000 prims.

We often find that simulator performance degrades rapidly past the 10,000 primitive limit per region and it requires a significant amount of work to get more than 15,000 prims into a region while the region itself is stable, especially when scripts and multiple users are factored into the equasion.

As an addendum to the above - the per region count is not reflective of the per-simulator count. OpenSim can host multiple regions per simulator instance and we have found that the simulator instance itself can host in excess of 100,000 primitives as long as they are distributed between multiple regions.

A note on SLAs and Uptime

OpenSim is still alpha software. If you expect 99%+ uptime, you will also expect to pay for it. Keeping a region running with guarunteed uptime requires active maintainence and attention by an engineer. This means it is impactical to expect 99%+ uptime unless you can afford a technician 24/7.

Our experience with client projects is between 97.6% and 99.2% uptime over the course of a month. (including scheduled and unscheduled maintainence) - this will vary slightly depending on the version of OpenSim used, number of users and other factors.

If someone claims to offer bug-free and stable deployments of OpenSim, they are probably lying or are very naieve. OpenSim will occasionally run into bugs. If you have a specific project in mind that requires a production deployment it is strongly recommended to hire a consultant who is familiar with the OpenSim project (eg DeepThink, IBM, 3Di, Tribal Media, etc.).

List of companies providing OpenSim hosting (note this is incomplete - there are likely to be others)

Customized OpenSim Hosting Providers

The following companies provide customisation services and will tailor your OpenSim install to your specific needs. These companies all provide pricing on a per-server basis rather than per-region.

DeepThink (Managed OpenSim hosting and Customisation services): http://www.deepthink.com.au

ReactionGrid (Managed OpenSim hosting): http://www.reactiongrid.com

3Di (Customisation services, hosting. Tokyo based.): http://www.3di-opensim.com/en/

Not listed here, but likely to offer these services are IBM, Tribal Media and Genkii.

Grid Hosting Providers

These companies provide hosting within the context of a grid. Most of these companies use very similarly specced hardware, however many of them share multiple clients per server - to use Second Life as an example, Linden Lab charges $295 per “island” which is stacked at 4 per server (or $1,180 per server per month).

While I did attempt to compile an exhaustive list of companies here - finding pricing and stacking ratios for many of them proved a difficult and time consuming process. You can find a more comprehensive list of grids (most of which offer region hosting), here:

http://opensimulator.org/wiki/Grid_List

Stability Remarks

As noted all over the OpenSimulator website - OpenSim is alpha quality software. To quote our downloads page

Please note: As OpenSim is still at an alpha code maturity stage, there is absolutely no guarantee that functionality works or is stable, even in the numbered releases. Certain features may not work either because the code is in rapid evolution, or because functionality expected by the Linden Labs Second Life viewer has simply not been implemented yet. However, constructive feedback is still welcomed.

If you are putting OpenSim into a production environment, make sure to speak with your hosting provider about what you plan to do - many of them may be able to make recommendations or tweaks to better suit your demands. If you have clients involved - it is often better to stick to an older more stable release as all releases are not nessecarily equal (eg 0.5.8 was somewhat stable, whereas 0.6.3 had appearance bugs).

If you are planning on using the ‘bleeding edge’ version of OpenSim in production, expect to get cut, or again quoting the downloads page:

There Be Dragons Beyond This Point

If you are truly feeling dangerous, adventurous, or want to help us test the next version of OpenSim you are welcome to grab the latest unstable code out of our subversion trunk. Any warnings previous expressed about the alpha nature of the code go double or triple if you are running directly off of trunk. Never, ever, ever, never run this in production environments, it is not suitable for that unless you are very familiar with the source code, and can hot fix any piece of it (that probably means you are an OpenSim core member). Feedback and testing on the unstable tree is appreciated, as that helps us make the next release better. If this scares you from using trunk, that was intended.

If it breaks, you get to keep both pieces.

Final Remarks

Running an Opensimulator instance can be a perfect fit for your organisation or person as long as you respect the limits. OpenSim can provide fantastic discounts to other commercial virtual worlds software, however it is worth remembering that it is still new and experimental - often this can be an advantage in customisability and features, but at the same time you need to respect the limitations of the software.

If you are unsure about whether to put OpenSim into production or how to do it in a way that fits what you need to do - like every Open Source piece of software, there are a lot of developers and organisations who can provide consulting services. Check the Core Developers List on the OpenSimulator wiki for a full list.

Update: OSGrid users - If you are reading this due to the twitter/osgrid notice, or the OSGrid news notice - the short of it is, we’ve converting the asset server. It’s going to take 24 hours, and we’re going to need to pull the grid down to do it. Downtime will begin approximately at 1AM Wednesday morning. It will take up to 24 hours to process the conversion - I’ll be posting information here on this blog as we proceed with updates.

For those just joining us on the OSGrid FragStore saga, read parts #1, and part #2 here.

Day 3: Conversion of the 1.5 million test assets has completed - and now we’ve got only the metadata in the table, I’m going to take some attempts at running queries. CAS storage eliminated slightly higher than my predicted 17% - coming in at around disk 23.49% savings on duplicated assets. Unfortunately the first bug has become apparent - creation date wasn’t preserved. I have patched the converter, so this will be filled when we do the final live conversion.

The following is the outputs of some queries I have run on this table, since it doesnt look like these take too long, we should be able to run these off the production server and update them in future. First, there are 15,957 groups of items with more than 5 objects using the same name in the system, of these, the top asset names are -

Returning to a query I was running yesterday on the CAS%s, the following is the updated chart with 1.5 million test assets loaded.

Fig 1. OSGrid CAS Duplicates (Day 3)

And if you have ever been curious about the distribution of assets in terms of asset type - here is a little chart showing the breakdown.

Fig 2. OSGrid Asset Quantities by Type

Full Conversion Date Set

We’ve decided to go ahead with the switchover this coming Wednesday, at 1AM PST. To go ahead with this, we are going to be bringing the grid offline for up to 24 hours. This hopefully will be the last time we need to go offline for more than 15 minutes - and also represents a move from our old asset hardware “Chenobyl” to our new server “Hindenburg” (which has a much larger hard disk capacity).

The following is my log of the OSGrid asset conversion saga.

Day 1: Today we’re taking a third attempt at doing the big fragstore conversion for OSGrid, for those not following the Saga of the Asset Server - about two months ago we started noticing major scalibility issues surrounding assets. Right now they are thrown into MySQL as a blob table, resulting in large amounts of waste both in duplicate content and in the fact we’re storing a filesystem inside of a relational database - you can read up on the earlier design decisions leading to FragStore here.

The previous two conversion attempts have suffered “mysterious MySQL glitches” which we assume may be related to various bugs with long running commands. Apparently the proper course of action when running a query that takes more than 60 seconds to process the command, is to freeze up entirely and stop processing requests - for now and evermore.

In an attempt to make this run a bit smoother - we’ve broken up the process into 2,000 batches of 1,000 assets each - previously our batch mechanism was using MySQL LIMIT X, Y which has the side effect of getting slower and slower as you progress down the table (thus causing the above); so we’ve shifted to using a numeric ID on the assets table. Putting the numeric ID on there allows us to at least index sequential accesses - LIMIT unfortunately will not use any form of index hinting.

mysql> ALTER TABLE `assets`
    -> ADD COLUMN `numericID`  int(11) UNSIGNED NOT NULL AUTO_INCREMENT AFTER `access_time`,
    -> DROP PRIMARY KEY,
    -> ADD PRIMARY KEY (`numericID`),
    -> ADD UNIQUE INDEX `assetID` (`id`);

Query OK, 1623826 rows affected (1 hour 11 min 2.54 sec)
Records: 1623826  Duplicates: 0  Warnings: 0

An hour and a bit later, the difference between the speed of processing before and after is pretty astounding

mysql> select id from assets limit 540000,10;
10 rows in set (11.53 sec)

mysql> select id from assets where numericID between 540000 AND 540009;
10 rows in set (0.42 sec)

It doesnt need a whole bunch of explanation to figure the above may help with our situation. Running the revised and simplified “AssetConverterMarkII” appears to go without a hitch - data is stored into the database, the metadata table is being filled correctly - all in all it appears to be functional. With one minor teeny little problem.

Only the first 4096 bytes of data are being written to the backend store. The remaining sectors of data are written - but consist entirely of zeros. Retrieving the data results in a buffer of the same length as the original stored asset - but often half the data is completely missing. An hour later, it looks like the data is being sent to the backend voldestore correctly, but either on the way there or on the way back, it loses something. Unfortunately it looks like the problem is outside the purview of the client adapter and is somewhere in the deep murk of the backend storage server.

Day 2: Rethinking time - after spending quite some time hunting for some alternatives, the simplest solution looks to be the best.

While I am keen to use Project Voldemort in the long term - in the short term debugging our implementation details are just not on my agenda. We use a IKVM cross-compiled connector library from Java, and the problem looks like it is sitting in there somewhere. Unfortunately debugging Java IKVM libraries from within .NET is painful at best, and not something easily fitting into our timescale.

The simplest solution is to throw the asset blobs onto the filesystem - filesystems are after all developed to handle tiny little files. Directories will slow down when there is more than about 3,000 entries within them, so we’re breaking storage up into “/b1/b2/hash.blob” - this means assuming an even distribution, approximately 30 files per directory at current size, scaling us up to a capacity of 100 million assets before we need to rethink the situation.

Distribution and redundancy are both things I am still keen to employ - putting us on the filesystem does allow us to look at things such as KosmosFS which provide transparent distributed filesystems on Linux, and also gives us the opportunity to look at commercial filestores down the road if we ever win the lottery.

Rewriting fragstore to use filesystem components where voldemort was employed took all of an hour and the asset converter was up and running - a lot faster too. Our conversion transfer rate on Voldemort was 66 assets per second. FragstoreFS?

10,000 Assets Processed (102.04 asset(s)/sec): 0 error(s) so far.

The second thing I wanted to test was just how big a savings we were getting from using Content Addressable Storage - with 10,000 processed, we ended up with 613 duplicates eliminated (6.1%). With 20,000 - 1390 (6.9%), with 90,000 - 8962 (9.95%). We’re hoping as the full dataset is processed - the % of duplicates eliminated continues to increase.

Fig 1. CAS Duplicate Savings

The next issue to present itself was a slowdown as the conversion occured - the number above (102.4) held firm for the first 10% of the conversion, then conversion speed began to massively taper off, first down to 71.39/sec, then down to 50.22/sec by 150,000 converted. My fears were a reprise of the situation we thought fixed on day 1 - slowdowns on accessing as we move further down the table.

Nebadon suggested that this infact might be actually because as OSGrid has become more popular the average size of an asset has increased over time - so we skipped a million rows down the table and started converting some of the later rows. Conversion speed? 68.31/sec. This indicates that yes, later assets are more expensive to process - but the conversion speed should still average the 60 or so per second we need to be able to convert the entire database in under 24 hours.

Appearing somewhat happy with the results, conversion on the complete database has started, but we wont know how well it has worked until tommorow.

Day 3: Stay tuned!

Ziah Zhangsun has written an interesting article on using MRMs in OpenSim, go take a look.

I thought it would be interesting to make some pretty charts representing the data behind OSGrid.org’s operations. The following is accurate as of today, but probably wont be tommorow. Some of the data is missing or has gaps in it - this is because during those periods I was not able to get accurate data and decided not to display it at all.

Servers

There are a total of 395 unique IP addresses connected to OSGrid, each roughly corresponds to a unique physical server hosting regions. Of these, 130 were compiled from SVN and give version information. (265 not reporting version).

The most popular version is r9332 with 35 unique servers running this revision. Next is a tie between the official 0.6.4 release and r9313 with 20 unique servers each. The remainder is distributed fairly evenly between r9307 and r9336. It is worth noting that version information was introduced in r9307 - so we cant quite yet see into the 60% that are running behind that version. What is interesting to note is however than 30% of the regions on OSGrid were updated with this revision already and indicates active upgrades and maintainence.

Fig 1. OpenSimulator Versions on OSGrid.org

Regions

At time of writing, there are 2,083 regions connected to OSGrid, owned by 386 unique avatars. (Averaging 5.3 regions per avatar) on 395 unique servers. Each server hosts on average 5.2 regions, the mode is 1, with 35% of servers hosting only a single region. The following chart displays the average number of regions per server over a range of values. Interestingly, this means that 1 in 7.5 users own their own region on OSGrid (compared to 1 in 600 in Second Life®)

Fig 2. Regions per Server on OSGrid

Users

The most interesting statistics are to do with users - currently there are 15,669 users registered on OSGrid.org. 20% of these users have logged in in the last week, 40% in the last month (75% in the last quarter). You can see the proportion of users who have logged in since a certain date on the following chart.

Fig 3. User logins since specified date

User Registrations are also fairly interesting to look at - OSGrid benefited enormously by Linden Lab’s Open Space pricing change announcement back in last November. Since the announcement, registrations on osgrid per day has doubled.

Fig 4. User Registrations per Day

Fig 5. Total User Registrations

It will be interesting to see if history repeats itself again when the next set of price increases occur in June.

Assets

A final note - I cannot make any charts on the asset table because, despite the presence of a creation date - running the query on our 2 million assets results in database meltdown. I can however give some quick figures on the asset table, there are 2,164,534 assets on OSGrid at present occupying a mere 65.4 GB (well less than a percentage of the Linden Lab asset cluster). Part of this is because unless an asset is in inventory, the central asset servers do not care to know about it.