OK, so we didn’t quite get to 100 as originally planned – but this time it wasn’t OpenSim’s fault. Yes, by the end you could tell the sim was straining – and at about 65 avatars, the physics engine finally choked on trying to solve a 15 avatar capsule interpenetration (or at least, my interpretation of the bug – analysis pending); but it kept on accepting logins and people kept arriving – and very quickly we hit 70, … 75, … 80 then peaked at 85 before running out of people, slipping back to 79 and manually shutting the sim down to grab the all important debug dump.

It’s important to note here – these were real clients, using SL-derived viewers. By comparison libsl is a lot friendlier on the packet engine than the full viewer, so bots tend to be a less effective test. (Plus users introduce randomness that bots cant quite emulate). Wright Plaza with 85 avatars and their attachments weighs in at a healthy 15,400 prims – so there was no shortage of texture of prim data to be sent to each client – it’s actually probably one of the nastiest sims to do load tests in – which makes it great for this. Furthermore the hardware it is located on isn’t exactly top of the line, or even middle-of-the-line.

The short news is – we’ve made some really impressive progress in the the last week. Earlier we got up to 50 – which was tweaked, tailored and adjusted to get us to where 100 or even 150 isn’t really that out of the question anymore. There’s three big causes for this – first, abandoning OpenJpeg for decoding J2K textures made some very noticable improvements to stability (it’s in progress to abandon it for Encoding too); this means we’re not crashing on the way up – which means we can hit higher concurrencies more reliably. Second – John Hurliman from Intel rewrote our throttle routines and some low-level packeting code, which delivered a big boost to packet performance. Third – multiple efforts to reduce memory use in key places, has at least halved operating memory requirements – at 85 concurrent, memory was peaking at a mere 1.7gb (~20mb/user).

A result of these improvements has been memory IO is no longer such a major bottleneck – we’re actually beggining to hit the point where CPU usage is nearly becoming a more important bottleneck (we were hitting 90% CPU at peak — although the physics interpenetration mentioned above might be distorting this, since it could lead to run-away CPU use) – which is a refreshing change, since it is a lot easier to optimise around, and the tools for CPU use profiling are a lot better than those for memory IO profiling – and produce a lot more meaningful information.

We’d like to continue these load tests – the information the devs have gotten in the last week has been absolutely invaluable. Having a big pool of testers able to jump in on a moments notice has resulted in getting performance fixes tested and integrated a lot faster than usual – it’s also helped stability, each crash has been diagnosed and debugged in series as it is encountered. It’d be very easy to say that performance & stability wise, more has happened in the last week than the last 6 months – and we still need your help to keep going. We’re going to be continuing these load tests next week – there will probably be another major effort at getting 100+ avatars in a sim next Friday (same time, 1PM PST). If you want to know when the next test is planned, and help out – either hang around in #opensim on Freenode, or follow @osgrid or @adamfrisby where I’ll announce them they come.

Next stop, 150.