OK, so we didn’t quite get to 100 as originally planned – but this time it wasn’t OpenSim’s fault. Yes, by the end you could tell the sim was straining – and at about 65 avatars, the physics engine finally choked on trying to solve a 15 avatar capsule interpenetration (or at least, my interpretation of the bug – analysis pending); but it kept on accepting logins and people kept arriving – and very quickly we hit 70, … 75, … 80 then peaked at 85 before running out of people, slipping back to 79 and manually shutting the sim down to grab the all important debug dump.

It’s important to note here – these were real clients, using SL-derived viewers. By comparison libsl is a lot friendlier on the packet engine than the full viewer, so bots tend to be a less effective test. (Plus users introduce randomness that bots cant quite emulate). Wright Plaza with 85 avatars and their attachments weighs in at a healthy 15,400 prims – so there was no shortage of texture of prim data to be sent to each client – it’s actually probably one of the nastiest sims to do load tests in – which makes it great for this. Furthermore the hardware it is located on isn’t exactly top of the line, or even middle-of-the-line.

The short news is – we’ve made some really impressive progress in the the last week. Earlier we got up to 50 – which was tweaked, tailored and adjusted to get us to where 100 or even 150 isn’t really that out of the question anymore. There’s three big causes for this – first, abandoning OpenJpeg for decoding J2K textures made some very noticable improvements to stability (it’s in progress to abandon it for Encoding too); this means we’re not crashing on the way up – which means we can hit higher concurrencies more reliably. Second – John Hurliman from Intel rewrote our throttle routines and some low-level packeting code, which delivered a big boost to packet performance. Third – multiple efforts to reduce memory use in key places, has at least halved operating memory requirements – at 85 concurrent, memory was peaking at a mere 1.7gb (~20mb/user).

A result of these improvements has been memory IO is no longer such a major bottleneck – we’re actually beggining to hit the point where CPU usage is nearly becoming a more important bottleneck (we were hitting 90% CPU at peak — although the physics interpenetration mentioned above might be distorting this, since it could lead to run-away CPU use) – which is a refreshing change, since it is a lot easier to optimise around, and the tools for CPU use profiling are a lot better than those for memory IO profiling – and produce a lot more meaningful information.

We’d like to continue these load tests – the information the devs have gotten in the last week has been absolutely invaluable. Having a big pool of testers able to jump in on a moments notice has resulted in getting performance fixes tested and integrated a lot faster than usual – it’s also helped stability, each crash has been diagnosed and debugged in series as it is encountered. It’d be very easy to say that performance & stability wise, more has happened in the last week than the last 6 months – and we still need your help to keep going. We’re going to be continuing these load tests next week – there will probably be another major effort at getting 100+ avatars in a sim next Friday (same time, 1PM PST). If you want to know when the next test is planned, and help out – either hang around in #opensim on Freenode, or follow @osgrid or @adamfrisby where I’ll announce them they come.

Next stop, 150.

So, I’m sitting here banging my head this morning over two issues. One, why the heck is terrain never deviating from zero height, and secondly why things werent looking quite right – as you could see in the previous posts it was clearly rendering prims but things were ‘missing’ or not quite right. Turns out the answer was both.

First, Terrain.

Yesterday I got the heightfield behaving properly, but couldnt understand why it was never being set when placed onto the live feed coming from the network stack. Answer turned out I was doing something stupid and had typo’d on a variable name for my indexer. Once that was fixed, we were rendering scenes like the one below.

It’s beggining to look a lot prettier, but as you may have noticed, the prims dont appear to line up in any recognisable pattern. While there is definetely a pattern there – something is very off.

Second, Objects – Part A.

Showing this one to Easy [Babcock] in the office, we quickly worked out that infact objects were never being rotated – every object had exactly the same identical zero rotation. After a few minutes debugging, this turns out to be related to the conversion from a Quaternion to a Euler rotation for WPF. Switching from Vector3 to Media.Quaternion internally solved the problem nicely.

Here’s a view of Abbotts aerodrome with the fix inplace.

Much better, although there’s still some clumps missing.

Second, Objects – Part B.

So, it’s looking like we’ve almost got this rendering correctly. At least object shape and rotation is being displayed correctly, although there’s still something lacking. It turns out that most of the bits missing corresponded nicely with camera view – so I’ve fixed this by telling libsl to ‘rove’ the camera position around the sim to download the entire thing. The above screenshot had this fix inplace.

This solved at least terrain loading completely and most objects.

Loading it up on OSGrid in Wright Plaza – everything seems to actually render properly now, at least so far as primitives go – we’re missing sculpties right now which form a large component of Wright Plaza’s design.

Second, Objects – Part C!? Huh?

Panning our Camera around a little, we notice something a little bit … odd. Namely that around 0,0,0 there’s a large congregation of primitives. I’ve been noticing this already and had discarded it as possibly being neighbouring sims in which case, I’ll just knock them out later.

But it turns out, there’s valuable prims being thrown there – it looks to me like maybe the child primitives in link sets having their “Position” being relative to the parent primitive rather than the sim (in OpenSim we have both .Position and .AbsolutePosition to seperate these).

So, I’m going to be working on that for the rest of this afternoon – after which I’m going to play with either texture rendering, or getting Meshmeriser to work so we can discard the dependency on the black-box GPL’d rendering library.