In the latest round of performance tests with OpenSim we invited our office as well as the contents of the OpenSim IRC channels to help do another performance test on OpenSim. At peak we managed to get 28 avatars into the sim and got a little bit more performance information about OpenSim and where our bottlenecks are, most notably discovering a few runaway memory problems when a ‘bad client’ connects.
I’ve attached at the bottom of this post, links to where you can get the profile dumps from this, if you are a programmer working on OpenSim you might be interested in taking a look at these. We use JetBrain’s dotTrace internally, so the files are formatted in dotTrace format – there is two sets of tests, one labelled ‘pos*.dtc’ which indicates we were using the POS Physics Engine. The second set are called ‘ode*.dtc’ which utilize OpenDynamicsEngine. All of these tests are CPU tests, we dont have any memory profiles of the test, however memory usage on the server seemed quite acceptable.
Bullet points
- A client with high packetloss can cause the sim to create very large resend queues. One such client produced over 113MB of RAM used in resending packets. (Plus plenty of CPU time spent managing that large Queue.).
- There’s no indexers on EntityManager for child primitives, doing lookups on Child objects are very slow.
- POS Physics produces exponential load the more avatars moving around, ODE produces a linear load. POS consumes less however when agents < 10.
- ODE is notoriously unstable when being run under a profiler, crashing on average once every 3 minutes. POS remains stable, although produces nasty lag spikes.
- OpenSim is stable with 25 ordinary agents, however pcampbot (our stress tester) can very quickly disable a sim.
Dot Trace Files
Get them here: http://www.adamfrisby.com/Profiles/
Great to see this level of detail – a spur-of-the-moment test on the ReactionGrid dev grid resulted in 40 simultaneous logins, all from a mix of real Hippo and SL clients on all the machines in my house, plus some external connections. I didn’t gather metrics that time, but will do something similar as soon as the opportunity presents itself. http://www.flickr.com/photos/digitalstrawberrygirl/3354103759/
Chris Hart
16 Mar 09 at 10:05 am
Interesting you got 18 clients running on a single box, still I would love to try get a test with 50-100 real users. It’s just always such a pain in the organisation of it.
Adam Frisby
16 Mar 09 at 10:22 am
I was utterly amazed to get 18 on the same box, but on lowest detail it seemed to handle it. Quad core Intel chip, 8Gb ram, ATI 4870 512Mb. The max I got on the next-best box, an old P4, was 6 clients, and the rest around 3-4 each. Organising a real public test would be worthwhile though. Hoping to do so at some point in the near future.
Chris Hart
16 Mar 09 at 12:51 pm
They are all Ruth so you can’t really look at them as normal avatars. The real load starts when they have attachments, prim hair, clothing, different skins etc…
Fact that there are leaks is true, I notice this when doing server resource checks, scripts cause leaks for example making a simulator freeze up after a day or even a few days. There is a lot in the aspect of the basic functions of Opensim that needs work. I would really like to see inventory folders opening up every time you click them, when you drag something out of inventory I would really like to see that the object is there on the land. Those kind of things are what stops people from using Opensim. I hope that in 2009 the basic functions get sorted out. Many things do work fine, logins, teleports, region crossing do very well for me while they don’t for others. I have major inventory issues while others do just fine with it. I really would like to see that you get this stable over all platforms, Windows, Linux, if you get to that point the overall impression with the general public will change and more people will find use for Opensim.
I did some inventory cleaning in Second Life a few days ago and noticed they don’t have it working after 10 years so something isn’t correct with the entire system. I had 98% perfect rezzing, still 2 out of 100 objects weren’t there when I dragged them out. I also noticed several items just had vanished, another well known issue, I paid for these things and now they were gone, people get frustrated and angry over this. How could Opensim ever start to have an economy without a reliable system that handles inventory?
What would happen if this occured in a MMORPG like World of Warcraft for example, people their magic potion doesn’t rez when they need it. How would this affect the gameplay of such a platform? Any MMORPG would lose users over this.
Opensim isn’t a bit of code sitting on sourceforge to show how you can make a virtual world server, it’s a product so I’m looking forward to see improvement in the core functions.
Virtual
16 Mar 09 at 8:20 pm
>> They are all Ruth so you can’t really look at them as normal avatars. The real load starts when they have attachments, prim hair, clothing, different skins etc…
The load from those is however fairly small – each avatar introduces 3×512x512 images for clothing, plus prims, being a worst case additional ~160KB per avatar. The average scene by comparison is about 20-50MB+.
The major contributor is simply having the avatar in the packet queues – with 1 client you only need to send one update for an object when you move it. With 50 clients, your sending a very significant amount more.
Re: inventory – the big issue with inventory is not that OpenSim handles it badly (it probably does), but that the region server shouldnt be handling it at all. The SL protocol isnt designed very well from a scaling angle, and the reality is that your inventory should sit with either an ISP-level organisation (like email), or on your local machine (as a folder that’s uploaded to the sim on demand).
Failing to do that and centralising it how SL does it right now is just simply unscalable – if you have 100,000 users with 1,000 inventory items each – your handling 100 million items. Bring that to a million users, and your talking a billion plus database records. It’s just simply unsustainable.
That all being said, I’m sure OpenSim will be significantly more stable by the end of the year – we’ve seen a huge amount of progress in 2008, although I still expect to see a lot more experimental feature support this year, particularly as the designs for more scalable client infrastructure are being settled (MMOX, OGP, etc)
Adam Frisby
16 Mar 09 at 10:20 pm
Re: inventory – the big issue with inventory is not that OpenSim handles it badly (it probably does), but that the region server shouldnt be handling it at all. The SL protocol isnt designed very well from a scaling angle, and the reality is that your inventory should sit with either an ISP-level organisation (like email), or on your local machine (as a folder that’s uploaded to the sim on demand).
Yes I think you are correct about that, we did tests running Opensim on a normal pc under windowsXP and then inventory works well with all objects stored on the same machine, yet when using it with high end servers, giving it all the resources it could want it just fails and there isn’t a thing we can do about it and that is what keeps people away. They come to a grid, first thing they want is getting items out of boxes to change their appearance and there you have it. “My inventory doesn’t load” “I don’t see the object I just got out of the box” “I gave something to another avatar and it doesn’t work” “My inventory is lagging” people conclude it doesn’t work and leave.
This is a major issue and that is why I mentioned it here. I think with this type of software (as with any type) some sort of trust relation is created between user and the computer program. I’m well aware of all the hard work that is done by all of the programmers and nobody is to blaim in any way.
Funny that you mention “Storage on your local machine” perhaps the solution is in that sentence. Since all objects you see rendered in world are getting downloaded to your machine perhaps inventory could be solved this way. No more objects that need to be pulled from servers and follow a complex route to get placed in a simulator. A huge reduction in lost objects due to packet loss. Inventory server just keeps track of UUID strings and the actual inventory system works like it does for MMORPG’s which are installed on home computers. You can even look at this as a more industry standard approach compared to the way it works today. It even seems a lot more “server friendly”.
Looking forward to see what Opensim does in 2009.
Virtual
17 Mar 09 at 12:43 am
I think to start fixing some of these things we need to really have control of the viewer. It’s a reason why I hope things like IdealistViewer take off this year, since having complete control over the platform gives us so many more tools to work with.
Adam Frisby
17 Mar 09 at 1:52 am
Has any consideration been given to distributing some of the server load to the resident’s pc using a type of “cloud computing” model to cluster those services like the SETTI screen saver did? Clients could volunteer to be part of the server distribution and “share” unused processor time and RAM on their systems to help offload some of the processes that aren’t a security risk.
Myron Curtis
17 Apr 09 at 4:15 pm
Probably not – some things can be offloaded, but very few can be offloaded to the client.
Scripts are potentially offloadable – but the environment running them has to be trustworthy and on a low-latency connection to the sim. A client connection doesnt satisfy either of these conditions. Packet delivery is also more potentially offloadable – it requires reworking the protocol to add some proper cryptographic signaturing to outbound packets, but a client could act as a proxy mirror for certain large assets (eg textures). Smaller assets (like object data) is too small to worry about (transmission costs are higher than the message.).
The guys implementing BadumnaSim (see forge.opensimulator.org) have had some success with the latter – however most of the improvements are in the area of texture delivery. OpenSim has had a lot of improvements in our internal texture handler, so I’m unsure how much benefit there is for running Badumna now; but some experimentation is appreciated.
Adam Frisby
18 Apr 09 at 4:12 am