Comments on: How to choose a good OpenSim host? http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/ ZOMGWTFHAI Tue, 02 Feb 2010 23:36:21 +0000 http://wordpress.org/?v=2.8.4 hourly 1 By: Marilyn Hill http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/comment-page-1/#comment-8560 Marilyn Hill Thu, 05 Nov 2009 18:44:36 +0000 http://www.adamfrisby.com/blog/?p=199#comment-8560 hey thanks, adam, i really love that chat facility i went into from osgrid, i got lots of questions answered there already, and i have 6 regions going good now so far because of the chat answers. its awesome, cheers!!! nowto work out the hosting, lol thanks for that great info here, too. hey thanks, adam, i really love that chat facility i went into from osgrid, i got lots of questions answered there already, and i have 6 regions going good now so far because of the chat answers. its awesome, cheers!!! nowto work out the hosting, lol thanks for that great info here, too.

]]> By: Adam Frisby http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/comment-page-1/#comment-8518 Adam Frisby Sun, 18 Oct 2009 20:04:34 +0000 http://www.adamfrisby.com/blog/?p=199#comment-8518 Not a clue, sorry Marilyn - you might want to try the normal support route of the opensim-users mailing list. Not a clue, sorry Marilyn – you might want to try the normal support route of the opensim-users mailing list.

]]> By: Marilyn Hill http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/comment-page-1/#comment-8514 Marilyn Hill Sat, 17 Oct 2009 17:22:40 +0000 http://www.adamfrisby.com/blog/?p=199#comment-8514 hi Adam, would you have any idea why there would be some kind of warning on the opensim folder that i downloaded as a binary installer, the 0.6.6 version. it looks like a red, round circle with a little exclaimation mark inside the red circle, and this is on the folder itself. also, the whole folder is read only and when i try to change that, it wont, and i cant change anything about the opensim i created, everytime i try to modify the external host name in the regions.ini file, it wont allow the change. i do have a brand new 8gig pc with that nasty windows vista 64 bit, and there is no way to even run anything in this folder as administrator to even get any permission to change anything the way it needs to be done. if you have any suggestions about what i could/should do about any of this i would so much appreciate it. :-) hi Adam,
would you have any idea why there would be some kind of warning on the opensim folder that i downloaded as a binary installer, the 0.6.6 version. it looks like a red, round circle with a little exclaimation mark inside the red circle, and this is on the folder itself. also, the whole folder is read only and when i try to change that, it wont, and i cant change anything about the opensim i created, everytime i try to modify the external host name in the regions.ini file, it wont allow the change. i do have a brand new 8gig pc with that nasty windows vista 64 bit, and there is no way to even run anything in this folder as administrator to even get any permission to change anything the way it needs to be done.

if you have any suggestions about what i could/should do about any of this i would so much appreciate it. :-)

]]> By: Dr B the Master http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/comment-page-1/#comment-8477 Dr B the Master Sat, 03 Oct 2009 12:26:45 +0000 http://www.adamfrisby.com/blog/?p=199#comment-8477 Adam,,,,,according to your earlier, estimated calculations, quote "So a region with 5,000 primitives, an average number of textures (50mb) and ten concurrent users you should allocate approximately 900MB of memory for .NET and 1.8GB for Mono.", this means for Mono, and 16 regions, a server WILL need (not forgetting scripts) approx 40GB RAM, and I don't think there's a computer system in existence that can access that amount of RAM. Quad cores can only access a max of 32GB, so for the everyday Joe, with 16GB, Duel Core, would only be able to run 5 regions, 5,000 prims, 10 concurrent avatars, 50MB of textures, and approx 100 active scripts (script memory usage depends on type of length of script). Adam,,,,,according to your earlier, estimated calculations, quote “So a region with 5,000 primitives, an average number of textures (50mb) and ten concurrent users you should allocate approximately 900MB of memory for .NET and 1.8GB for Mono.”, this means for Mono, and 16 regions, a server WILL need (not forgetting scripts) approx 40GB RAM, and I don’t think there’s a computer system in existence that can access that amount of RAM. Quad cores can only access a max of 32GB, so for the everyday Joe, with 16GB, Duel Core, would only be able to run 5 regions, 5,000 prims, 10 concurrent avatars, 50MB of textures, and approx 100 active scripts (script memory usage depends on type of length of script).

]]> By: Adam Frisby http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/comment-page-1/#comment-8225 Adam Frisby Wed, 24 Jun 2009 03:09:37 +0000 http://www.adamfrisby.com/blog/?p=199#comment-8225 Hi Miguel, I can tell you this is a persistent issue with Mono - whether or not the issue has spread is not my concern, the issue is primarily that Mono does have serious memory management problems - as I outlined before, the issue is mostly visible with scripts, users and time. Any tests you do need to have concurrent users (you can use some of our demo bots such as pcampbot to see the problem) and it needs to run for at least 24hrs. I would suggest contacting 'nebadon' on #opensim-dev on irc.freenode.net and asking him to take a look at Wright Plaza, which is the best example - this server needs to be restarted daily due to this memory issue. Adam Hi Miguel,

I can tell you this is a persistent issue with Mono – whether or not the issue has spread is not my concern, the issue is primarily that Mono does have serious memory management problems – as I outlined before, the issue is mostly visible with scripts, users and time. Any tests you do need to have concurrent users (you can use some of our demo bots such as pcampbot to see the problem) and it needs to run for at least 24hrs.

I would suggest contacting ‘nebadon’ on #opensim-dev on irc.freenode.net and asking him to take a look at Wright Plaza, which is the best example – this server needs to be restarted daily due to this memory issue.

Adam

]]> By: Adam Frisby http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/comment-page-1/#comment-8211 Adam Frisby Sun, 14 Jun 2009 22:34:13 +0000 http://www.adamfrisby.com/blog/?p=199#comment-8211 Recycling instances is quite possible - it's easy enough to do. The problem is mainly resuming user sessions; in the process of resuming - we lose all the user socket connections (they get disconnected, etc). Startup time can also be a problem; initialising user scripts by the truckload (not uncommon to have 1000+ user scripts running simultaneously.) and other things can prevent a very quick resume. The big issue is that unlike apache, we're maintaining a multiuser session environment - that means that each user is experiencing the same data being updated in realtime; if we split users across processes, we could end up with a horrible amount of duplicated data in memory; along with the woes of interprocess communication. Right now, we do restart the process regularly - but it's by force rather than choice (ie we hit swap and everything stops). Recycling instances is quite possible – it’s easy enough to do.

The problem is mainly resuming user sessions; in the process of resuming – we lose all the user socket connections (they get disconnected, etc).

Startup time can also be a problem; initialising user scripts by the truckload (not uncommon to have 1000+ user scripts running simultaneously.) and other things can prevent a very quick resume.

The big issue is that unlike apache, we’re maintaining a multiuser session environment – that means that each user is experiencing the same data being updated in realtime; if we split users across processes, we could end up with a horrible amount of duplicated data in memory; along with the woes of interprocess communication.

Right now, we do restart the process regularly – but it’s by force rather than choice (ie we hit swap and everything stops).

]]> By: Adam Frisby http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/comment-page-1/#comment-8210 Adam Frisby Sun, 14 Jun 2009 22:30:47 +0000 http://www.adamfrisby.com/blog/?p=199#comment-8210 We use a mix - but the experience is fairly consistent. 1.9/2.0 did give us some benefits, and the newer 2.5 builds have improved things - we're now able to keep some of the regions up for up to 2 days between restarts. I can tell you that the problem is visible on both x64 and 86 architectures, for at least the following versions; 1.9.1, 2.0, 2.2, 2.5. The problem does not appear to be unique to any distribution or kernel version; and is fairly reproducible under any test conditions. We use a mix – but the experience is fairly consistent.

1.9/2.0 did give us some benefits, and the newer 2.5 builds have improved things – we’re now able to keep some of the regions up for up to 2 days between restarts.

I can tell you that the problem is visible on both x64 and 86 architectures, for at least the following versions; 1.9.1, 2.0, 2.2, 2.5. The problem does not appear to be unique to any distribution or kernel version; and is fairly reproducible under any test conditions.

]]> By: Miguel de Icaza http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/comment-page-1/#comment-8209 Miguel de Icaza Sun, 14 Jun 2009 22:30:18 +0000 http://www.adamfrisby.com/blog/?p=199#comment-8209 Adam, I have a final question. It seems to me that OpenSim must have some sort of system to snapshot the system and resume execution from a snapshot to ensure that it can recover from a crash. A crash could be caused by all sorts of events, like operating system crashes, power failures, bugs in OpenSim itself, the underlying libraries, or any other source of problems. If this is the case, I assume that OpenSim would support some mechanism to serialize its state, and resume execution from a serialized state. If that is the case, and running OpenSim on Mono does have leaks, what is preventing admins from recycling OpenSim instances (this is common in server scenarios, for example Apache recycles processes after certain number of transactions). And this is also how Google ensures operations: they assume that there will be failures, and write the code to recover from failures. Miguel. Adam,

I have a final question.

It seems to me that OpenSim must have some sort of system to snapshot the system and resume execution from a snapshot to ensure that it can recover from a crash.

A crash could be caused by all sorts of events, like operating system crashes, power failures, bugs in OpenSim itself, the underlying libraries, or any other source of problems.

If this is the case, I assume that OpenSim would support some mechanism to serialize its state, and resume execution from a serialized state.

If that is the case, and running OpenSim on Mono does have leaks, what is preventing admins from recycling OpenSim instances (this is common in server scenarios, for example Apache recycles processes after certain number of transactions).

And this is also how Google ensures operations: they assume that there will be failures, and write the code to recover from failures.

Miguel.

]]> By: Miguel de Icaza http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/comment-page-1/#comment-8208 Miguel de Icaza Sun, 14 Jun 2009 22:21:34 +0000 http://www.adamfrisby.com/blog/?p=199#comment-8208 Hello Adam, Thanks for the extra information. Many objects is not a problem for the GC. It just means that the GC will take longer, but it does not mean that it will give up on collecting them. Mono has one important difference when it comes to memory management when compared to .NET, it does not have a compacting collector (yet) and older versions of Mono used too many resources (generics caused vtables to bloat out of proportion, megabytes of tables at a time). Compacting collection can become a problem if allocations have a pattern like this: allocate large object, wait for it to be collected, allocate objects that fit in the previous hole, allocate the same large object (triggering a new heap expansion). It would help if you guys could provide information as the configuration that you are using (Mono version, Linux version, architecture). Hello Adam,

Thanks for the extra information.

Many objects is not a problem for the GC. It just means that the GC will take longer, but it does not mean that it will give up on collecting them.

Mono has one important difference when it comes to memory management when compared to .NET, it does not have a compacting collector (yet) and older versions of Mono used too many resources (generics caused vtables to bloat out of proportion, megabytes of tables at a time).

Compacting collection can become a problem if allocations have a pattern like this: allocate large object, wait for it to be collected, allocate objects that fit in the previous hole, allocate the same large object (triggering a new heap expansion).

It would help if you guys could provide information as the configuration that you are using (Mono version, Linux version, architecture).

]]> By: Adam Frisby http://www.adamfrisby.com/blog/2009/05/how-to-choose-a-good-opensim-host/comment-page-1/#comment-8207 Adam Frisby Sun, 14 Jun 2009 22:04:15 +0000 http://www.adamfrisby.com/blog/?p=199#comment-8207 Hey Miguel, The problem isnt really visible until you start getting users involved. OpenSim wont use nearly any memory until you have regular users and a reasonably complex scene. The problem appears to be leak related - we recycle a lot of classes internally; which results in a lot of objects to track in the GC, however, when memory is allocated in Mono - it is never reclaimed back later. We're not sure why - memory dumps dont indicate much (if and when they run at all.) Under .NET - it does get reclaimed. Memory use for a Windows machine isnt one directional - after 12 or so hours, Mono can be consuming 2+ GB, while .NET is still chugging along at 400mb. (Although they will have both started out similarly.) Our test servers for OSGrid are probably the best demonstration of the problem - you can see the memory charts here: http://plaza02.osgrid.org/stats/detail.php?graph=3&tree=&filter= for Mono. For Windows, I'll need to get in contact with you and arrange some kind of demo. Do you want to send me an email: adam@deepthink.com.au Hey Miguel,

The problem isnt really visible until you start getting users involved. OpenSim wont use nearly any memory until you have regular users and a reasonably complex scene.

The problem appears to be leak related – we recycle a lot of classes internally; which results in a lot of objects to track in the GC, however, when memory is allocated in Mono – it is never reclaimed back later. We’re not sure why – memory dumps dont indicate much (if and when they run at all.)

Under .NET – it does get reclaimed. Memory use for a Windows machine isnt one directional – after 12 or so hours, Mono can be consuming 2+ GB, while .NET is still chugging along at 400mb. (Although they will have both started out similarly.)

Our test servers for OSGrid are probably the best demonstration of the problem – you can see the memory charts here: http://plaza02.osgrid.org/stats/detail.php?graph=3&tree=&filter= for Mono. For Windows, I’ll need to get in contact with you and arrange some kind of demo. Do you want to send me an email: adam@deepthink.com.au

]]>