• From: the Netherlands
  • Registered: 2006-06-22
  • Posts: 899

Topic: Mongrel stops?

hi guys,

I'm having a new problem with Mongrel.. When I first run it, it seems to be working fine but after a day or two it bails. Apache starts reporting this error:

[Mon Feb 12 10:32:35 2007] [error] [client *ip*] proxy: error reading status line from remote server 127.0.0.1
[Mon Feb 12 10:32:35 2007] [error] [client *ip*] proxy: Error reading from remote server returned by /

And a bit later this:
[Mon Feb 12 11:23:23 2007] [error] proxy: BALANCER: (balancer://mongrel_cluster_enveloppen.nl). All workers are in error state

The mongrel process is still running but trying to access http://10.0.0.6:8001 directly just times out after a while.. When try to restart mongrel this is what happens:

Mongrel::TimeoutError (Mongrel timed out this thread: shutdown):
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:675:in `write'
    /var/lib/gems/1.8/gems/activerecord-1.15.1/lib/active_record/vendor/mysql.rb:532:in `write'
    /var/lib/gems/1.8/gems/activerecord-1.15.1/lib/active_record/vendor/mysql.rb:506:in `command'
    /var/lib/gems/1.8/gems/activerecord-1.15.1/lib/active_record/vendor/mysql.rb:238:in `close'
    /var/lib/gems/1.8/gems/activerecord-1.15.1/lib/active_record/connection_adapters/mysql_adapter.rb:236:in `disconnect!'
    /var/lib/gems/1.8/gems/activerecord-1.15.1/lib/active_record/connection_adapters/mysql_adapter.rb:231:in `reconnect!'
    /var/lib/gems/1.8/gems/activerecord-1.15.1/lib/active_record/connection_adapters/abstract_adapter.rb:93:in `verify!'
    /var/lib/gems/1.8/gems/activerecord-1.15.1/lib/active_record/connection_adapters/abstract/connection_specification.rb:109:in `verify_active_connections!'
    /var/lib/gems/1.8/gems/activerecord-1.15.1/lib/active_record/connection_adapters/abstract/connection_specification.rb:108:in `each_value'
    /var/lib/gems/1.8/gems/activerecord-1.15.1/lib/active_record/connection_adapters/abstract/connection_specification.rb:108:in `verify_active_connections!'
    /var/lib/gems/1.8/gems/rails-1.2.1/lib/dispatcher.rb:111:in `prepare_application'
    /var/lib/gems/1.8/gems/rails-1.2.1/lib/dispatcher.rb:39:in `dispatch'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel/rails.rb:78:in `process'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel/rails.rb:76:in `synchronize'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel/rails.rb:76:in `process'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:618:in `process_client'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:617:in `each'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:617:in `process_client'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:736:in `run'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:736:in `initialize'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:736:in `new'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:736:in `run'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:720:in `initialize'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:720:in `new'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel.rb:720:in `run'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel/configurator.rb:271:in `run'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel/configurator.rb:270:in `each'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel/configurator.rb:270:in `run'
    /var/lib/gems/1.8/gems/mongrel-1.0/bin/mongrel_rails:127:in `run'
    /var/lib/gems/1.8/gems/mongrel-1.0/lib/mongrel/command.rb:211:in `run'
    /var/lib/gems/1.8/gems/mongrel-1.0/bin/mongrel_rails:243
    /bin/mongrel_rails:18:in `load'
    /bin/mongrel_rails:18

After that I have to stop and start Mongrel manually, which makes it work again. TimeoutError is making me think Mongrel's become unresponsive..

Any ideas what could be causing something like this?

-- marsvin

Last edited by marsvin (2007-02-12 08:16:21)

Tutorial:
* Multiple child models in a dynamic form
  • Ceres
  • First Class
  • Offline
  • Registered: 2007-01-13
  • Posts: 115

Re: Mongrel stops?

Do you have any idea as to how much memory your dispatches are using? I'm no expert but it sounds like a memory leak of some sort in your application if it stops running after a few days...

http://weblog.textdrive.com/article/175 … urce-usage

These tips are for shared hosting, but can give you insight into your own issue even on a private server.

  • From: the Netherlands
  • Registered: 2006-06-22
  • Posts: 899

Re: Mongrel stops?

I've checked out that blog, there's a lot of useful stuff in there but mostly it either doesn't apply (no fastcgi) or I'm already doing it (keeping an eye on the logs, keeping an eye on system resources.) That point about fastcgi leaking memory in development mode doesn't apply to mongrel I assume right?

The mongrel process in question does run on a vps but ram and swap utilization haven't exceeded more than 15% so far.. Also OpenVZ keeps track of whenever processes run out of resources (whether it's memory, disk space or anything else) but it shows now errors.

# vzmemcheck
Output values in %
 LowMem  LowMem     RAM MemSwap MemSwap   Alloc   Alloc   Alloc
   util  commit    util    util  commit    util  commit   limit
   1.41  105.00   11.85    2.03   48.29    3.37   48.29   52.40

This represents resource usage (in %) of all vps combined. Util is the amount used, commit is the amount allocated for use by the vps. As long as the util columns stay under the commit values things should be good.

Finally the apache/mongrel processes themselves look pretty lean in my opinion.. Here's what ps reports:
[code=ps] USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     13984  0.0  0.2 104444  9452 ?        Ss   11:44   0:00 /usr/sbin/apache2 -k start
www-data 13989  0.0  0.2 105800  9952 ?        S    11:44   0:09 /usr/sbin/apache2 -k start
www-data 13990  0.0  0.2 105776  9668 ?        S    11:44   0:01 /usr/sbin/apache2 -k start
www-data 13991  0.0  0.2 105804  9668 ?        S    11:44   0:01 /usr/sbin/apache2 -k start
www-data 13992  0.0  0.2 105772  9644 ?        S    11:44   0:01 /usr/sbin/apache2 -k start
www-data 13993  0.0  0.2 105844  9704 ?        S    11:44   0:00 /usr/sbin/apache2 -k start
root     14138  0.0  1.4 127944 58996 ?        Sl   11:45   0:22 /usr/bin/ruby1.8 /bin/mongrel_rails start -d -p 8001
www-data 14140  0.0  0.2 105792  9656 ?        S    11:45   0:01 /usr/sbin/apache2 -k start
www-data 14141  0.0  0.2 105828  9684 ?        S    11:45   0:00 /usr/sbin/apache2 -k start
www-data 14142  0.0  0.2 105784  9896 ?        S    11:45   0:01 /usr/sbin/apache2 -k start
www-data 16219  0.0  0.2 105852  9956 ?        S    13:27   0:00 /usr/sbin/apache2 -k start
www-data 17934  0.0  0.2 105760  9584 ?        S    14:29   0:00 /usr/sbin/apache2 -k start[/code]
1.4% mem for the mongrel process itself and .2 for each of the apache proxies.

I'll keep an eye on resource usage the next few days but I can't see that being the problem to be honest. Really all I've been doing is running one or two test rails apps so far.

-- marsvin

Edit:
I just remembered something, could this be related to my upgrading to rails 1.2 and not properly freezing the app?

Last edited by marsvin (2007-02-12 15:09:11)

Tutorial:
* Multiple child models in a dynamic form
  • From: the Netherlands
  • Registered: 2006-06-22
  • Posts: 899

Re: Mongrel stops?

It may have been an ImageMagick related problem. I noticed every time the app crashed it was preceded by ImageMagick reporting a file not found error. I haven't been able to reproduce it manually (or even stresstesting with ab) but for now I've made the app check the file before it tries to open it (which I should've done in the first place) and will keep an eye on it.. If the crashes persist I'll remove the IM method altogether and just pre-process the images, see if that makes a difference..

Tutorial:
* Multiple child models in a dynamic form
  • Ceres
  • First Class
  • Offline
  • Registered: 2007-01-13
  • Posts: 115

Re: Mongrel stops?

ImageMagik seems to cause a lot of problems. I just recently learned about this library:

http://seattlerb.rubyforge.org/ImageScience.html

Maybe that is an alternative for you if ImageMagik really is the culprit?

  • From: the Netherlands
  • Registered: 2006-06-22
  • Posts: 899

Re: Mongrel stops?

I'll have a look, thanks for the tip smile

When and if the problem becomes clearer I'll keep the forum posted wink

Tutorial:
* Multiple child models in a dynamic form
  • From: the Netherlands
  • Registered: 2006-06-22
  • Posts: 899

Re: Mongrel stops?

So far no luck.. I've tried deploying a second site to the test environment which doesn't use IM at all. Running the ab stress test it works just fine but after a few hours the app just hangs and all I can do is restart mongrel.. Same errors as before "Error calling Dispatcher.dispatch #<Mongrel::TimeoutError: Mongrel timed out this thread: shutdown>"

Still nothing that indicates memory leaks or unreasonable use of resources.

More later...

Tutorial:
* Multiple child models in a dynamic form
  • Ceres
  • First Class
  • Offline
  • Registered: 2007-01-13
  • Posts: 115

Re: Mongrel stops?

What version of Mongrel are you using?

Not sure if you've seen this thread:

http://rubyforge.org/pipermail/mongrel- … 01774.html

Could it be something wrong with Mongrel and not your application?

  • From: the Netherlands
  • Registered: 2006-06-22
  • Posts: 899

Re: Mongrel stops?

Sounds a lot like what I was experiencing, the thread seems to be on mongrel 0.3 though whereas I've been using 1.0.. I'd expect whatever caused their problems to be fixed by now (not that I fully understood what they were saying wink)

That said I've since moved the test app from the development environment to production after reading about possible memory leaks in certain plugins when running in development. This seems (so far) to stabilize things but that could just be because more stuff is cached so the problem doesn't surface as quickly I suppose.

But thanks for that link, I'll be digging into that some more... smile

Last edited by marsvin (2007-02-15 16:23:52)

Tutorial:
* Multiple child models in a dynamic form
  • Ceres
  • First Class
  • Offline
  • Registered: 2007-01-13
  • Posts: 115

Re: Mongrel stops?

Yea... I hate unsolvable deployment issues. I hope you can find the solution, because I'm guessing I'll run into the same issues some day.

  • From: the Netherlands
  • Registered: 2006-06-22
  • Posts: 899

Re: Mongrel stops?

Especially ones with vague symptoms like this, can't help but feel it'll come back to bite me on the backside.. My friend said I was being stupid trying to keep apps running in development mode for extended time to begin with but that's never been a problem before.

Like I said I'll keep the board posted if there's anything new to report.

Tutorial:
* Multiple child models in a dynamic form
  • From: the Netherlands
  • Registered: 2006-06-22
  • Posts: 899

Re: Mongrel stops?

To anyone still following this, apparently mongrel should automatically install the fastthread gem as a dependency. It hadn't in my case and apparently that's a problem as the default ruby thread handling mechanism has a tendency to time out.

I installed the fastthread gem

  gem install fastthread

And so far the problem hasn't reoccured. I'll keep you posted if it does.

Tutorial:
* Multiple child models in a dynamic form
  • Ceres
  • First Class
  • Offline
  • Registered: 2007-01-13
  • Posts: 115

Re: Mongrel stops?

That's kind of strange as the mongrel gem asked if I wanted to install fastthread when I installed mongrel. Good to know that you are well on your way to solving your issue though.

  • From: the Netherlands
  • Registered: 2006-06-22
  • Posts: 899

Re: Mongrel stops?

I don't believe it was in my case but I'm not 100% sure. In any case, today one of the instances went down again so it's not the solution I was looking for :\

Tutorial:
* Multiple child models in a dynamic form
  • From: the Netherlands
  • Registered: 2006-06-22
  • Posts: 899

Re: Mongrel stops?

Heyhey,

After considerable time searching for the solution to this and a ton of red herrings, it seems like the problem is finally resolved.

Here's where I found the answer: Rails-deploy ml

It seems the ruby-based mysql driver is a bit sluggish in some respects and causes mongrel to timeout its connection to the db. Installing the binary version (in my case as simple as an "aptitude install libmysql-ruby") fixed it..

It's been a few days and I haven't had a mongrel go down yet so it looks like this time it's for real wink

Tutorial:
* Multiple child models in a dynamic form
  • Ceres
  • First Class
  • Offline
  • Registered: 2007-01-13
  • Posts: 115

Re: Mongrel stops?

Awesome. I hope I never run into this issue, but if I do, at least I'll know what to try first.

Thanks.