Project

General

Profile

Bug #2295

segmentation faults

Added by tomer.doron (tomer doron) over 9 years ago. Updated about 8 years ago.

Status:
Open
Priority:
Normal
Assignee:
-
ruby -v:
ruby 1.9.1p243 (2009-07-16 revision 24175) [i686-linux]
[ruby-core:26371]

Description

=begin
My server crashes 3-4 times a day with segmentation faults. the fault is not always a same spot, so the attached is just an example of one occurrence. I will keep adding more reports as they come along.

The application is a ruby in rails stack on top of mysql.

Ruby compiled from source on fedora 8 running on EC2.
=end


Files

segfault1.txt (20.5 KB) segfault1.txt tomer.doron (tomer doron), 10/28/2009 08:41 AM
segfault2.txt (3.55 KB) segfault2.txt tomer.doron (tomer doron), 10/28/2009 08:45 AM
segfault3.txt (20.5 KB) segfault3.txt tomer.doron (tomer doron), 10/28/2009 08:46 AM
segfault3.txt (16 KB) segfault3.txt tomer.doron (tomer doron), 10/28/2009 08:49 AM
related_error.txt (1.9 KB) related_error.txt tomer.doron (tomer doron), 10/28/2009 12:43 PM
segfault4.txt (29.6 KB) segfault4.txt tomer.doron (tomer doron), 10/28/2009 03:43 PM

Related issues

Related to Backport191 - Bug #2019: Segmentation fault in ruby 1.9.1-p243 on second requestRejected08/31/2009Actions
Related to Backport191 - Bug #2309: Segfault, repeatable testcase, possibly related to [2295], [2019], [1937].Feedback10/30/2009Actions
Related to Ruby master - Bug #4350: Segmentation fault and Bus ErrorRejected01/31/2011Actions

History

#1

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
another example
=end

#2

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
same as attachment 1, so you can ignore this one
=end

#3

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
another example
=end

#4

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
looks similar to what is described in #2019
=end

#5

Updated by nobu (Nobuyoshi Nakada) over 9 years ago

  • Status changed from Open to Feedback

=begin
Which of MySQL/Ruby or Ruby/MySQL, and what version of it and Rails?
=end

#6

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
rails 2.3.4
mysql gem 2.8.1

=end

#7

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
another c level exception that shows up in log and may be related
=end

#8

Updated by marcandre (Marc-Andre Lafortune) over 9 years ago

  • Status changed from Feedback to Open

=begin

=end

#9

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
and one last example for today :)
=end

#10

Updated by rogerdpack (Roger Pack) over 9 years ago

=begin
any way to reliably recreate? Maybe try it with the trunk version of 1.9.1?
GL.
-r
=end

#11

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
while this is happening on production several times a day, there is not a specific scenario i can refer you to. it is more random in nature like memory issues tend to be. so, while I cant say it is easily recreatable, it is most certainly reliably recreatable... I can keep uploading the crash reports as they come through.

few data points:

  • this is running on fedora 8 VM in amazon EC2
  • ruby compiled from source.

=end

#12

Updated by rogerdpack (Roger Pack) over 9 years ago

=begin
Have you tried it on other servers? under valgrind?
Can you create unit tests that thrash the system until it occurs? (No reproducibility...hard).
-r
=end

#13

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
Same exact applications was deployed on a different production environment using OSX server with ruby 1.9.1 from MacPorts, I did not experience these issues in that environment. As I moved my servers to amazon EC2 I redeployed on amazon's Fedora 8 core VMs and compiled ruby 1.9.1 from source. At first everything seemed fine, but then as users started pounding the system these crashes started to appear, as this is a production environment I could not stay with it crashing all the time and I rebuilt the server with ruby 1.8 and the issues are gone.

I still have the 1.91. installation available, but without stress it would be more difficult to recreate. I can certainly install valgrind and do some further debugging. will update with info once I have some.
=end

#14

Updated by jarrednicholls (Jarred Nicholls) over 9 years ago

=begin
tomer,

I ran into seg faults on my EC2 boxes with 1.9.1 as well. If your issue is that same as mine, you need to change your kernel and ramdisk for your EC2 instance from the default (which I believe is 2.6.21) to a different version (i.e., 2.6.18). Amazon is doing updates to fix their 2.6.21 and their Ubuntu Server 2.6.31 (beta) kernel. But, if you load the 2.6.18 kernel and associated ramdisk, you might not have those problems anymore. However, my issue literally crashed the kernel/instance and forced a reboot of the instance. But, you can at least give it a shot.

Kernel ID: aki-f5c1219c
Ramdisk ID: ari-dbc121b2

I wish you luck. For me, I simply went back to using Passenger with Ruby EE (which is 1.8.7), and did a patch to the Passenger gem to do asynchronous request dispatching to allow for long running I/O requests to run in parallel so they wouldn't block short quick requests from getting through. I'll move to 1.9 when everything is more stable, and/or go to JRuby if I need true & fast multi-threading. I tried JRuby and was having issues with Cookie Store and decided its memory footprint was way too big for the time being.

Jarred
=end

#15

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
I have set up a new EC2 instance with the suggested kernel, but i am afraid that has not solved the issue and I am still getting similar segmentation faults. I will have to roll back again to 1.8 until this is resolved.

Because the segmentation faults are inconsistent and not unique to one area of the code, I assume this is somewhere deeper in the EC2 kernel or core ruby code or (and more likely) in the combination of 1.9 and EC2 kernel. if anyone has good suggestions as to how to trace the root cause, I am willing to invest the time to do the research.
=end

#16

Updated by tomer.doron (tomer doron) over 9 years ago

=begin
attempted upgrading to 1.9.1-p376, ruby compiled from source using gcc with --enable-shared option

similar segmentation faults occur on both fedora 8 and ubuntu 9.04 jaunty

=end

Also available in: Atom PDF