Bug #10335: ssl connect using net/http on solaris x86 fails with core dump

• Status: Third Party's Issue
• Assignee: -
• Target version: -
• ruby -v: ruby 2.1.3p242 (2014-09-19 revision 47630) [i386-solaris2.10]
• Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN

[tosmi (Toni Schmidbauer)]

we tried to compile ruby 2.1.3 on solaris x86 with the current sun studio compiler 12.3 (for running puppet).

we used the following configure options to compile openssl:

./Configure  --prefix=/opt/puppet shared solaris-x86-cc

we tried the following openssl version:

1.0.1e
1.0.1i
1.0.0n

configure options used to compile ruby:

CPPFLAGS="-I /opt/puppet/include" LDFLAGS="-R /opt/puppet/lib -L /opt/puppet/lib" ./configure --without-gcc --prefix=$(PREFIX) --enable-shared --disable-install-doc --disable-install-rdoc

we tried ruby

2.0.0-p481
2.0.0-p576
2.1.3

but the following example test scripts fails with a core dump:

require "net/https"
require "uri"

uri = URI.parse("https://<insert ssl host here>")
https = Net::HTTP.new(uri.host, uri.port)
https.use_ssl = true
https.verify_mode = OpenSSL::SSL::VERIFY_NONE

request = Net::HTTP::Get.new(uri.request_uri)

response = https.request(request)
response.body
response.status
response["header-here"] # All headers are lowercaserequire 'socket'

please find the ruby dump output attached to this bug report.

running pstack on the core file shows:

# pstack core 
core 'core' of 11021:   /opt/puppet/bin/ruby /tmp/test.rb
-----------------  lwp# 1 / thread# 1  --------------------
 fedfc067 _lwp_kill (1, 6) + 7
 feda3883 raise    (6) + 1f
 fed829e1 abort    (398, 85781e8, 83b1e28, 83b1e58, 81a5756, 82b36cc) + cd
 080c10a5 rb_bug   (82b36cc, fe9f3511) + 55
 081a5756 sigsegv  (b, 83b2104, 83b1f04) + 86
 fedf93ef __sighndlr (b, 83b2104, 83b1f04, 81a56d0) + f
 fedef0cf call_user_handler (b, 83b2104, 83b1f04) + 22b
 fedef210 sigacthandler (b, 83b2104, 83b1f04) + 7c
 --- called from signal handler with signal 11 (SIGSEGV) ---
 fe9f3511 tls1_PRF (c030, fea18040, d, 8771188, 20, 0) + 451
 fe9f4ec9 tls1_generate_master_secret (bf4229a4, 472a4e9d, f376ce83, 38d14648, b70aefec, 2615730e) + 69
 3a1f69b5 ???????? ()
-----------------  lwp# 2 / thread# 2  --------------------
 fedfbd67 __pollsys (feed3fa8, 2, 0, 0) + 7
 feda100a poll     (feed3fa8, 2, ffffffff) + 52
 082392c5 timer_thread_sleep (82e1584, 0) + 65
 082393bc thread_timer (82e1584) + 5c
 fedf8feb _thr_setup (febf0200) + 4e
 fedf92f0 _lwp_start (febf0200, 0, 0, feed3ff8, fedf92f0, febf0200)

we are on Oracle Solaris 10 1/13 s10x_u11wos_24a X86

any help would be appreciated

thanks
toni

[ngoto (Naohisa Goto)]

• Status changed from Open to Feedback

Could you please provide results of the following commands and
check if these are linked with intended OpenSSL and zlib libraries?

 % ldd /opt/puppet/lib/ruby/2.1.0/i386-solaris2.10/openssl.so
 % ldd /opt/puppet/lib/ruby/2.1.0/i386-solaris2.10/digest.so
 % ldd /opt/puppet/lib/ruby/2.1.0/i386-solaris2.10/zlib.so

In Solaris, to prevent linking with pre-installed OpenSSL in /usr/lib,
setting an environment variable PKG_CONFIG_PATH may generally be needed
during build process of ruby. For example,

 % PKG_CONFIG_PATH=/opt/puppet/lib/pkgconfig:/usr/lib/pkgconfig
 % export PKG_CONFIG_PATH

[tosmi (Toni Schmidbauer)]

Naohisa Goto wrote:

% ldd /opt/puppet/lib/ruby/2.1.0/i386-solaris2.10/openssl.so

    libssl.so.1.0.0 =>       /opt/puppet/lib/libssl.so.1.0.0
    libcrypto.so.1.0.0 =>    /opt/puppet/lib/libcrypto.so.1.0.0
    libsocket.so.1 =>        /lib/libsocket.so.1
    libnsl.so.1 =>   /lib/libnsl.so.1
    libpthread.so.1 =>       /lib/libpthread.so.1
    librt.so.1 =>    /lib/librt.so.1
    libdl.so.1 =>    /lib/libdl.so.1
    libcrypt_d.so.1 =>       /usr/lib/libcrypt_d.so.1
    libm.so.2 =>     /lib/libm.so.2
    libc.so.1 =>     /lib/libc.so.1
    libmp.so.2 =>    /lib/libmp.so.2
    libmd.so.1 =>    /lib/libmd.so.1
    libscf.so.1 =>   /lib/libscf.so.1
    libaio.so.1 =>   /lib/libaio.so.1
    libgen.so.1 =>   /lib/libgen.so.1
    libdoor.so.1 =>  /lib/libdoor.so.1
    libuutil.so.1 =>         /lib/libuutil.so.1

% ldd /opt/puppet/lib/ruby/2.1.0/i386-solaris2.10/digest.so

    libpthread.so.1 =>       /lib/libpthread.so.1
    librt.so.1 =>    /lib/librt.so.1
    libsocket.so.1 =>        /lib/libsocket.so.1
    libdl.so.1 =>    /lib/libdl.so.1
    libcrypt_d.so.1 =>       /usr/lib/libcrypt_d.so.1
    libm.so.2 =>     /lib/libm.so.2
    libc.so.1 =>     /lib/libc.so.1
    libaio.so.1 =>   /lib/libaio.so.1
    libmd.so.1 =>    /lib/libmd.so.1
    libnsl.so.1 =>   /lib/libnsl.so.1
    libgen.so.1 =>   /lib/libgen.so.1
    libmp.so.2 =>    /lib/libmp.so.2
    libscf.so.1 =>   /lib/libscf.so.1
    libdoor.so.1 =>  /lib/libdoor.so.1
    libuutil.so.1 =>         /lib/libuutil.so.1

% ldd /opt/puppet/lib/ruby/2.1.0/i386-solaris2.10/zlib.so

    libz.so.1 =>     /usr/lib/libz.so.1
    libpthread.so.1 =>       /lib/libpthread.so.1
    librt.so.1 =>    /lib/librt.so.1
    libsocket.so.1 =>        /lib/libsocket.so.1
    libdl.so.1 =>    /lib/libdl.so.1
    libcrypt_d.so.1 =>       /usr/lib/libcrypt_d.so.1
    libm.so.2 =>     /lib/libm.so.2
    libc.so.1 =>     /lib/libc.so.1
    libaio.so.1 =>   /lib/libaio.so.1
    libmd.so.1 =>    /lib/libmd.so.1
    libnsl.so.1 =>   /lib/libnsl.so.1
    libgen.so.1 =>   /lib/libgen.so.1
    libmp.so.2 =>    /lib/libmp.so.2
    libscf.so.1 =>   /lib/libscf.so.1
    libdoor.so.1 =>  /lib/libdoor.so.1
    libuutil.so.1 =>         /lib/libuutil.so.1

In Solaris, to prevent linking with pre-installed OpenSSL in /usr/lib,
setting an environment variable PKG_CONFIG_PATH may generally be needed
during build process of ruby. For example,

looks like we are using our self-compiled ssl libraries in /opt/puppet/lib.

thanks for very fast response

toni

[ngoto (Naohisa Goto)]

The function "tls1_PRF" exists in libssl.so, and I still doubt if the OpenSSL is correctly installed.
Could you please provide more information about OpenSSL?

 % ldd /opt/puppet/lib/libssl.so.1.0.0
 % ldd /opt/puppet/lib/libcrypto.so.1.0.0
 % ldd /opt/puppet/lib/engines/*.so

[tosmi (Toni Schmidbauer)]

• File ldd_engines.txt ldd_engines.txt added

Naohisa Goto wrote:

The function "tls1_PRF" exists in libssl.so, and I still doubt if the OpenSSL is correctly installed.
Could you please provide more information about OpenSSL?

seems like i found the solution... the culprit seems to be the default optimization flags the sun studio compiler uses
when compiling openssl. it's compiling openssl with -O and according to the oracle documentation

http://docs.oracle.com/cd/E24457_01/html/E21990/bjapr.html#bjavm

this defaults to -xO3. setting the optimization to -xO2 results in a working openssl.

i still have to verify this with a more recent openssl version.

thanks
toni

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)

[tosmi (Toni Schmidbauer)]

Toni Schmidbauer wrote:

i still have to verify this with a more recent openssl version.

so i'm able to confirm the solution, this is a problem with compiling openssl and not ruby:

ruby 2.0.0p576 / openssl 1.0.1i -> works
ruby 2.1.3 / openssl 1.0.1i -> works

you have to replace -O with -xO2 for a working openssl library when comiling with sun studio 12.3.

thanks for your help and sorry for the noise

toni

[ngoto (Naohisa Goto)]

• Status changed from Feedback to Third Party's Issue

Thanks.

FYI, in my environment (SPARC Enterprise M5000), compiling OpenSSL 1.0.1i with "-xO5" and with "-m64" by using SolarisStudio 12.3 works fine.

[tosmi (Toni Schmidbauer)]

Naohisa Goto wrote:

FYI, in my environment (SPARC Enterprise M5000), compiling OpenSSL 1.0.1i with "-xO5" and with "-m64" by using SolarisStudio 12.3 works fine.

thats true. i just compiled openssl on a sparc system (T5-2) and it works with the default options.

so for any further reference: this is only related to compiling openssl on i386 with sun studio 12.3.

thanks
toni