Bug #6137: openssl: hardcoded MD5 use leads to SSL server failure in FIPS mode

• Status: Closed
• Assignee: MartinBosslet (Martin Bosslet)
• Target version: -
• ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [i386-linux]

[jared.jennings.ctr (Jared Jennings)]

=begin
I've got a host configured to be compliant with ((<U.S. Federal Information Processing Standard 140-2|URL:http://csrc.nist.gov/publications/fips/fips140-2/fips1402.pdf>)) (FIPS 140-2). On this host, the OpenSSL library refuses to do an MD5 checksum, because the MD5 algorithm is not FIPS Approved.

When I try to run Puppet's master subcommand, it sets up a secure HTTP server using WEBrick, which in turn uses the openssl module. But in the OpenSSL::SSL::SSLServer class, at source:ext/openssl/lib/openssl/ssl.rb@33695#L149, the MD5 digest is used to make a session ID from a context. On my host this fails as follows:

/usr/lib/ruby/1.8/openssl/digest.rb:55:in `initialize': Digest initialization failed.: unknown cipher (OpenSSL::Digest::DigestError)
    from /usr/lib/ruby/1.8/openssl/digest.rb:55:in `initialize'
    from /usr/lib/ruby/1.8/openssl/digest.rb:30:in `digest'
    from /usr/lib/ruby/1.8/openssl/digest.rb:30:in `digest'
    from /usr/lib/ruby/1.8/openssl/digest.rb:46:in `hexdigest'
    from /usr/lib/ruby/1.8/openssl/digest.rb:46:in `hexdigest'
    from /usr/lib/ruby/1.8/openssl/ssl-internal.rb:143:in `initialize'
    from /usr/lib/ruby/1.8/webrick/ssl.rb:94:in `new'
    from /usr/lib/ruby/1.8/webrick/ssl.rb:94:in `listen'
    from /usr/lib/ruby/1.8/webrick/ssl.rb:93:in `collect!'
    from /usr/lib/ruby/1.8/webrick/ssl.rb:93:in `listen'
    from /usr/lib/ruby/1.8/webrick/server.rb:63:in `initialize'
    from /usr/lib/ruby/1.8/webrick/httpserver.rb:24:in `initialize'
    from /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:33:in `new'
    from /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:33:in `listen'
    [...]

I'm not sure exactly how, but ext/openssl/lib/openssl/ssl.rb from the source tree appears to be installed as /usr/lib/ruby/1.8/openssl/ssl-internal.rb on the system.

I replaced the instantiation of OpenSSL::Digest::MD5 with OpenSSL::Digest::SHA256 on my own system. The puppet master command worked, and no other bad things happened. Accordingly I suggest this change for Ruby in general. - Reasons to make the change:

• Anyone trying to use OpenSSL::SSL::SSLServer who is in the U.S. government, a company contracting with the U.S. government, or possibly a bank, will appreciate if it works. (That's who cares about FIPS 140-2.)
• I haven't seen any migration issues.
• According to my reading of the code, any cryptographic hash will do.

Possible reasons not to make the change:

• SHA256 takes more time than MD5. I haven't checked how often the hash is called. Embedded servers that use OpenSSL::SSL::SSLServer may slow down.
• SHA256 hash values are longer than those of MD5. I don't think the hash values are stored in any variables with fixed size, but I haven't exhaustively confirmed it.

=end

[MartinBosslet (Martin Bosslet)]

• Status changed from Open to Assigned
• Assignee set to MartinBosslet (Martin Bosslet)

Thank you for your detailed analysis! Judging from OpenSSL's documentation [1] the parameter can potentially be any binary data and should not be critical for the overall security, so it might be possible to just use no hash at all. I'll try to find out what the original motivation for using a hash was.

One issue with mandating SHA-256 is that unfortunately there are also users that run Ruby on top of old versions of OpenSSL that do not yet support SHA-2 type hashes, so I'd prefer the hashless version if possible.

I could also find nothing in the codebase that would indicate migration issues in case we changed the value to something else.

[1] http://www.openssl.org/docs/ssl/SSL_CTX_set_session_id_context.html

[jared.jennings.ctr (Jared Jennings)]

SHA-1 would work too - for a few years, until it's cryptographically broken enough to no longer be FIPS approved, like MD5 is. But if you could do it without hashing, that would quash the problem forever.

Ah - here's why to hash it: http://www.openssl.org/docs/ssl/SSL_CTX_set_session_id_context.html says that if "the length sid_ctx_len of the session id context sid_ctx [exceeds] the maximum allowed length of SSL_MAX_SSL_SESSION_ID_LENGTH," an error results. According to my /usr/include/openssl/ssl.h, SSL_MAX_SSL_SESSION_ID_LENGTH is 32. The value on which the digest is based is $0, i.e. the name of the Ruby program being run. This would not often exceed 32 characters, but it easily could. The context has to be different between applications, so you probably can't just truncate $0 to 32 characters and have done. Ergo, a digest. It doesn't need to be an amazing digest; MD5 will do. Until now :/

[MartinBosslet (Martin Bosslet)]

Oh, thanks for the hint! Yes, now it makes sense why MD5 was used. I still think we don't need a crypto-strength hash just to simply bound a value by length, especially if it's not critical to security otherwise. We could use Object#hash for that instead, which could be turned into a binary string for this purpose and will not conflict with any FIPS restrictions.

I need to make sure, however, that the value will not enter any security-critical phase during the actual connection verification. I'll check how the value is used within OpenSSL.

• Status changed from Assigned to Closed
• % Done changed from 0 to 100

This issue was solved with changeset r36005.
Jared, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

• lib/openssl/ssl.rb: Use a simple random number to generate the
  session id. MD5, as was used before, causes problems when
  using a FIPS version of OpenSSL. Issue was found by Jared
  Jennings, thank you!
  [ruby-trunk - Bug #6137]

[vo.x (Vit Ondruch)]

Since this was originally reported against 1.8.7, could you please backport it? Or is 1.8.7 maintenance already over? Thank you.

[jared.jennings.ctr (Jared Jennings)]

Patch for 1.8.7 is at https://bugzilla.redhat.com/show_bug.cgi?id=802946#c4 (this won't be news to you, Vit, but any other readers of this issue may not know about https://bugzilla.redhat.com/show_bug.cgi?id=802946)