Bug #20853
closedHash key retrieval after Process.warmup
Description
This was first reported as an issue against the LaunchDarkly SDK and Sidekiq SDKs as an issue in Ruby 3.3.1. I can still in Ruby 3.3.5.
Overview of behavior
The LaunchDarkly SDK maintains a hash of feature and segment information. This hash is indexed by 2 keys which are themselves hashes. They are defined as:
FEATURES = {
namespace: "features",
priority: 1,
get_dependency_keys: lambda { |flag| (flag[:prerequisites] || []).map { |p| p[:key] } },
}.freeze
SEGMENTS = {
namespace: "segments",
priority: 0,
}.freeze
When running the attached script, we often see an error indicating that the flag is not found. If you comment out the Process.warmup
line, this error will go away.
The require 'active_support/all'
directive is not strictly necessary. Including it causes the failure to more frequently occur, presumably due to the increase work in the warmup phase.
Hash key and access
When debugging the LaunchDarkly code, I noticed some odd behavior. Even though the hash seems populated, the FEATURES key could not index into it correctly.
Below is a bit of code, modified from the SDK, to show how we are trying to interact with the hash. The puts are added to clarify the issue.
def get(kind, key)
@lock.with_read_lock do
@items.keys.each do |k|
puts "##### Does #{kind[:namespace]} match the key #{k}? #{k == kind} #{k.eql?(kind)} #{k.hash == kind.hash}"
end
puts "###### Does @items include it? #{@items.include?(kind)}"
coll = @items[kind]
f = coll.nil? ? nil : coll[key.to_sym]
(f.nil? || f[:deleted]) ? nil : f
end
end
Running without Process.warmup
If we run the attached script without the Process.warmup
call, we will see that the keys align with the FEATURES key we provided. Additionally, the hash does see that the key exists, resulting in an evaluation result of true at the end.
That output follows.
##### Does features match the key {:namespace=>"segments", :priority=>0}? false false false
##### Does features match the key {:namespace=>"features", :priority=>1, :get_dependency_keys=>#<Proc:0x00007ee6174f9738 /home/mkeeler/code/launchdarkly/ruby-server-sdk.git/main/lib/ldclient-rb/in_memory_store.rb:17 (lambda)>}? true true true
###### Does @items include it? true
true
Running with Process.warmup
If you enable the Process.warmup
option, and run the script a few times, you will see output similar to the following.
Note that while the keys are ==
, eql?
, and their .hash
values are equal, the hash does not see that the FEATURES key exists.
##### Does features match the key {:namespace=>"segments", :priority=>0}? false false false
##### Does features match the key {:namespace=>"features", :priority=>1, :get_dependency_keys=>#<Proc:0x0000774f1cbfd730 /home/mkeeler/code/launchdarkly/ruby-server-sdk.git/main/lib/ldclient-rb/in_memory_store.rb:17 (lambda)>}? true true true
###### Does @items include it? false
I, [2024-10-30T14:34:33.842145 #974568] INFO -- : [LDClient] Unknown feature flag "sample-feature". Returning default value
false
Workarounds
We have worked around this issue by replacing these keys with classes. You can refer to the PR here for those details.
You can also resolve this by triggering a rehash
on the hash after the Process.warmup
has run.
Files
Updated by peterzhu2118 (Peter Zhu) 5 days ago
Thank you for reporting this bug.
I believe the issue here is that because Process.warmup
runs a GC compaction, it can cause objects to move. While this is usually not a problem, Proc#hash is calculated using pointer values. Since the pointer values can change, the hash value can change as well. For example:
p = proc {}
puts p.hash # => 2411193347801829553
GC.verify_compaction_references(expand_heap: true, toward: :empty)
puts p.hash # => 1198623829435689723
This causes the hash value for the Hash to change as well:
hash = {}
key = {:namespace=>"features", :priority=>1, :get_dependency_keys=>proc {} }
hash[key] = 1
puts hash.key?(key) # => true
GC.verify_compaction_references(expand_heap: true, toward: :empty)
puts hash.key?(key) # => false
I'll work on a fix for this problem.
Updated by peterzhu2118 (Peter Zhu) 5 days ago
- Backport changed from 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN to 3.1: UNKNOWN, 3.2: REQUIRED, 3.3: REQUIRED
Updated by peterzhu2118 (Peter Zhu) 5 days ago
I have a fix available here: https://github.com/ruby/ruby/pull/11966
Please let me know if this fixes your issues.
Updated by keelerm84 (Matthew Keeler) 5 days ago
Yes, that appears to have resolved the issue. Thank you for such a quick turnaround.
Updated by byroot (Jean Boussier) 5 days ago
- Backport changed from 3.1: UNKNOWN, 3.2: REQUIRED, 3.3: REQUIRED to 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED
Updated by peterzhu2118 (Peter Zhu) 3 days ago
- Status changed from Open to Closed
Applied in changeset git|29c480dd6fca993590c82078ba797e2c4e876ac7.
[Bug #20853] Fix Proc#hash to not change after compaction
The hash value of a Proc must remain constant after a compaction, otherwise
it may not work as the key in a hash table.
Updated by k0kubun (Takashi Kokubun) about 4 hours ago
- Backport changed from 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED to 3.1: WONTFIX, 3.2: REQUIRED, 3.3: DONE
ruby_3_3 6db39f4677ff50aacfe54bd9dda052c09e1c6ab0 merged revision(s) 29c480dd6fca993590c82078ba797e2c4e876ac7.