https://redmine.ruby-lang.org/https://redmine.ruby-lang.org/favicon.ico?17113305112014-03-14T19:08:59ZRuby Issue Tracking SystemRuby master - Feature #9638: [PATCH] limit IDs to 32-bits on 64-bit systemshttps://redmine.ruby-lang.org/issues/9638?journal_id=457872014-03-14T19:08:59Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>sparse array is described in ruby-core:55079</p> Ruby master - Feature #9638: [PATCH] limit IDs to 32-bits on 64-bit systemshttps://redmine.ruby-lang.org/issues/9638?journal_id=468762014-05-26T05:38:59Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>I'm not sure if this is possible anymore due to SymbolGC<br>
No big deal, though.</p> Ruby master - Feature #9638: [PATCH] limit IDs to 32-bits on 64-bit systemshttps://redmine.ruby-lang.org/issues/9638?journal_id=468812014-05-27T02:45:27Zngoto (Naohisa Goto)ngotogenome@gmail.com
<ul></ul><p>I'm using machines that have 2TB or more main memory. I think the machines can treat more than 2**32 symbols and I want to use full 64-bit capacity.</p> Ruby master - Feature #9638: [PATCH] limit IDs to 32-bits on 64-bit systemshttps://redmine.ruby-lang.org/issues/9638?journal_id=468882014-05-27T03:22:03Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>I am OK with closing this issue (but I'm not sure if I have permissions<br>
to close on redmine).</p>
<p>However, your applications need more than 2**32 different symbols?<br>
That scares me :*(<br>
How much memory do your Ruby processes use?</p>
<p>The Symbol table currently takes at least (48 + 48 + 40 = 136) bytes per<br>
symbol on 64-bit, so 136 * (2 ** 32) is 544 gigabytes just for the<br>
symbol table (w/fstrings) in your app. That does not even account for<br>
memory of symbols with string representations longer than 23 bytes,<br>
nor the memory for hash table buckets.</p>
<p>I need to know because I am also looking into using khash[1] for the<br>
symbol table. By default, khash internal buckets/counters are all<br>
32-bits. We can tweak khash to use 64-bit counters if needed,<br>
but 2**32 symbols really should be enough.</p>
<p>The symbol table with khash might reduce memory overhead to ~90 bytes<br>
per-symbol on average, though...</p>
<p>[1] git clone <a href="https://github.com/attractivechaos/klib.git" class="external">https://github.com/attractivechaos/klib.git</a><br>
mruby also uses khash for (all?) its hash table needs.</p> Ruby master - Feature #9638: [PATCH] limit IDs to 32-bits on 64-bit systemshttps://redmine.ruby-lang.org/issues/9638?journal_id=469252014-05-28T01:02:34Znormalperson (Eric Wong)normalperson@yhbt.net
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Rejected</i></li></ul> Ruby master - Feature #9638: [PATCH] limit IDs to 32-bits on 64-bit systemshttps://redmine.ruby-lang.org/issues/9638?journal_id=471312014-06-10T08:30:52Zko1 (Koichi Sasada)
<ul></ul><p>(2014/03/15 4:07), <a href="mailto:normalperson@yhbt.net" class="email">normalperson@yhbt.net</a> wrote:</p>
<blockquote>
<p>Also limits symbol space to prevent OOM.</p>
</blockquote>
<p>What is OOM?<br>
Out of memory?</p>
<p>Symbol GC doesn't help?</p>
<p>--<br>
// SASADA Koichi at atdot dot net</p> Ruby master - Feature #9638: [PATCH] limit IDs to 32-bits on 64-bit systemshttps://redmine.ruby-lang.org/issues/9638?journal_id=471492014-06-10T18:50:21Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>SASADA Koichi <a href="mailto:ko1@atdot.net" class="email">ko1@atdot.net</a> wrote:</p>
<blockquote>
<p>(2014/03/15 4:07), <a href="mailto:normalperson@yhbt.net" class="email">normalperson@yhbt.net</a> wrote:</p>
<blockquote>
<p>Also limits symbol space to prevent OOM.</p>
</blockquote>
<p>What is OOM?<br>
Out of memory?</p>
</blockquote>
<p>Yes, out-of-memory.</p>
<blockquote>
<p>Symbol GC doesn't help?</p>
</blockquote>
<p>It does; but OOM was a secondary concern of mine.</p>
<p>I mainly wanted 32-bit ID so it might be easier to pack some structs<br>
on 64-bit machines. 64-bit ID is not a big issue, though.</p> Ruby master - Feature #9638: [PATCH] limit IDs to 32-bits on 64-bit systemshttps://redmine.ruby-lang.org/issues/9638?journal_id=536832015-08-06T13:12:31Zngoto (Naohisa Goto)ngotogenome@gmail.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-4 priority-default closed" href="/issues/11420">Feature #11420</a>: Introduce ID key table into MRI</i> added</li></ul>