https://redmine.ruby-lang.org/https://redmine.ruby-lang.org/favicon.ico?17113305112019-06-14T23:59:49ZRuby Issue Tracking SystemRuby master - Feature #15923: New independent string without memcpyhttps://redmine.ruby-lang.org/issues/15923?journal_id=785862019-06-14T23:59:49Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul></ul><p>puchuu (Andrew Aladjev) wrote:</p>
<blockquote>
<p>Is it possible to create independent string from source string without memcpy that will be freed automatically?</p>
</blockquote>
<p>In C there are several ways to free a memory region, depending how that string was allocated.<br>
"Every string must be able to be freed using free()" is simply a wrong assertion.</p>
<p>So no, there is no way for ruby to automatically free a memory allocated by others.<br>
C is not made that way.</p> Ruby master - Feature #15923: New independent string without memcpyhttps://redmine.ruby-lang.org/issues/15923?journal_id=786012019-06-15T13:42:53Zluke-gru (Luke Gruber)luke.gru@gmail.com
<ul></ul><p>I think what puchuu is asking is if he can pass a malloc'd string to a ruby function that will create a new string object that frees the given underlying buffer when the string object is destructed. Having read the code, I didn't come upon such a case but I imagine it's possible with a slight hack (untested by me, however):</p>
<pre><code class="C syntaxhl" data-language="C"><span class="n">VALUE</span> <span class="n">str</span> <span class="o">=</span> <span class="n">rb_str_new_static</span><span class="p">(</span><span class="n">buffer</span><span class="p">,</span> <span class="n">buflen</span><span class="p">);</span> <span class="cm">/* no malloc or memcpy done here, just ownership change of buffer */</span>
<span class="n">RUBY_FL_UNSET</span><span class="p">(</span><span class="n">str</span><span class="p">,</span> <span class="n">STR_NOFREE</span><span class="p">);</span> <span class="cm">/* STR_NOFREE isn't actually defined in internal.h unfortunately, it's currently same as FL_USER18, but could change. */</span>
</code></pre>
<p>Perhaps a new ruby string creation function would be useful? Something like <code>rb_str_new_take()</code>. Just a thought.</p>
<p>Of course the allocator used to allocate the buffer would have to be the same as Ruby's allocator or bad things will happen...</p> Ruby master - Feature #15923: New independent string without memcpyhttps://redmine.ruby-lang.org/issues/15923?journal_id=786142019-06-16T06:32:42Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p><code>ruby_xfree</code> != <code>free</code>.<br>
Using the former on malloc'ed buffer can cause a crash.</p> Ruby master - Feature #15923: New independent string without memcpyhttps://redmine.ruby-lang.org/issues/15923?journal_id=786212019-06-16T13:11:41Zluke-gru (Luke Gruber)luke.gru@gmail.com
<ul></ul><p>Thank you Nobu, I thought that might be the case but was unaware as I'm not familiar with the GC subsystem. Also I think shyouhei was saying the same thing, I was just too dense to understand the specifics of what he was saying :)</p>
<p>Having taken a cursory look, it seems ruby is adding some bookkeeping information at the start of every memory buffer allocated by <code>ruby_xmalloc</code> and family. It returns the memory after this bookkeeping information (the actual buffer size asked for), and when this buffer is given to <code>ruby_xfree</code>, ruby calculates the actual starting point by moving backwards 1 bookkeeping structure, then passes this to <code>free</code>.</p>
<p>So, you would have to allocate using <code>ruby_xmalloc</code> and friends anyway, in which case it seems useless to provide such a function like <code>rb_str_new_take</code>.</p> Ruby master - Feature #15923: New independent string without memcpyhttps://redmine.ruby-lang.org/issues/15923?journal_id=792412019-07-09T14:14:04Zalanwu (Alan Wu)
<ul></ul><p>Instead of working on a separate buffer then asking Ruby to take ownership, you could make changes to the buffer of a string:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="n">VALUE</span> <span class="n">new_string</span> <span class="o">=</span> <span class="n">rb_str_new</span><span class="p">(</span><span class="s">""</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="n">rb_str_resize</span><span class="p">(</span><span class="n">new_string</span><span class="p">,</span> <span class="n">size_you_want</span><span class="p">);</span>
<span class="n">do_work</span><span class="p">(</span><span class="n">RSTRING_PTR</span><span class="p">(</span><span class="n">new_string</span><span class="p">),</span> <span class="n">RSTRING_LEN</span><span class="p">(</span><span class="n">new_string</span><span class="p">));</span>
</code></pre>
<p>Would this be good enough?</p> Ruby master - Feature #15923: New independent string without memcpyhttps://redmine.ruby-lang.org/issues/15923?journal_id=792422019-07-09T14:22:49Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>It should be OK when passing the buffer from callers, but doesn't work with a library which returns a buffer allocated inside.</p>
<p>FYI: you can allocate the buffer by <code>rb_str_new(NULL, size_you_want)</code> at once.</p> Ruby master - Feature #15923: New independent string without memcpyhttps://redmine.ruby-lang.org/issues/15923?journal_id=797772019-07-21T19:21:29Zpuchuu (Andrew Aladjev)aladjev.andrew@gmail.com
<ul></ul><p>nobu (Nobuyoshi Nakada) wrote:</p>
<blockquote>
<p>It should be OK when passing the buffer from callers, but doesn't work with a library which returns a buffer allocated inside.</p>
<p>FYI: you can allocate the buffer by <code>rb_str_new(NULL, size_you_want)</code> at once.</p>
</blockquote>
<p>Thanks all, I see. Ruby has some kind of internal memory allocation mechanism and it is not recommended to use strings allocated outside.</p>
<p>Integration of <code>rb_str_resize</code> into buffer growth mechanism is a good but complex solution. I will keep string copy.</p> Ruby master - Feature #15923: New independent string without memcpyhttps://redmine.ruby-lang.org/issues/15923?journal_id=802592019-07-30T07:51:58Zko1 (Koichi Sasada)
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Rejected</i></li></ul><p>I didn't all comments, but it seems solved.<br>
Please reopen it if it is my mistake.</p> Ruby master - Feature #15923: New independent string without memcpyhttps://redmine.ruby-lang.org/issues/15923?journal_id=819892019-10-12T08:39:41Zpuchuu (Andrew Aladjev)aladjev.andrew@gmail.com
<ul></ul><p>I've implemented string bindings using growing ruby string. It was a bit tricky - I had to use <code>rb_protect</code>. I will leave here a link, so everyone can see an example. <a href="https://github.com/andrew-aladev/ruby-lzws/blob/master/ext/lzws_ext/string.c" class="external">https://github.com/andrew-aladev/ruby-lzws/blob/master/ext/lzws_ext/string.c</a></p>