https://redmine.ruby-lang.org/https://redmine.ruby-lang.org/favicon.ico?17113305112014-10-03T16:48:22ZRuby Issue Tracking SystemRuby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=491872014-10-03T16:48:22Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/49187/diff?detail_id=35531">diff</a>)</li></ul><p>So Wieso wrote:</p>
<blockquote>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">require</span> <span class="s1">'libfile'</span><span class="p">,</span> <span class="ss">into: :Lib</span> <span class="c1"># keyword-argument</span>
<span class="nb">require</span> <span class="s1">'libfile'</span> <span class="k">in</span> <span class="no">Lib</span> <span class="c1"># with keyword, also defining a module Lib at current binding (unless defined? Lib)</span>
<span class="n">require_qualified</span> <span class="s1">'libfile'</span><span class="p">,</span> <span class="ss">:Lib</span>
</code></pre>
</blockquote>
<p>Why the first and the last use a symbol?<br>
The second will be difficult as <code>require</code> is not a reserved word but a mere method call.</p>
<blockquote>
<ul>
<li>maybe also allow a binding as argument, not only a module?</li>
</ul>
</blockquote>
<p>Does it make sense?</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=491882014-10-03T16:48:39Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-2 priority-4 priority-default" href="/issues/5643">Feature #5643</a>: require/load options and binding option</i> added</li></ul> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=492242014-10-06T12:49:50Zsowieso (So Wieso)sowieso@dukun.de
<ul></ul><p>I chose the symbol <code>:Lib</code>, as I thought Ruby would complain if the constant <code>Lib</code> would not exist at this time. The keyword <code>in</code> would define it, if it would not exist. I would prefer if we could solve it without using symbols, but writing <code>module Lib; end</code> before the first require doesn't look nice.</p>
<p>Sorry, I didn't consider that <code>require</code> is a method, so I guess the keyword option (<code>in</code>) doesn't fit.<br>
( Alternatively we could define suffix <code>in</code> as enclosing the given module:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">require</span> <span class="s1">'file'</span> <span class="k">in</span> <span class="no">Lib</span>
<span class="c1"># is equivalent</span>
<span class="k">module</span> <span class="nn">Lib</span>
<span class="nb">require</span> <span class="s1">'file'</span>
<span class="k">end</span>
</code></pre>
<p>but then require has to check for its nesting.<br>
)</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=492272014-10-06T13:46:12Zmikegee (Michael Gee)michaelpgee@gmail.com
<ul></ul><p>So Wieso wrote:</p>
<blockquote>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">require</span> <span class="s1">'file'</span> <span class="k">in</span> <span class="no">Lib</span>
<span class="c1"># is equivalent</span>
<span class="k">module</span> <span class="nn">Lib</span>
<span class="nb">require</span> <span class="s1">'file'</span>
<span class="k">end</span>
</code></pre>
</blockquote>
<p>I don't like changing <code>require</code> with one argument to mean something else. It would break too much legacy code.</p>
<p>If you want to require a feature into your current namespace how about this:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">module</span> <span class="nn">Lib</span>
<span class="nb">require</span> <span class="s1">'lib'</span><span class="p">,</span> <span class="ss">in: </span><span class="nb">self</span>
<span class="k">end</span>
</code></pre> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=492302014-10-06T14:59:37Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>So Wieso wrote:</p>
<blockquote>
<p>I chose the symbol <code>:Lib</code>, as I thought Ruby would complain if the constant <code>Lib</code> would not exist at this time. The keyword <code>in</code> would define it, if it would not exist. I would prefer if we could solve it without using symbols, but writing <code>module Lib; end</code> before the first require doesn't look nice.</p>
</blockquote>
<p>It's ambiguous if <code>Lib</code> is a module or a class, when only the name is provided.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=492462014-10-07T12:27:26Zsowieso (So Wieso)sowieso@dukun.de
<ul></ul><p>Michael Gee wrote:</p>
<blockquote>
<p>I don't like changing <code>require</code> with one argument to mean something else. It would break too much legacy code.</p>
</blockquote>
<p>You are definitely right here, we should not do that.</p>
<p>Nobuyoshi Nakada wrote:</p>
<blockquote>
<p>It's ambiguous if Lib is a module or a class, when only the name is provided.</p>
</blockquote>
<p>Is it? If the constant is already defined, take it (class or module). If not create a new module by this name.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=492542014-10-07T13:50:43Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>So Wieso wrote:</p>
<blockquote>
<p>Nobuyoshi Nakada wrote:</p>
<blockquote>
<p>It's ambiguous if Lib is a module or a class, when only the name is provided.</p>
</blockquote>
<p>Is it? If the constant is already defined, take it (class or module). If not create a new module by this name.</p>
</blockquote>
<p>If you don't want to create a module by that name, you don't need to use the name.<br>
Should not introduce implicit conversion between a name and a module.<br>
You can use anonymous module too if is is a module object.</p>
<p>I think this feature should be an instance method of <code>Module</code>, similar to <code>load</code> rather than <code>require</code>.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=562092016-01-20T17:28:04Zjwmittag (Jörg W Mittag)Ruby-Lang@JoergWMittag.De
<ul></ul><p>Nobuyoshi Nakada wrote:</p>
<blockquote>
<p>I think this feature should be an instance method of <code>Module</code>, similar to <code>load</code> rather than <code>require</code>.</p>
</blockquote>
<p>Yes, I believe having <code>Module#load</code> and possibly <code>Module#require</code> and <code>Module#require_relative</code> would be the most logical:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="c1"># not namespaced:</span>
<span class="nb">require</span> <span class="s1">'foo'</span>
<span class="k">module</span> <span class="nn">Bar</span><span class="p">;</span><span class="k">end</span>
<span class="c1"># namespaced:</span>
<span class="k">module</span> <span class="nn">Bar</span>
<span class="nb">require</span> <span class="s1">'foo'</span>
<span class="k">end</span>
</code></pre>
<p>Re-using those names would be a backwards-incompatible change, though. I have seen people use the second form sometimes.</p>
<p>This is a replacement for the hard-to-use <code>wrap</code> optional argument to <code>Kernel#load</code>:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">load</span> <span class="s1">'foo'</span><span class="p">,</span> <span class="kp">true</span>
<span class="c1"># is almost equivalent to </span>
<span class="no">Module</span><span class="p">.</span><span class="nf">new</span><span class="p">.</span><span class="nf">load</span> <span class="s1">'foo'</span>
</code></pre>
<p>This can be naturally extended to <code>Binding#load</code>, <code>Binding#require</code>, and <code>Binding#require_relative</code>.</p>
<p>As with the <code>wrap</code> argument to <code>Kernel#load</code>, the question is, what should references like <code>::String</code> in the loaded script refer to? I think it would be nice if it offered <em>complete</em> isolation by default, with an option to revert to the current behavior of <code>load</code>, e.g. with an API like this:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Module</span>
<span class="k">def</span> <span class="nf">load</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="ss">pollute_global: </span><span class="kp">false</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=663182017-08-29T08:57:16Zhsbt (Hiroshi SHIBATA)hsbt@ruby-lang.org
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-2 priority-4 priority-default" href="/issues/13847">Feature #13847</a>: Gem activated problem for default gems</i> added</li></ul> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=924132021-06-10T05:51:55Zjaesharp (J Lynn)
<ul></ul><p>I'd like to note that there exists a gem called modules ( <a href="https://rubygems.org/gems/modules" class="external">https://rubygems.org/gems/modules</a> ) which uses <code>Kernel#load</code> with the <code>wrap=true</code> option in order to implement a module import/export resolution system similar in nature to the one described here and to node.js's commonjs module system semantics. Perhaps this is sufficient to meet people's needs, if brought in?</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=924142021-06-10T06:18:49Ztexpert (Aurel Branzeanu)
<ul></ul><p>jaesharp (J Lynn) wrote in <a href="#note-10">#note-10</a>:</p>
<blockquote>
<p>I'd like to note that there exists a gem called modules ( <a href="https://rubygems.org/gems/modules" class="external">https://rubygems.org/gems/modules</a> ) which uses <code>Kernel#load</code> with the <code>wrap=true</code> option in order to implement a module import/export resolution system similar in nature to the one described here and to node.js's commonjs module system semantics. Perhaps this is sufficient to meet people's needs, if brought in?</p>
</blockquote>
<p>And another similar gem is modulation ( <a href="https://rubygems.org/gems/modulation/versions/0.25" class="external">https://rubygems.org/gems/modulation/versions/0.25</a> )</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=924152021-06-10T08:15:37Zjaesharp (J Lynn)
<ul></ul><p>texpert (Aurel Branzeanu) wrote in <a href="#note-11">#note-11</a>:</p>
<blockquote>
<p>And another similar gem is modulation ( <a href="https://rubygems.org/gems/modulation/versions/0.25" class="external">https://rubygems.org/gems/modulation/versions/0.25</a> )</p>
</blockquote>
<p>It looks much more developed and actively maintained! Thank you! :)</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=991412022-09-15T03:19:23Zshioyama (Chris Salzberg)
<ul></ul><p>The <code>wrap</code> option to <code>load</code> has recently been expanded to allow passing a module instead of a boolean (<a href="https://bugs.ruby-lang.org/issues/6210" class="external">https://bugs.ruby-lang.org/issues/6210</a>) (thanks <a class="user active user-mention" href="https://redmine.ruby-lang.org/users/1604">@jeremyevans0 (Jeremy Evans)</a>)! This opens the door to new approaches to this problem.</p>
<p>I think there is a huge opportunity to improve Ruby, and particularly to improve its "namespace hygiene", in the way JS module systems work in Javascript.</p>
<p>Unlike others I've seen implemented in Ruby (including the <a href="https://github.com/lambdabaa/modules" class="external">modules</a> gem, mentioned above), I'd like this to work not only for code that explicitly enables it (with <code>export</code>, etc.) but also for code that does not.</p>
<p><strong>I want to make module imports work for any gem, without any changes.</strong> This may not be as hard as it seems.</p>
<p>I've created a gem called <a href="https://github.com/shioyama/im" class="external">Im</a>, which depends on a few patches in <a href="https://github.com/shioyama/ruby/tree/import_modules" class="external">this Ruby branch</a> (<a href="https://github.com/shioyama/ruby/commit/c4387043437b0851306ef199f9417be609c98827" class="external">c43870</a> is the main one, plus the bug fix in <a href="https://bugs.ruby-lang.org/issues/18960" class="external">https://bugs.ruby-lang.org/issues/18960</a>). My goal here is to load <em>existing gems</em>, including some of the ones we use often. I want to focus on something concrete which would potentially immediately impact how gems use the global namespace.</p>
<p>With this gem and the patches provided, you can e.g. load <code>activemodel</code> and use it in this way:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">require</span> <span class="s2">"im"</span>
<span class="kp">extend</span> <span class="no">Im</span> <span class="c1"># adds `import` method</span>
<span class="n">mod</span> <span class="o">=</span> <span class="n">import</span> <span class="s2">"activemodel"</span>
</code></pre>
<p>At this point, <code>ActiveModel</code> has been loaded but it is not in the global namespace, and can only be found in <code>mod</code>:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">ActiveModel</span>
<span class="c1">#=> uninitialized constant ActiveModel (NameError)</span>
<span class="n">mod</span><span class="o">::</span><span class="no">ActiveModel</span>
<span class="c1">#=> ActiveModel</span>
</code></pre>
<p>What is very interesting to me is that, <em>using Ruby's built-in temporary/permanent naming functionality</em>, we can alias either all of the gem or parts of it.</p>
<p>So for example, naming the module immediately names everything inside of it:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">MyRails</span> <span class="o">=</span> <span class="n">mod</span>
</code></pre>
<p>Now we have <code>MyRails::ActiveModel</code> (but no <code>::ActiveModel</code>).</p>
<p>You can confirm with the example in the gem's readme that this namespaced ActiveModel works (at least for simple stuff, haven't tested more thoroughly yet).</p>
<p>There are a few things I do to make this work:</p>
<ol>
<li>In the Ruby patch, I make the <code>wrap</code> argument to <code>load</code> apply to further <code>require</code>s inside the loaded script. This is actually as simple as removing (actually commenting out) a line.</li>
<li>Apply the <code>top_wrapper</code> created in <code>load</code> to native extensions (specifically, <code>rb_define_class</code>, <code>rb_define_module</code> & <code>rb_define_global_const</code>)</li>
<li>Make named modules/classes loaded in a wrapped script via <code>load</code> return their name minus the namespace as their temporary name when called with <code>name</code>. This is necessary to use gems like Rails which use <code>name</code> to load files, etc.</li>
<li>Resolve top-level references (<code>::Foo</code>) when loaded with <code>wrap</code> to the top of the module namespace, rather than the "absolute" top. For now I've done this in the gem using <code>const_missing</code>, but I intend on moving this to the Ruby patch.</li>
</ol>
<p>With the changes above, the gem is able to setup a registry, patch <code>Kernel#require</code>, <code>Modul#autoload</code> etc and make this all work (sort of). This is very similar to how Zeitwerk works.</p>
<p>To make this all work, I use a lot of aliasing constants from one module namespace to another. This actually seems to work pretty well! So for example, code like this (similar to ActiveSupport):</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">NilClass</span>
<span class="k">def</span> <span class="nf">to_s</span>
<span class="c1"># ...</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>would define a <em>new</em> class under the module namespace. So I alias <code>NilClass</code> to <code>mod::NilClass</code> (same for other top-level, predefined constants), which then makes this work. This also works <em>between</em> imports, so if you import one thing, and import another, shared dependencies are aliased between the import modules.</p>
<p>What is amazing to me is <em>how little is required to make this work</em>.</p>
<p>Obviously this is very much a proof of concept at this point (and I want to emphasize that). There are issues I am aware of working on fixing (e.g. try loading <code>ActiveRecord::Base</code>, it will fail currently). But if you look at the changes to Ruby, they only impact code that uses the <code>wrap</code> option to <code>load</code>, and there are only a few of them. Also, to me at least, they feel quite natural, since they are simply extending a concept that already exists (<code>top_wrapper</code>) to other places where it is not yet applied.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=991612022-09-16T06:52:59ZEregon (Benoit Daloze)
<ul></ul><p>This NilClass definition, even if reassigning global ::NilClass doesn't have any effect on nil though, isn't it?<br>
Or do you actually define mod::NilClass = NilClass before loading the ActiveSupport files?</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=991622022-09-16T07:34:35Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul></ul><p>Pure-ruby codes could perhaps be loaded multiple times side-by-side, but the problem is a DLL.<br>
Loading ActiveRecord::Base won't work because when it tries to dynamic-link libpq.so or libmysqlclient.so or whatever, that can already be loaded under another namespace; then fails.<br>
This is not only about rb_define_method etc. We cannot control how DLLs are loaded by the operating system.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=991652022-09-16T10:52:04Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>It is declared that <code>dlopen</code> will not fail in such cases.<br>
<a href="https://pubs.opengroup.org/onlinepubs/9699919799/functions/dlopen.html" class="external">https://pubs.opengroup.org/onlinepubs/9699919799/functions/dlopen.html</a></p>
<blockquote>
<p>Only a single copy of an executable object file shall be brought into the address space, even if <code>dlopen()</code> is invoked multiple times in reference to the executable object file, and even if different pathnames are used to reference the executable object file.</p>
</blockquote>
<hr>
<p><code>LoadLibrary</code> on Windows is similar.<br>
<a href="https://docs.microsoft.com/en-us/cpp/build/loadlibrary-and-afxloadlibrary?view=msvc-170" class="external">https://docs.microsoft.com/en-us/cpp/build/loadlibrary-and-afxloadlibrary?view=msvc-170</a></p>
<blockquote>
<p>If the call to <code>LoadLibrary</code> specifies a DLL module that is already mapped into the address space of the calling process, the function returns a handle of the DLL and increments the reference count of the module.</p>
</blockquote>
<hr>
<p>I couldn't find explicit descriptions for <code>load</code> on AIX.<br>
<a href="https://www.ibm.com/docs/en/aix/7.3?topic=l-load-loadandinit-subroutines" class="external">https://www.ibm.com/docs/en/aix/7.3?topic=l-load-loadandinit-subroutines</a></p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=991712022-09-16T21:45:01Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul></ul><blockquote>
<p>Loading ActiveRecord::Base won't work because when it tries to dynamic-link libpq.so or libmysqlclient.so or whatever, that can already be loaded under another namespace; then fails.</p>
</blockquote>
<p>An orthogonal question is whether loading two versions of Active Record should load two versions of all the dependencies of Active Record?</p>
<p>I have only quickly skimmed <code>Im</code>, so I may say innacurate things, but it seems to me that what would be desirable would be to load a "namespace" in isolation, but not necessarily its dependencies as well.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=991722022-09-17T01:53:03Zshioyama (Chris Salzberg)
<ul></ul><blockquote>
<p>I have only quickly skimmed Im, so I may say innacurate things, but it seems to me that what would be desirable would be to load a "namespace" in isolation, but not necessarily its dependencies as well.</p>
</blockquote>
<p><code>Im</code> uses a registry to track files that have been loaded by their filenames, using TracePoint to also track which classes/modules were created in the process. It also patches <code>require</code>, so that if somewhere else in another import, the same file is required, the constants can all be aliased, allowing them to "share" the same dependencies without actually loading any one file multiple times.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=991732022-09-17T02:00:59Zshioyama (Chris Salzberg)
<ul></ul><blockquote>
<p>This NilClass definition, even if reassigning global ::NilClass doesn't have any effect on nil though, isn't it?<br>
Or do you actually define mod::NilClass = NilClass before loading the ActiveSupport files?</p>
</blockquote>
<p>Yes, <code>mod::NilClass = NilClass</code> is assigned in the module before passing it to the first <code>load</code>, so when loading core extensions ActiveSupport sees <code>mod::NilClass</code> and this simply points to <code>::NilClass</code>.</p>
<p>You can confirm it works:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="kp">nil</span><span class="p">.</span><span class="nf">blank?</span>
<span class="c1"># undefined method `blank?' for nil:NilClass (NoMethodError)</span>
<span class="nb">require</span> <span class="s2">"im"</span>
<span class="kp">extend</span> <span class="no">Im</span>
<span class="n">mod</span> <span class="o">=</span> <span class="n">import</span> <span class="s2">"active_support"</span>
<span class="c1">#=> <#Im::Import root: active_support></span>
<span class="no">ActiveSupport</span>
<span class="c1"># `const_missing': uninitialized constant ActiveSupport (NameError)</span>
<span class="n">mod</span><span class="o">::</span><span class="no">ActiveSupport</span>
<span class="c1">#=> ActiveSupport</span>
<span class="kp">nil</span><span class="p">.</span><span class="nf">blank?</span>
<span class="c1">#=> true</span>
</code></pre> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=991742022-09-17T02:32:14Zshioyama (Chris Salzberg)
<ul></ul><p><a class="user active user-mention" href="https://redmine.ruby-lang.org/users/10">@shyouhei (Shyouhei Urabe)</a></p>
<blockquote>
<p>Pure-ruby codes could perhaps be loaded multiple times side-by-side</p>
</blockquote>
<p>Not multiple times, loaded only once in the namespace where <code>require</code> is called (wrap module), then when loaded again original constants are aliased to the new namespace (see my explanation above to Jean). I do this currently in the gem, but it could alternatively be done in Ruby itself.</p>
<blockquote>
<p>This is not only about rb_define_method etc. We cannot control how DLLs are loaded by the operating system.</p>
</blockquote>
<p>Thanks! This is the kind of thing I felt I must be missing.</p>
<p>So it seems there are a relatively small number of these. All of <code>PG</code> and whatever it defines at toplevel could be made global and aliased from namespaces, which I think would avoid that problem.</p>
<p>The reason why I focused on <code>rb_define_class</code> etc is that without that change libraries with native extensions like <code>CGI</code> get split, the Ruby part being under the isolated namespace and the other being toplevel. It's not a problem to have some stuff (like those using DLL) at toplevel, but having a library split between both would definitely be a problem.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=991952022-09-18T10:43:04Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul></ul><p>May I ask someone the problem this ticket is currently trying to address? I’m confused.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=992062022-09-19T08:25:22Zduerst (Martin Dürst)duerst@it.aoyama.ac.jp
<ul></ul><p>shyouhei (Shyouhei Urabe) wrote in <a href="#note-21">#note-21</a>:</p>
<blockquote>
<p>May I ask someone the problem this ticket is currently trying to address? I’m confused.</p>
</blockquote>
<p>Same question here.</p>
<p>My (I hope average Ruby progarmmer) understanding of how gems/modules/namespaces are supposed to work currently is as follows (overview):</p>
<ul>
<li>Each gem uses a (top) module, where the module name is (modulo some case/... changes) the same as the gem name.</li>
<li>Each gem puts its constants (which included classes and modules) into that gem's (top) module.</li>
<li>rubygems.org makes sure that each gem name can only be used for one gem, and thus module names used by different gems are different.</li>
</ul>
<p>The above is mostly a social contract on how to use modules in gems, but my understanding is that it is widely understood and followed. I'm sure there are exceptions, but I'd guess they happen mostly in toy "gems" written by beginners.</p>
<p>My guess at reasons for this proposal are the following (but I would of course like to know the real reasons):</p>
<ol>
<li>Namespaces are a precious resource, and the less we sit on it, the better, even if practically, there may not be much of a problem.</li>
<li>Some people are used to how modules work in JavaScript, and want the same in Ruby.</li>
<li>Different modules may require the same modules but e.g. with different versions. (I didn't find any discussion of versions above, however, and the discussion of "using the same module by aliasing when it's included more than once" seems to indicate that different versions are not relevant here. Also, versioning is mostly bundler's job.</li>
<li>Companies (but not only companies) using Ruby have internal libraries with modules. They don't want to squat on a gem name (doing so may reveal company internals including business plans), but they don't want to risk a future name conflict.</li>
</ol>
<p>The above are only guesses. There may by more than one reason. But I think we should make it/them as explicit as possible.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=992072022-09-19T08:32:29Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul></ul><p><a class="user active user-mention" href="https://redmine.ruby-lang.org/users/50">@duerst (Martin Dürst)</a> / <a class="user active user-mention" href="https://redmine.ruby-lang.org/users/10">@shyouhei (Shyouhei Urabe)</a></p>
<p>The main goal is to avoid accidental dependency. By not exposing some namespaces globally, you can force developers to have to declare what they depend on, which makes them realize they're about to depend on something they shouldn't.</p>
<p>You can see <a href="https://github.com/Shopify/packwerk" class="external">https://github.com/Shopify/packwerk</a> as prior art trying to enforce boundaries using static analysis.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=992092022-09-19T17:23:05Zvo.x (Vit Ondruch)v.ondruch@tiscali.cz
<ul></ul><p>I'd love to see if RubyGems/Bundler could stop vendoring packages:</p>
<p><a href="https://github.com/rubygems/rubygems/tree/master/lib/rubygems" class="external">https://github.com/rubygems/rubygems/tree/master/lib/rubygems</a><br>
<a href="https://github.com/rubygems/rubygems/tree/master/bundler/lib/bundler/vendor" class="external">https://github.com/rubygems/rubygems/tree/master/bundler/lib/bundler/vendor</a></p>
<p>After all, Ruby could ship multiple versions of some library if old version is needed for whatever reason.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=992112022-09-20T10:58:29Zretro (Josef Šimánek)
<ul></ul><p>vo.x (Vit Ondruch) wrote in <a href="#note-24">#note-24</a>:</p>
<blockquote>
<p>I'd love to see if RubyGems/Bundler could stop vendoring packages:</p>
<p><a href="https://github.com/rubygems/rubygems/tree/master/lib/rubygems" class="external">https://github.com/rubygems/rubygems/tree/master/lib/rubygems</a><br>
<a href="https://github.com/rubygems/rubygems/tree/master/bundler/lib/bundler/vendor" class="external">https://github.com/rubygems/rubygems/tree/master/bundler/lib/bundler/vendor</a></p>
<p>After all, Ruby could ship multiple versions of some library if old version is needed for whatever reason.</p>
</blockquote>
<p>I had initially same idea looking at <a href="https://bugs.ruby-lang.org/issues/6210" class="external">https://bugs.ruby-lang.org/issues/6210</a>, but it is not this easy actually (mostly for rubygems/bundler updates).</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=992232022-09-21T09:46:37Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>The proposed PR is simple, small, and appears to have few performance or compatibility issues. So I am basically positive about this proposal.</p>
<blockquote>
<p>The main goal is to avoid accidental dependency.</p>
</blockquote>
<p>I guess that the ultimate goal is to modularize the monolith to microservices, and that this proposal is for the intermediate stage (i.e., to modularize the monolith in a process). Am I right? It is not so obvious to me that this intermediate step would be useful, maybe because I don't have enough experience in monolith development :-)</p>
<blockquote>
<p>Resolve top-level references (::Foo) when loaded with wrap to the top of the module namespace, rather than the "absolute" top. For now I've done this in the gem using const_missing, but I intend on moving this to the Ruby patch.</p>
</blockquote>
<p>This approach looks not very robust. If there is a constant <code>Foo</code> defined in the top-level, I think it does not work.</p>
<pre><code># foo.rb
A = :foo
p ::A
</code></pre>
<pre><code># main.rb
A = :main
load "foo.rb", Module.new #=> expect: :foo, Im hack: :main
</code></pre>
<p>More dedicated support in the Ruby core side would be necessary, I think. But I am curious about how much the change brings performance degeneration.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=992242022-09-21T10:26:44Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul></ul><blockquote>
<p>I guess that the ultimate goal is to modularize the monolith to microservices, and that this proposal is for the intermediate stage (i.e., to modularize the monolith in a process). Am I right?</p>
</blockquote>
<p>No, we have absolutely no intention to go with microservices, quite the opposite. The goal is to modularize in process so that you can more easily enforce that certain areas are decoupled from others without having to deal with the headaches of network calls.</p>
<blockquote>
<p>This approach looks not very robust. If there is a constant Foo defined in the top-level, I think it does not work.</p>
</blockquote>
<p>Yeah, IMHO it uses too many fragile monkey patches and Tracepoint hooks to approximate the desired result. I think such a feature would need to be baked in Ruby itself with probably a keyword etc. But in the meantime "Im" is an interesting experimentation ground.</p>
<p>Also if such first class feature was to be designed, I think the Python import system would be a better model than NodeJS's <code>require()</code>.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=992282022-09-21T12:39:04Zshioyama (Chris Salzberg)
<ul></ul><p><a class="user active user-mention" href="https://redmine.ruby-lang.org/users/18">@mame (Yusuke Endoh)</a></p>
<blockquote>
<p>This approach looks not very robust. If there is a constant Foo defined in the top-level, I think it does not work.</p>
</blockquote>
<p>Yes absolutely, this is the problem with <code>const_missing</code>. So this would ultimately need something at the language level.</p>
<p>To be clear, my goal is that Ruby would implement the parts of this problem which are not implementable in gem code, and lets a gem like Im do the rest (similar to Zeitwerk's relationship to <code>autoload</code>).</p>
<p><a class="user active user-mention" href="https://redmine.ruby-lang.org/users/7941">@byroot (Jean Boussier)</a></p>
<blockquote>
<p>IMHO it uses too many fragile monkey patches and Tracepoint hooks to approximate the desired result. I think such a feature would need to be baked in Ruby itself with probably a keyword etc. But in the meantime "Im" is an interesting experimentation ground.</p>
</blockquote>
<p>Yes, I'm using whatever I can without changing Ruby too much to make things work. Also, it's of course a WIP.</p>
<p>But ultimately, my idea is for the gem to motivate the changes that are needed at the language level. I find the Tracepoint hook actually works ok (and Zeitwerk also uses Tracepoint), but the <code>const_missing</code> is definitely a temporary hack.</p>
<p><a class="user active user-mention" href="https://redmine.ruby-lang.org/users/10">@shyouhei (Shyouhei Urabe)</a></p>
<blockquote>
<p>May I ask someone the problem this ticket is currently trying to address? I’m confused.</p>
</blockquote>
<p>Matz talked about the problem ("better packages") in his <a href="https://youtu.be/Dp12a3KGNFw?t=2956" class="external">2021 Euruko talk</a>.</p>
<p>From my point of view, there are two major problems the proposal here would solve.</p>
<p>The first, namespace pollution, is the problem that gems can park themselves anywhere they like in the global namespace. Of course most gems are good citizens and limit themselves to a single module constant, but the fact that as a consumer of code (<em>any</em> code) you have to <em>trust</em> third parties not to pollute the shared namespace is, IMO, a huge problem.</p>
<p>The proposal here would make the consumer ("importer") of such code have full control over its position in the global namespace. As a user of a gem, you could name it whatever you want, or even pick and choose parts of the gem and put them wherever you want in your namespace (e.g. <code>AS = mod::ActiveSupport</code>). Of course this control is not <em>absolute</em>, since core top-level constants would be shared (so e.g. activesupport would still be able to patch shared classes) — so this is not entirely without side-effects — but nonetheless it would be a huge improvement over what we currently have.</p>
<p>Note this also would open the door to things Ruby has never been able to do, e.g. load two versions of the same gem in the same namespace. Useful? I don't know, but if it's not hard to do, I think Ruby should make it <em>possible</em>.</p>
<p>The second problem is the one that Jean mentioned. <a href="https://github.com/Shopify/packwerk" class="external">Packwerk</a> has become quite popular among companies with large Rails monoliths as a way to isolate parts of the application by "enforcing" boundaries. But Packwerk adds yet another level of organization ("packages") on top of many others we already have ("gems", "modules", in the case of Rails also "engines", etc.)</p>
<p>What is to me so special about the idea here is that it would allow you to isolate parts of an application, while also sharing code as necessary.</p>
<p>e.g. <code>Shop</code> is a class that is used in many places in the Shopify monolith. We can "share" that class in the shop component (<code>components/shop_identity</code>) with another component, <code>components/platform</code>, without sharing <em>all</em> shop-related code, like this:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">platform</span> <span class="o">=</span> <span class="n">import</span> <span class="s2">"components/platform"</span>
<span class="n">shop_identity</span> <span class="o">=</span> <span class="n">import</span> <span class="s2">"components/shop_identity"</span>
<span class="n">platform</span><span class="o">::</span><span class="no">Shop</span> <span class="o">=</span> <span class="n">shop_identity</span><span class="o">::</span><span class="no">Shop</span>
</code></pre>
<p>To me, this is a very elegant way of representing dependencies between components of an application. Rather than just give a part of the code <em>full</em> access to another part of the code (and vice versa), you just share whatever parts are needed, and not the rest.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=992292022-09-21T17:08:08Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>byroot (Jean Boussier) wrote in <a href="#note-27">#note-27</a>:</p>
<blockquote>
<p>No, we have absolutely no intention to go with microservices, quite the opposite. The goal is to modularize in process so that you can more easily enforce that certain areas are decoupled from others without having to deal with the headaches of network calls.</p>
</blockquote>
<p>Thank you for the explanation. So "in-process microservices" is the final goal. The concept is easy for me to understand. TBH, I am not sure if it is practically very useful (maybe because I don't have monolith experience. I am not against the proposal).</p>
<p>We will discuss this topic tomorrow at the dev meeting. If <a class="user active user-mention" href="https://redmine.ruby-lang.org/users/13">@matz (Yukihiro Matsumoto)</a> is positive about the idea, it would be good to have a separate ticket with a clear explanation of the motivation and all the core features needed to implement Im reasonably, i.e., without using fragile hacks like const_missing.</p>
<p>shioyama (Chris Salzberg) wrote in <a href="#note-28">#note-28</a>:</p>
<blockquote>
<p>To be clear, my goal is that Ruby would implement the parts of this problem which are not implementable in gem code, and lets a gem like Im do the rest (similar to Zeitwerk's relationship to <code>autoload</code>).</p>
</blockquote>
<p>(This is a side note.) I think this is an approach, not a goal. And TBH, I don't think this is the best approach. In my opinion, ideally, language extension-like features such as Zeitwerk and ActiveSupport::Concern should be provided in the core. We are in this situation because Ruby is so flexible to allow such language extensions to be implemented outside, and maybe because it is more lightweight to design and improve a external gem than a core feature, but it's a little disconcerting to say this as if it were an ideal situation.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=992302022-09-21T18:11:52Zjeremyevans0 (Jeremy Evans)merch-redmine@jeremyevans.net
<ul></ul><p>I think that trying to require into a module with code that was not designed for it will break things. One example is when using absolute constant references (those that start with <code>::</code>). Consider this code:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">A</span><span class="p">;</span> <span class="k">end</span>
<span class="k">class</span> <span class="nc">B</span> <span class="o"><</span> <span class="no">BasicObject</span>
<span class="no">C</span> <span class="o">=</span> <span class="o">::</span><span class="no">Object</span>
<span class="no">D</span> <span class="o">=</span> <span class="o">::</span><span class="no">A</span>
<span class="k">end</span>
</code></pre>
<p>This is an example of where you would generally use absolute constant references, because constant lookup in <code>BasicObject</code> will not look up constants in <code>Object</code>/top-level. However, any case where you are using absolute constant references should have this issue.</p>
<p>How would the code above work when loaded into a module? If absolute constant references are resolved through the module, the access to <code>::Object</code> breaks, since that is not defined in the module. If absolute constant references are not resolved through the module, the access to <code>::A</code> breaks, since it would no longer be defined at <code>Object</code>/top-level. Looking in the module first and then <code>Object</code>/top-level (or vice-versa) feels ad-hoc, and either approach has corner cases where it breaks.</p>
<p>It looks like <code>Im</code> attempts to handle the above case by copying global constants into the module. I doubt we would want to do that in <code>load</code> or <code>require</code>.</p>
<p>shioyama (Chris Salzberg) wrote in <a href="#note-19">#note-19</a>:</p>
<blockquote>
<blockquote>
<p>This NilClass definition, even if reassigning global ::NilClass doesn't have any effect on nil though, isn't it?<br>
Or do you actually define mod::NilClass = NilClass before loading the ActiveSupport files?</p>
</blockquote>
<p>Yes, <code>mod::NilClass = NilClass</code> is assigned in the module before passing it to the first <code>load</code>, so when loading core extensions ActiveSupport sees <code>mod::NilClass</code> and this simply points to <code>::NilClass</code>.</p>
<p>You can confirm it works:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="kp">nil</span><span class="p">.</span><span class="nf">blank?</span>
<span class="c1"># undefined method `blank?' for nil:NilClass (NoMethodError)</span>
<span class="nb">require</span> <span class="s2">"im"</span>
<span class="kp">extend</span> <span class="no">Im</span>
<span class="n">mod</span> <span class="o">=</span> <span class="n">import</span> <span class="s2">"active_support"</span>
<span class="c1">#=> <#Im::Import root: active_support></span>
<span class="no">ActiveSupport</span>
<span class="c1"># `const_missing': uninitialized constant ActiveSupport (NameError)</span>
<span class="n">mod</span><span class="o">::</span><span class="no">ActiveSupport</span>
<span class="c1">#=> ActiveSupport</span>
<span class="kp">nil</span><span class="p">.</span><span class="nf">blank?</span>
<span class="c1">#=> true</span>
</code></pre>
</blockquote>
<p>To me, this example is a perfect indication of why we shouldn't support this. This uses <code>import</code> to load <code>ActiveSupport</code>, so that <code>ActiveSupport</code> is not added to top level namespace, but all of the core extensions added by <code>ActiveSupport</code> are still active. The namespace isolation is only partial, it is not complete.</p>
<p>There is discussion about how this could allow multiple versions of the same gem versions to work. How would that work if the gem makes modifications to core classes, as <code>ActiveSupport</code> does? Let's say you are including/prepending a module in the class in both versions, overriding a method, and and calling <code>super</code> for default behavior. Seems like you would get the behavior for both versions, which is unlikely to be desirable. The situation is worse if a method aliasing approach is used, since running <code>alias orig_method method; def method; code; orig_method; end</code> twice would likely result in a method that causes <code>SystemStackError</code>.</p>
<p>I'm against <code>require</code> accepting a module similar to load, and against making the module wrapping behavior transitive, so that <code>require</code> and <code>load</code> automatically use the currently wrapping module.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=993222022-09-25T12:54:08Zshioyama (Chris Salzberg)
<ul></ul><p>I think there are some misunderstandings, which are at least partly my fault for not being clearer about my intentions here.</p>
<p>There are two things I presented earlier in this thread:</p>
<ul>
<li>changes to Ruby (<a href="https://github.com/shioyama/ruby/tree/import_modules" class="external">here</a>) to make the <code>load</code> wrap module transitive in <code>require</code>, including native extension code, and some other smaller changes around <code>name</code>.</li>
<li>a gem (<a href="https://github.com/shioyama/im" class="external">Im</a>) which takes advantage of those changes to implement imports</li>
</ul>
<p>I do not intend to propose here that everything that's in the gem be implemented in Ruby itself. My intention is to motivate changes at the language level necessary to open the door to gems like Im do implement imports in whatever way gem owners see fit.</p>
<blockquote>
<p>It looks like Im attempts to handle the above case by copying global constants into the module. I doubt we would want to do that in <code>load</code> or <code>require</code>.</p>
</blockquote>
<p>Agreed, I had always imagined that this was something that could be done at the gem level, or not at all, depending on needs.</p>
<blockquote>
<p>To me, this example is a perfect indication of why we shouldn't support this. This uses <code>import</code> to load <code>ActiveSupport</code>, so that <code>ActiveSupport</code> is not added to top level namespace, but all of the core extensions added by <code>ActiveSupport</code> are still active. The namespace isolation is only partial, it is not complete.</p>
</blockquote>
<p>The namespace isolation <em>as implemented in the gem</em> is only partial. The Ruby side should IMO be as close as possible to absolute isolation, so e.g. toplevel references in the wrapped module should resolve to the module top, not absolute top. This way the importer has full control over what constants to make available to the wrapped script.</p>
<p>But this of course poses a problem for basic classes that can be monkeypatched (<code>String</code>, <code>Hash</code>, etc) I don't know what the solution for that is but it seems to me the problem there is more with monkeypatching as a common practice than it is with imports (potentially) giving access to those classes across multiple imports.</p>
<blockquote>
<p>I'm against <code>require</code> accepting a module similar to <code>load</code>, and against making the module wrapping behavior transitive, so that <code>require</code> and <code>load</code> automatically use the currently wrapping module.</p>
</blockquote>
<p>To be clear, I'm also against the former.</p>
<p>The latter (transitivity of the wrap module to <code>require</code>) is a dealbreaker though because if we do not have transitivity, then existing code will never be importable without rewriting it. I am not suggesting it has to be <code>load</code>, but transitivity <em>somewhere</em> to <code>require</code> is going to be necessary in order for any kind of code importing to be practically useful.</p>
<p>The discussion of loading multiple versions of a gem is interesting, but I don't think it's terribly relevant to deciding whether Ruby should support transitivity of wrapping behaviour. What I implemented in the gem, with constant aliases, was one way of dealing with core extensions, and it works (sort of) for loading gems like Rails that monkey-patch core classes. Of course this would cause problems loading multiple versions of a gem, but I'm not suggesting Ruby ever should do this.</p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=993542022-09-27T00:54:19Zshioyama (Chris Salzberg)
<ul></ul><p>At the last Developer Meeting, it was suggested I create a new issue for the topic we've been discussing since <a href="https://bugs.ruby-lang.org/issues/10320#note-13" class="external">my comment above</a> since there are new developments, so I've done that:</p>
<p><a href="https://bugs.ruby-lang.org/issues/19024" class="external">https://bugs.ruby-lang.org/issues/19024</a></p> Ruby master - Feature #10320: require into modulehttps://redmine.ruby-lang.org/issues/10320?journal_id=993552022-09-27T01:27:31Zhsbt (Hiroshi SHIBATA)hsbt@ruby-lang.org
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-4 priority-default closed" href="/issues/19024">Feature #19024</a>: Proposal: Import Modules</i> added</li></ul>