Bug #17354: Module#const_source_location is misleading for constants awaiting autoload

• Status: Open
• Priority: Normal
• Assignee: -
• Target version: -
• ruby -v: ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-darwin20]
• Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN

[tomstuart (Tom Stuart)]

Feature #10771 added Module#const_source_location as a way to find the source location of a constant’s definition. Bug #16764 reported that it didn’t work correctly for autoloaded constants, instead giving the source location of the autoload call site. This was fixed in v3_0_0_preview1 in 92730810 and backported to v2_7_2 in c65aae11.

However, #const_source_location still returns the autoload call site for constants which have not yet been loaded:

% echo 'class Foo; end' > foo.rb

% irb
>> Module.const_defined?(:Foo)
=> false
>> Module.const_source_location(:Foo)
=> nil

>> autoload :Foo, './foo'
=> nil

>> Module.const_defined?(:Foo)
=> true
>> Module.const_source_location(:Foo)
=> ["(irb)", 3]

>> Module.const_get(:Foo)
=> Foo

>> Module.const_defined?(:Foo)
=> true
>> Module.const_source_location(:Foo)
=> ["./foo.rb", 1]

This edge case is undocumented and surprising. It looks like a bug to the programmer who receives the autoload location instead of one of the documented return values of #const_source_location (nil, [], or the definition’s source location).

We could either:

• change the behaviour of #const_source_location to return [] for constants awaiting autoload, which is consistent with the return value of Module#const_defined? in this case (“if the constant is not present but there is an autoload for it, true is returned directly”), as well as the return value of #const_source_location for other constants whose source location is unknown (“if the constant is found, but its source location can not be extracted (constant is defined in C code), empty array is returned”); or
• document the current behaviour of #const_source_location to make it less surprising.

I recommend the first option — although the current behaviour was recently specified in source:spec/ruby/core/module/const_source_location_spec.rb@6d059674#L209, it doesn’t seem intentional — but if that’s not feasible, simply documenting this edge case would also be an improvement.

[mame (Yusuke Endoh)]

Do you think what is the purpose of Module#const_source_location? Unfortunately, the original motivation is not expressed in #10771. IMO, it is for debugging. I think, when we are trying to find the definition of a constant, it is actually useful to see the line number that calls autoload, instead of seeing an empty array.

[ufuk (Ufuk Kayserilioglu)]

mame (Yusuke Endoh) In my use-case, I would very much like Module#const_source_location to tell me where the constant is loaded or would be loaded from if it is autoloaded. I am doing runtime reflection to discover types in gems and their associated methods, constants, etc in Tapioca, and my biggest problem so far has been detecting which file a constant is originally loaded from. I'd been waiting for Module#const_source_location to give me that information, even if the constant has not been loaded yet. The autoload file location, theoretically, could point to a file in a different gem or even to bootsnap if it is in play, which stops this use-case from working properly.

I would very much like Module#const_source_location to work like this:

% echo 'class Foo; end' > foo.rb

% irb
>> Module.const_defined?(:Foo)
=> false
>> Module.const_source_location(:Foo)
=> nil

>> autoload :Foo, './foo'
=> nil

>> Module.const_defined?(:Foo)
=> true
>> Module.const_source_location(:Foo)
=> ["./foo.rb", nil]

>> Module.const_get(:Foo)
=> Foo

>> Module.const_defined?(:Foo)
=> true
>> Module.const_source_location(:Foo)
=> ["./foo.rb", 1]

since Ruby basically knows where Foo will be loaded from, but cannot tell us what line number it will be from without actually loading the file. Thus, I propose the line number be nil in that case.

Do you think that works?

[tomstuart (Tom Stuart)]

Do you think what is the purpose of Module#const_source_location? Unfortunately, the original motivation is not expressed in #10771. IMO, it is for debugging. I think, when we are trying to find the definition of a constant, it is actually useful to see the line number that calls autoload, instead of seeing an empty array.

Like ufuk (Ufuk Kayserilioglu), I’m trying to use #const_source_location to do runtime reflection in tooling, not for debugging, so the autoload call site is not useful.

[jeremyevans0 (Jeremy Evans)]

ufuk (Ufuk Kayserilioglu) wrote in #note-2:

mame (Yusuke Endoh) In my use-case, I would very much like Module#const_source_location to tell me where the constant is loaded or would be loaded from if it is autoloaded. I am doing runtime reflection to discover types in gems and their associated methods, constants, etc in Tapioca, and my biggest problem so far has been detecting which file a constant is originally loaded from. I'd been waiting for Module#const_source_location to give me that information, even if the constant has not been loaded yet. The autoload file location, theoretically, could point to a file in a different gem or even to bootsnap if it is in play, which stops this use-case from working properly.

I would very much like Module#const_source_location to work like this:

% echo 'class Foo; end' > foo.rb

% irb
>> Module.const_defined?(:Foo)
=> false
>> Module.const_source_location(:Foo)
=> nil

>> autoload :Foo, './foo'
=> nil

>> Module.const_defined?(:Foo)
=> true
>> Module.const_source_location(:Foo)
=> ["./foo.rb", nil]

>> Module.const_get(:Foo)
=> Foo

>> Module.const_defined?(:Foo)
=> true
>> Module.const_source_location(:Foo)
=> ["./foo.rb", 1]

since Ruby basically knows where Foo will be loaded from, but cannot tell us what line number it will be from without actually loading the file. Thus, I propose the line number be nil in that case.

This is not always true. Ruby does not know that the constant's location will be in that file. All it knows is that requiring that file should result in the constant being available after. The file mentioned may require or load another file with the constant definition. It may not even load it, you could end up with a NameError when referencing the constant.

If the location of the autoload definition is not useful, I think the main reasonable alternative would be [], as Ruby does not know where the constant will be defined. This is the same value used for constants defined in C.

[mame (Yusuke Endoh)]

ufuk (Ufuk Kayserilioglu) tomstuart (Tom Stuart) I'm unsure if I could understand your use case correctly, but maybe does Module#autoload? help you?

[mame (Yusuke Endoh)]

An example. You can ignore the result of const_source_location if autoload? returns non-nil.

$ irb
irb(main):001:0> autoload(:Foo, "./foo")
=> nil
irb(main):002:0> Module.autoload?(:Foo)
=> "./foo"
irb(main):003:0> Module.const_source_location(:Foo)
=> ["(irb)", 1]
irb(main):004:0> Foo
=> Foo
irb(main):005:0> Module.autoload?(:Foo)
=> nil
irb(main):006:0> Module.const_source_location(:Foo)
=> ["/home/mame/work/ruby/foo.rb", 1]

[tenderlovemaking (Aaron Patterson)]

jeremyevans0 (Jeremy Evans) wrote in #note-4:

This is not always true. Ruby does not know that the constant's location will be in that file. All it knows is that requiring that file should result in the constant being available after. The file mentioned may require or load another file with the constant definition. It may not even load it, you could end up with a NameError when referencing the constant.

If the location of the autoload definition is not useful, I think the main reasonable alternative would be [], as Ruby does not know where the constant will be defined. This is the same value used for constants defined in C.

I think asking for the const source location on something that hasn't been autoloaded yet should cause autoload to be triggered, then get the const source location. I don't think returning [] is reasonable because as you say, loading the constant could result in a NameError and the constant never being defined at all. When source location is [] we know it's for something that is defined, we just don't know where (it's implemented in C). In the autoload case it's something that is potentially defined, but we can't know unless the autoload is triggered.

It's weird that const_source_location would return a value for a constant that can never be defined:

irb(main):001:0> File.read "foo.rb"
=> "class Bar\nend\n"
irb(main):002:0> autoload(:Foo, "./foo.rb")
=> nil
irb(main):003:0> Module.const_source_location(:Foo)
=> ["(irb)", 2]
irb(main):004:0> Foo
Traceback (most recent call last):
        4: from /Users/aaron/.rubies/ruby-trunk/bin/irb:23:in `<main>'
        3: from /Users/aaron/.rubies/ruby-trunk/bin/irb:23:in `load'
        2: from /Users/aaron/.rubies/ruby-trunk/lib/ruby/gems/3.0.0/gems/irb-1.2.7/exe/irb:11:in `<top (required)>'
        1: from (irb):4
NameError (uninitialized constant Foo)
irb(main):005:0> Module.const_source_location(:Foo)
=> ["(irb)", 2]

[tenderlovemaking (Aaron Patterson)]

We can probably never change this behavior due to backwards compatibility, but along the same lines, I think this behavior is weird too:

irb(main):001:0> File.read "foo.rb"
=> "class Bar\nend\n"
irb(main):002:0> autoload(:Foo, "./foo.rb")
=> nil
irb(main):003:0> Object.const_defined?(:Foo)
=> true
irb(main):004:0> Foo
Traceback (most recent call last):
        4: from /Users/aaron/.rubies/ruby-trunk/bin/irb:23:in `<main>'
        3: from /Users/aaron/.rubies/ruby-trunk/bin/irb:23:in `load'
        2: from /Users/aaron/.rubies/ruby-trunk/lib/ruby/gems/3.0.0/gems/irb-1.2.7/exe/irb:11:in `<top (required)>'
        1: from (irb):4
NameError (uninitialized constant Foo)
irb(main):005:0> Object.const_defined?(:Foo)
=> false

Foo was never defined and could never be defined, yet const_defined? returned true. It seems like const_defined? should return :not_yet or something 😆

[ufuk (Ufuk Kayserilioglu)]

mame (Yusuke Endoh) wrote in #note-6:

An example. You can ignore the result of const_source_location if autoload? returns non-nil.

$ irb
irb(main):001:0> autoload(:Foo, "./foo")
=> nil
irb(main):002:0> Module.autoload?(:Foo)
=> "./foo"
irb(main):003:0> Module.const_source_location(:Foo)
=> ["(irb)", 1]
irb(main):004:0> Foo
=> Foo
irb(main):005:0> Module.autoload?(:Foo)
=> nil
irb(main):006:0> Module.const_source_location(:Foo)
=> ["/home/mame/work/ruby/foo.rb", 1]

mame (Yusuke Endoh) Thank you! That actually works for my use-case. In that case, I suggest that we leave the current implementation as is, and document the behaviour as tomstuart (Tom Stuart) had originally suggested.