Bug #20218: aset/masgn/op_asgn with keyword arguments

• Status: Closed
• Assignee: -
• Target version: -
• Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN

[jeremyevans0 (Jeremy Evans)]

I found that use of keyword arguments in multiple assignment is broken in 3.3 and master:

h = {a: 1}
o = []
def o.[]=(*args, **kw)
  replace([args, kw])
end

# This segfaults as RHS argument is not a hash
o[1, a: 1], _ = [1, 2]


# This passes the RHS argument as keywords to the method, treating keyword splat as positional argument
o[1, **h], _ = [{b: 3}, 2]
o
# => [[1, {:a=>1}], {:b=>3}]

Before 3.3, keyword arguments were treated as positional arguments.

This is similar to #19918, but for keyword arguments instead of block arguments.

@matz (Yukihiro Matsumoto) indicated he wanted to prohibit block arguments in aset/masgn and presumably also op_asgn (making them SyntaxErrors). Can we also prohibit keyword arguments in aset/masgn/op_asgn?

Note that aset treats keyword arguments as regular arguments:

o[1, a: 1] = 2
o
# => [[1, {:a=>1}, 2], {}]

o[1, **h] = {b: 3}
o
# => [[1, {:a=>2}, {:b=>3}], {}]

While op_asgn treats keyword arguments as keywords:

h = {a: 2}
o = []
def o.[](*args, **kw)
  concat([:[], args, kw])
  x = Object.new
  def x.+(v)
    [:x, v]
  end
  x
end
def o.[]=(*args, **kw)
  concat([:[]=, args, kw])
end
o[1, a: 1] += 2
o
# => [:[], [1], {:a=>1}, :[]=, [1, [:x, 2]], {:a=>1}]

o.clear
o[1, **h] += {b: 3}
o
# => [:[], [1], {:a=>2}, :[]=, [1, [:x, {:b=>3}]], {:a=>2}]

[matz (Yukihiro Matsumoto)]

OK, prohibit keyword arguments in aset.

Matz.

[kddnewton (Kevin Newton)]

Does this also include blocks? Sorry I can't remember if that was officially decided or not. Also I'm presuming this would be for Ruby 3.4?

[mame (Yusuke Endoh)]

Does this also include blocks?

Yes. @nobu (Nobuyoshi Nakada) wrote a patch in #19918, so a block will also be prohibited.

Note that keyword arguments and a block are allowed in the normal method call style: obj.[]=(1, 2, kw: 1, &blk).

[nobu (Nobuyoshi Nakada)]

https://github.com/nobu/ruby/tree/ary_set-prohibited-args

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|0d5b16599a4ad606619228623299b931c48b597b.

---

[Bug #20218] Reject keyword arguments in index

[bughit (bug hit)]

In this issue there's no consideration of compatibility or utility. This is a breaking change. The ability to pass kwargs to index methods has been in ruby for a long time, probably from the inception of kwargs and I have code that uses that. Ruby doesn't have a spec so MRI behavior is effectively the spec so you can't say using kwargs in index methods was somehow "wrong". It wasn't "wrong" empirically and it's not "wrong" conceptually. kwargs just allow for more variability in store/lookup operations via index methods, it allows you to control where/how something is stored/looked up.

this is from 2.6

module IndexTest
  @store = {}

  def self.store
    @store
  end

  def self.key(name, namespace: nil)
    name = "#{namespace}:#{name}" if namespace
    name
  end

  def self.[](name, namespace: nil)
    p [name, namespace]
    @store[key(name, namespace: namespace)]
  end

  def self.[]=(name, opts = {}, val)
    p [name, opts, val]
    @store[key(name, namespace: opts[:namespace])] = val
  end


end

IndexTest['foo'] = 1
p IndexTest['foo']
IndexTest['foo', namespace: 'bar'] = 2
p IndexTest['foo', namespace: 'bar']
p IndexTest.store

So kwargs in index methods should be preserved for the sake of compatibility and utility. The only reasonable breaking change here is for []= to have real kwargs, rather than the middle positional kwarg collector hash in the above example.

[bughit (bug hit)]

@matz (Yukihiro Matsumoto) Why is it necessary to introduce a breaking change here by removing useful, long-standing syntax? See example in the previous post.

[kddnewton (Kevin Newton)]

There were long-standing bugs with aset within masgn. Things like:

foo, bar[1, 2, qux: 1, &qaz], baz = *qoz

and in other expressions like:

begin
rescue => bar[1, 2, qux: 1, &qaz]
end

and:

for foo, bar[1, 2, qux: 1, &qaz], baz in qoz do end

I kind of thought these were the only expressions that were going to lose the ability to have keywords/blocks. I think maybe we need some clarification here, because:

foo[1, 2, bar: 1, &baz] = qux

always worked as far as I know.

[bughit (bug hit)]

The release notes, which is what caught my attention, are categorical:

Keyword arguments are no longer allowed in index. [Bug #20218]

[jeremyevans0 (Jeremy Evans)]

In Ruby 3.3, behavior is inconsistent:

a = Class.new do
  def [](*a, **kw, &b)
    p([a, kw, b])
    0
  end
  alias []= []
end.new
b = proc{}

# Regular assignment treats keywords as positional
a[1, 2, bar: 3, &b] = 4
# [[1, 2, {:bar=>3}, 4], {}, #<Proc:0x00000b17febec6e0 (irb):8>]

# Operator assignment respects keywords
a[1, 2, bar: 3, &b] += 4
# [[1, 2], {:bar=>3}, #<Proc:0x00000b17febec6e0 (irb):8>]
# [[1, 2, 4], {:bar=>3}, #<Proc:0x00000b17febec6e0 (irb):8>]

# Mass assignment crashes process
a[1, 2, bar: 3, &b], _ = 4, 5

Due to keyword argument separation, the regular assignment behavior for keywords should be considered incorrect. It could be changed to make it similar to operator assignment, but that would be worse in terms of backwards incompatibility, because it would silently change behavior. With the change to make it invalid syntax, at least the few Ruby users using the syntax know to update their code.

You can still call the [] and []= methods with keywords and a block, using send/public_send, in which case you can specify which arguments are positional and which are keywords.

[kddnewton (Kevin Newton)]

Thanks for the clarification @jeremyevans0 (Jeremy Evans). I think it's also worth noting you should be able to call them normally with a call operator, as in:

a.[]=(1, 2, 4, bar: 3, &b)

[bughit (bug hit)]

These are not arguments for removing a useful, long-standing syntactic feature, but for fixing the various edge cases with it.

Multiple assignment segfaults in 2.6 too. I didn't know about it but it did not prevent me from using the feature, but this "fix" sure will.

It could be changed to make it similar to operator assignment, but that would be worse in terms of backwards incompatibility

As someone who's is actually using this functionality, that's a change that would be acceptable to accommodate. kwargs separation is already there so you have to deal with the fallout in multiple contexts. So no it would not be generally "worse". The feature should remain, I already illustrated its utility.

[bughit (bug hit)]

@matz (Yukihiro Matsumoto) Why is this feature being removed instead of fixed?

[Eregon (Benoit Daloze)]

Jeremy's argument was not super clear to me so I took a deeper look.

Using a slight variant from the script in https://bugs.ruby-lang.org/issues/20218#note-10:

a = Class.new do
  def [](*a, **kw)
    p([a, kw])
    0
  end
  alias []= []
end.new

# Regular index
a[1, 2, bar: 3]

# Regular assignment
a[1, 2, bar: 3] = 4

# Operator assignment
a[1, 2, bar: 3] += 4

# Mass assignment
eval 'a[1, 2, bar: 3], _ = 4, 5' unless RUBY_VERSION.start_with?('3.3')

ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
[[1, 2], {:bar=>3}] # []
[[1, 2, {:bar=>3}, 4], {}] # []=
[[1, 2, {:bar=>3}], {}] # [] of []+=
[[1, 2, {:bar=>3}, 4], {}] # []= of []+=
[[1, 2, {:bar=>3}, 4], {}] # multiple-assignment []=

(same on `truffleruby 24.0.1` FWIW)

ruby 3.3.1 (2024-04-23 revision c56cd86388) [x86_64-linux]
[[1, 2], {:bar=>3}] # []
[[1, 2, {:bar=>3}, 4], {}] # []=
[[1, 2], {:bar=>3}] # [] of []+=
[[1, 2, 4], {:bar=>3}] # []= of []+=
SEGV # multiple-assignment []=

So in 3.2.2 the behavior was mostly consistent, kwargs in the various index methods were all treated as positional arguments.
Except for [] which does treat kwargs as kwargs.

In 3.3.1 []+= treats kwargs as kwargs for both the [] and []= calls (but not for a lone []= call), so that's already a "silent change" yet it makes a lot of sense, these things are kwargs in syntax, it feels natural they should be kwargs in semantics.

On ruby-master, the code in https://bugs.ruby-lang.org/issues/20218#note-6 does give a SyntaxError.
The problem is for IndexTest['foo', namespace: 'bar'] = 2
(IndexTest['foo', namespace: 'bar'] works fine).
A workaround is to use IndexTest.[]=('foo', { namespace: 'bar' }, 2) instead.

BTW let's note that code uses:

def self.[]=(name, opts = {}, val)

So this seems difficult to evolve, because if []= (and []+=) would pass kwargs as kwargs for the []= call, then that definition of []= would receive name='foo', opts=2, val={namespace: 'bar'}.
Which would also break because of e.g. opts[:namespace] with opts=2.

To fix this, the definition would need to become

def self.[]=(name, val, **opts)

but that would not work on existing Ruby releases then.

OTOH, that would already be the correct and necessary signature for the += case on 3.3.1:

module IndexTest
  @store = {}

  def self.store
    @store
  end

  def self.key(name, namespace: nil)
    name = "#{namespace}:#{name}" if namespace
    name
  end

  def self.[](name, namespace: nil)
    p [name, namespace]
    @store[key(name, namespace: namespace)] || 0
  end

  # def self.[]=(name, opts = {}, val) # no implicit conversion of Symbol into Integer (TypeError) for `opts[:namespace])` below
  def self.[]=(name, val, **opts) # works
    p [name, opts, val]
    @store[key(name, namespace: opts[:namespace])] = val
  end


end

IndexTest['foo', namespace: 'bar'] += 2

Given all this and the fact there is no simple way to define []= in a way that accepts caller kwargs as kwargs and yet doesn't break existing code, I can see the idea why to make this a SyntaxError, as it seems the only way to make []/[]= consistent and not break existing definitions of []=.
But it seems rather incompatible and not sure the consistency is worth it.
I don't really consider consistency with the block argument for []/[]=, because I think nobody ever uses that and something that probably should never have been supported given the lack of need for it.

An alternative seems to go back to 3.2-like (3.2 + *1) behavior, i.e. [] always receives kwargs, []= never receives kwargs (to not break existing definitions of []=).
The 3.3 behavior for the []= call as part of []+= seems surprising and hard to use (requires defining a []= accepting kwargs but that won't work for a direct []= call such as a[1, kw: 2] = 3.
(*1) OTOH the 3.3 change for the [] call part of []+= seems good, [] should receive kwargs.

[Eregon (Benoit Daloze)]

Eregon (Benoit Daloze) wrote in #note-14:

A workaround is to use IndexTest.[]=('foo', { namespace: 'bar' }, 2) instead.

Ah there is actually a much simpler change:

# original:
IndexTest['foo', namespace: 'bar'] = 2
# This works instead on ruby-master:
IndexTest['foo', { namespace: 'bar' }] = 2

I think if we could mention what to change in the SyntaxError it would be a big help.
Currently:

ruby 3.4.0dev (2024-05-30T12:13:10Z master 78bfde5d9f) [x86_64-linux]
index-test.rb: 
index-test.rb:28: keyword arg given in index (SyntaxError)
...dexTest['foo', namespace: 'bar'] = 2
...               ^~~~~~~~~~~~~~~~

[Eregon (Benoit Daloze)]

Aside: running https://bugs.ruby-lang.org/issues/20218#note-10 on ruby-master gives a pretty unreadable error:

$ ruby index-kwargs-org.rb 
index-kwargs-org.rb: --> index-kwargs-org.rb
unexpected keyword arg given in index; keywords are not allowed in index expressionsunexpected block arg given in index; blocks are not allowed in index expressions
...

[Eregon (Benoit Daloze)]

• Has duplicate Bug #20513: the feature of kwargs in index assignment has been removed without due consideration of utility, compatibility, consistency and logic added

[matz (Yukihiro Matsumoto)]

It is the direction of Ruby's evolution to separate keyword arguments from normal arguments, just as Ruby 3.0 promoted the separation of keyword arguments. This proposal goes against this direction and cannot be accepted. We investigated and found out there are some compatibility issues, but we consider the impact is minimal.

Matz.

[bughit (bug hit)]

matz (Yukihiro Matsumoto) wrote in #note-18:

It is the direction of Ruby's evolution to separate keyword arguments from normal arguments, just as Ruby 3.0 promoted the separation of keyword arguments. This proposal goes against this direction and cannot be accepted.

Which proposal? To actually fix this bug rather than remove the kwargs feature? How so?

Index assignment can be changed/fixed to have real, separated keyword arguments that would be 100% in line with this evolution of Ruby.

Instead of the following (from 2.6):

  def self.[]=(name, opts = {}, val)
    p [name, opts, val]
    @store[key(name, namespace: opts[:namespace])] = val
  end

you'd have:

  def self.[]=(name, val, namespace: nil)
    p [name, opts, val]
    @store[key(name, namespace: namespace)] = val
  end

The invocation syntax would remain the same: IndexTest['foo', namespace: 'bar'] = 2

So why is it preferable to remove the feature entirely from index assignment and have this incoherent inconsistency between [] (supports kwargs) and []= (doesn't support kwargs), rather than fix kwarg issues in []=?

[bughit (bug hit)]

@matz (Yukihiro Matsumoto) Your post argued against a strawman that preserving kwargs in []= would necessarily violate kwarg separation. Since that is obviously not true, []= can be given real, separated kwargs, instead of removing them entirely, the matter remains unsettled.

why is it preferable to remove the feature entirely from index assignment and have this incoherent inconsistency between [] (supports kwargs) and []= (doesn't support kwargs), rather than fix kwarg issues in []=?