Feature #16147: List Comprehensions in Ruby

• Status: Open
• Assignee: -
• Target version: -

[sammomichael (Samuel Michael)]

List comprehensions are present in many languages and programmers are quite fond of their simplicity and power. Add to that the fact that Ruby has a for...in loop that is rarely used but could possibly be repurposed.

Currently we can already do a hack like this to make Ruby support list comprehension syntax:

S = [for x in 0...9 do $* << x*2 if x.even? end, $*][1]
# [0, 4, 8, 12, 16]

Still, it would be far nicer if the for...in loop would return the desired array automatically, this is one way to approach that taking advantage of lambda bracket invocation syntax:

c = -> x do $*.clear             
  if x['if'] && x[0] != 'f' .  
    y = x[0...x.index('for')]    
    x = x[x.index('for')..-1]
    (x.insert(x.index(x.split[3]) + x.split[3].length, " do $* << #{y}")
    x.insert(x.length, "end; $*")
    eval(x)
    $*)
  elsif x['if'] && x[0] == 'f'
    (x.insert(x.index(x.split[3]) + x.split[3].length, " do $* << x")
    x.insert(x.length, "end; $*")
    eval(x)
    $*)
  elsif !x['if'] && x[0] != 'f'
    y = x[0...x.index('for')]
    x = x[x.index('for')..-1]
    (x.insert(x.index(x.split[3]) + x.split[3].length, " do $* << #{y}")
    x.insert(x.length, "end; $*")
    eval(x)
    $*)
  else
    eval(x.split[3]).to_a
  end
end 

so basically we are converting a string to proper ruby syntax for loop then we can use python syntax in a string to do:

c['for x in 1..10']
c['for x in 1..10 if x.even?']
c['x**2 for x in 1..10 if x.even?']
c['x**2 for x in 1..10']

# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# [2, 4, 6, 8, 10]
# [4, 16, 36, 64, 100]
# [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

[Dan0042 (Daniel DeLorme)]

What's wrong with existing well-established object-oriented ruby idioms?

(1..10).to_a
(1..10).select(&:even?)
(1..10).select(&:even?).map{ |x| x**2 }
(1..10).map{ |x| x**2 }

or in ruby 2.7:

(1..10).to_a
(1..10).select(&:even?)
(1..10).select(&:even?).map{ _0**2 }
(1..10).map{ _0**2 }

or if #16120 was accepted ;-D

(1..10).to_a
(1..10).select{.even?}
(1..10).select{.even?}.map{.**2}
(1..10).map{.**2}

[shevegen (Robert A. Heiler)]

I am not sure how serious the proposal is; but I will assume, for sake of simplicity, that
the suggestion is "real". First a minor comment - it should be filed under "features"
rather than "bug", since it is a suggestion for a change, rather than a bug, in my
opinion; but this is an aside.

Next, I will briefly comment on this statement:

Ruby has a for...in loop that is rarely used but could possibly be repurposed.

Let's for a moment briefly ignore whether people use the for loop or not (they
actually do; but let's ignore this for the moment).

Matz has said several times before that the transition from ruby 2.0 (2.x) to
3.0 will not be anywhere near as "problematic" as 1.8.x to 2.x was. So for
this reason alone, I believe that IF this were to be approved, it would have
to come way after 3.0, possibly 3.1 at earliest or even after that - so I
guess a few years from this point. (Ruby 3.0 will be released next year,
so we don't have that much time really; 15 months or so, give or take.)

You also wrote that the "for" loop is rarely used. Well, this is partially
true. I myself use almost exclusively .each and loop {}. But - I know of
other people who use "for" quite a bit, in particular when you have like
a matrix and iterate through it; in this case I can understand that a for
loop is used, even if I personally prefer .each and loop {}. So I am not
sure if your general comment is correct.

I assume that one reason why a for loop exists, and also other aliases,
such as .map <- -> .collect, and as to why ruby is multi-paradigm, was
that matz wanted to make ruby convenient to use for people with different
background. Ruby itself has incorporated useful ideas and paradigms from
other languages too. (My personal opinion is that, by far, the strongest
point of ruby, in regards to philosophies, is its OOP-centric nature, but
you may disagree. People use ruby in different ways, though. Some of the
ruby code looks quite alien; and matz has said sometimes, in the past,
that he was sometimes surprised to see how people use ruby.)

The philosophy of "more than one way to do it" also means that the "lesser
ways", that is, used more sporadically, will not necessarily be removed
merely because most ruby users may not use it. There are many other examples
here, such as @@class_variables. I don't use the latter myself, and would
rather see them go, but there are others who use class variables just fine,
even in the ruby code base (the recent rewrite of irb for example has some
class variables). So I think this is not a good metric in regards to
"adoption". But this is also not the main issue here.

The next comment to make .... hmm. List comprehensions remind me of python.

I use both ruby and python fine, but python feels a bit strange, in many
ways. Including python's OOP way (I absolutely hate explicit self by far
the most; mandatory indent is so minor compared to having to pass self
to every function/method in a class). IMO, ruby's way to filter is a
LOT more natural and easier to both understand and read than are python's
list comprehensions.

Granted, you were not confining your suggestion to python alone, so there
may be other list comprehensions that are succinct and elegant. I don't
really see them in the proposal per se, but perhaps they exist; I don't
know. I think that ruby's existing ways are very succinct, though.
Ultimately I don't think ruby needs list comprehensions, most definitely
not the pythonic list comprehensions.

You gave an example as-is in ruby:

S = [for x in 0...9 do $* << x*2 if x.even? end, $*][1] # [0, 4, 8, 12, 16]

Off the top of my mind, this would be better, in my opinion:

(0..16).select {|entry| entry % 4 == 0} => [0, 4, 8, 12, 16]

I am sure there are many other ways (Dan0042 provided more examples),
but IMO, the latter is so much more readable than the first variant,
so I don't think there is any real improvement.

You gave more examples such as:

c = -> x do $*.clear             
  if x['if'] && x[0] != 'f' . 

I don't really know what this is. It almost looks as if there was an
attempt made to make this less readable.

Anyway you only have to convince matz, but I am not sure if the
proposal as it presently is has a good chance for implementation;
there seem to be too many trade offs associated with it in its
current form. Or perhaps there is some additional reasoning or
advantage missing; I am not quite seeing the improvement or the
stated need, to be honest.

Best luck to you nonetheless.

[sammomichael (Samuel Michael)]

• Tracker changed from Bug to Feature
• Backport deleted (2.5: UNKNOWN, 2.6: UNKNOWN)

[sammomichael (Samuel Michael)]

There is nothing wrong with the current ruby syntax of using enumerables such as map/select/filter_map to generate the same type of list. But as you said ruby is all about many ways of doing things. And as many languages offer some version of this feature (Julia, Python, CoffeeScript, etc.) it could be beneficial for ruby to incorporate its own variation. There is something natural and refreshing about using pure mathematical set builder notation with keywords for...in to generate an array. [*1..10].filter_map{@1**2 if @1.even?} vs [x**2 for x in 1..10 if x.even?] The second version has less symbols, less keystrokes and is more human readable for beginners imo. Also I didn't mean to suggest that we retire the normal for...in loop (which I use myself) to break older code or that nobody ever uses it, but I am suggesting that we extend it if possible, just as was done with making "in" the keyword for the new pattern matching system.

[sammomichael (Samuel Michael)]

sammomichael (Samuel Michael) wrote:

shevegen (Robert A. Heiler) wrote:

I am not sure how serious the proposal is; but I will assume, for sake of simplicity, that
the suggestion is "real". First a minor comment - it should be filed under "features"
rather than "bug", since it is a suggestion for a change, rather than a bug, in my
opinion; but this is an aside.

Yes it is real, and yes, I should have changed it to be a feature and not a bug

Next, I will briefly comment on this statement:

Ruby has a for...in loop that is rarely used but could possibly be repurposed.

Let's for a moment briefly ignore whether people use the for loop or not (they
actually do; but let's ignore this for the moment).

I use Ruby's for loop too, that is why I said "rarely" rather than never or that it was fully deprecated. It has to be admitted it is less common in Ruby vernacular than in other languages, since we will use each or other enumerables for most cases. Also I never intended that we break compatibility with earlier versions or that we wouldn't still be able to use for loop, rather to extend those keywords for an additional use of generating a list comprehension to give an alternate optional syntax to doing what we currently can do with grep or filter_map.

Matz has said several times before that the transition from ruby 2.0 (2.x) to
3.0 will not be anywhere near as "problematic" as 1.8.x to 2.x was. So for
this reason alone, I believe that IF this were to be approved, it would have
to come way after 3.0, possibly 3.1 at earliest or even after that - so I
guess a few years from this point. (Ruby 3.0 will be released next year,
so we don't have that much time really; 15 months or so, give or take.)

Obviously filter_mapping was considered important enough to add a enumerable specifically for that, I am suggesting we offering an additional and alternate notation using [for..in] inside of brackets, obviously the final syntax would depend on the parser and how it would be easiest to modify without breaking compatibility with previous versions. A few years is fine by me, IF it were to be approved, that is a reasonable trajectory...

I assume that one reason why a for loop exists, and also other aliases,
such as .map <- -> .collect, and as to why ruby is multi-paradigm, was
that matz wanted to make ruby convenient to use for people with different
background. Ruby itself has incorporated useful ideas and paradigms from
other languages too.

My point exactly! Offering an alternate notation to (1..10).filter_map{|x|x**2 if x>5} such as [for x**2 in 1..10 if x > 5] or [x**2 for x in 1..10 if x < 5] would only serve to benefit the language and invite more people into the tent. To me they are both readable but to beginners from a math background or coming from other languages with list comprehensions the latter options would be more familiar.

The philosophy of "more than one way to do it" also means that the "lesser
ways", that is, used more sporadically, will not necessarily be removed
merely because most ruby users may not use it. There are many other examples
here, such as @@class_variables. I don't use the latter myself, and would
rather see them go, but there are others who use class variables just fine,
even in the ruby code base (the recent rewrite of irb for example has some
class variables). So I think this is not a good metric in regards to
"adoption". But this is also not the main issue here.

The next comment to make .... hmm. List comprehensions remind me of python.

I use both ruby and python fine, but python feels a bit strange, in many
ways. Including python's OOP way (I absolutely hate explicit self by far
the most; mandatory indent is so minor compared to having to pass self
to every function/method in a class). IMO, ruby's way to filter is a
LOT more natural and easier to both understand and read than are python's
list comprehensions.

I think that depends on your background and disposition and I don't wish to argue which is better nor suggest we imitate python style syntax exactly, just it is impossible to deny that it is a super-popular and much-lauded feature of Python and other languages and there is obviously a reason for its popularity. IMHO if adapted into Ruby it could be a very popular addition.

Granted, you were not confining your suggestion to python alone, so there
may be other list comprehensions that are succinct and elegant. I don't
really see them in the proposal per se, but perhaps they exist; I don't
know. I think that ruby's existing ways are very succinct, though.
Ultimately I don't think ruby needs list comprehensions, most definitely
not the pythonic list comprehensions.

You gave an example as-is in ruby:

S = [for x in 0...9 do $* << x*2 if x.even? end, $*][1] # [0, 4, 8, 12, 16]

Off the top of my mind, this would be better, in my opinion:

(0..16).select {|entry| entry % 4 == 0} => [0, 4, 8, 12, 16]

Your example doesn't really do the same thing.

(0..9).filter_map{|x|x*2 if x.even?} would be a better comparison.

And S = [for x in 0...9 do $* << x*2 if x.even? end, $*][1] # [0, 4, 8, 12, 16] is admittedly a hacky way to get it done and I stated that it was not as clean as I would like, thus the motivation for making the feature request. I would like to be able to have this generate an array automatically without storing the elements manually:
[for x*2 in 0..9 if x.even?] . . or even omit the "in" like this is very succinct and clean IMO [x*2 for 0..9 if x.even?]

I am sure there are many other ways (Dan0042 provided more examples),
but IMO, the latter is so much more readable than the first variant,
so I don't think there is any real improvement.

You gave more examples such as:

c = -> x do $*.clear             
  if x['if'] && x[0] != 'f' . 

I don't really know what this is. It almost looks as if there was an
attempt made to make this less readable.

This is the lambda used to evaluate a list comprehension string and return an array in the examples below it. I grant it is a little headache-inducing, but it is just a workaround to demonstrate one approach since [] in ruby can also be used to invoke lambdas.

Anyway you only have to convince matz, but I am not sure if the
proposal as it presently is has a good chance for implementation;
there seem to be too many trade offs associated with it in its
current form. Or perhaps there is some additional reasoning or
advantage missing; I am not quite seeing the improvement or the
stated need, to be honest.

Best luck to you nonetheless.

Thank you, and it would be an honor to even have the opportunity to be denied by matz. I do think there is an advantage to an alternate syntax. As I said it is one of the most often touted features of Python and other languages because it is very clean and succinct way to generate a set of elements.

[[https://en.wikipedia.org/wiki/Comparison_of_programming_languages_(list_comprehension)]]

Dan0042 (Daniel DeLorme) wrote:

What's wrong with existing well-established object-oriented ruby idioms?

(1..10).to_a
(1..10).select(&:even?)
(1..10).select(&:even?).map{ |x| x**2 }
(1..10).map{ |x| x**2 }

Again nothing wrong with those examples, however ruby is multi-paradigm and can afford to offer another way to represent these IMO. In some cases one might prefer to use enumerable methods, in others more mathematical set notation might fit better. It would be up to you.

(1..10).select(&:even?) could also be done with [for x in 1..10 if x.even?]
(1..10).select(&:even?).map{ |x| x**2 } [for x**2 in 1..10 if x.even?]

or in ruby 2.7:

(1..10).to_a
(1..10).select(&:even?)
(1..10).select(&:even?).map{ _0**2 }

(1..10).filter_map{@1**2 if @1.even?} . vs [for x**2 in 1..10 if x.even?]

(1..10).map{ _0**2 }

or if #16120 was accepted ;-D

```ruby
(1..10).to_a
(1..10).select{.even?}
(1..10).select{.even?}.map{.**2}
(1..10).map{.**2}

[sammomichael (Samuel Michael)]

In reiteration of my main points, Ruby is a dynamic multi-paradigm language which should when possible embrace a variety of modes to achieve the same result.

SO what would be the benefits of list comprehension in Ruby and what would it look like?

1. Creating an array with only [] square brackets is fun and makes intuitive sense for beginners
2. "for .. in .." syntax is succinct, flexible, and has a good spoken rhythm/easy to verbalize
3. Easier for users of other languages to assimilate into Ruby as they don't need to know all our enumerable library right away.

Ruby already has functioning list comprehension syntax but it could be optimized:

[for x in 1..10 do x**2 if x%2==0 end] # this is Ruby but we can't capture the value
[for x in 1..10; x**2 if x.even? end] # since 2.7 Ruby it would also be better if it called Filter_Map under the hood instead of each

proposal to use some syntax to use a splat or additional indicator to allow a smoother shorthand for list comprehensions

[for x in 1..10 do x**2 if x.even? end] #=> 1..10  (normal Ruby)

below we propose a syntax in which we splat the for loop to return the stored result not the caller

[*for x in 1..10 do x**2 if x.even? end] #=> [4, 16, 36, 64, 100]

under the hood this could be syntactic sugar for

(1..10).filter_map{@1**2 if @1.even?}  #=> [4, 16, 36, 64, 100]

the key differences to notice. although there are less keystrokes with filter_map, you had to know the method name, and the exact arrangement of parentheses and curly braces, and whether to use a block or numbered parameter. If you decided to use pre-2.7 ruby you would have said:

(1..10).filter_map{|x|x**2 if x.even?}

vs with proposed list syntax:

[*for x in 1..10; x**2 if x.even? end]

I think both could and should be standard ruby syntax of achieving the same result of quickly generating a filter mapped array of the given elements.

[nobu (Nobuyoshi Nakada)]

sammomichael (Samuel Michael) wrote:

[for x in 1..10 do x**2 if x.even? end] #=> 1..10  (normal Ruby)

It is an Array from 0 to 10 now.

below we propose a syntax in which we splat the for loop to return the stored result not the caller

[*for x in 1..10 do x**2 if x.even? end] #=> [4, 16, 36, 64, 100]

Obviously the latter result conflicts with the former.
Which result do you expect from the following code?

a = for x in 1..10 do x**2 if x.even? end
[*a] #=> ????

[*(for x in 1..10 do x**2 if x.even? end)] #=> ????

[Eregon (Benoit Daloze)]

Just my opinion: I don't think we should have list comprehensions in Ruby.

They are just a small subset of the far more powerful Enumerable methods, and IMHO are often less readable (e.g., nested list comprehensions in Python are what I would call unreadable).
Also, I think they basically don't scale, they might look nice for small expressions, but as soon as it becomes bigger it's no longer possible to use the list comprehension syntax.
So I think it's better to use map/select/filter_map directly.

[sammomichael (Samuel Michael)]

nobu (Nobuyoshi Nakada) wrote:

sammomichael (Samuel Michael) wrote:

[for x in 1..10 do x**2 if x.even? end] #=> 1..10  (normal Ruby)

It is an Array from 0 to 10 now. Correct thanks I should have wrote [1..10]

below we propose a syntax in which we splat the for loop to return the stored result not the caller

[*for x in 1..10 do x**2 if x.even? end] #=> [4, 16, 36, 64, 100]

Obviously the latter result conflicts with the former.
Which result do you expect from the following code?

Hi, I was suggesting a splat operator as one way to create a special syntax, under the hood instead of calling each method it would treat this as a filter map, map, or filter operation, and return the result in place. My intention is to create a flexible alternative syntax to enumerable syntax or using external iteration to do the same thing. Of course it doesn't have to be a splat, just some way to indicate you are invoking the shorthand for Ruby list comprehension, not using Ruby traditional for (.each) loop. Splat could be interpreted here as spreading the values from the block into a new array.

a = for x in 1..10 do x**2 if x.even? end
[*a] #=> [4, 16, 36, 64, 100] a = 1..10 => [1,2,3,4,5,6,7,8,9,10]

for loop results in 1..10 and spreads out to an array
the new syntax will not work on ranges only can be declared within [] square brackets at the time of invocation

[*(for x in 1..10 do x**2 if x.even? end)] #=> [1,2,3,4,5,6,7,8,9,10]

here the expression in parenthesis would resolve to range 1..10 and be splatted to an array
I'm not sure if there is a reason to do this though, but I am open to other opinions if there is something better that should happen

[sammomichael (Samuel Michael)]

Eregon (Benoit Daloze) wrote:

Just my opinion: I don't think we should have list comprehensions in Ruby.

Thanks I appreciate the feedback!

They are just a small subset of the far more powerful Enumerable methods, and IMHO are often less readable (e.g., nested list comprehensions in Python are what I would call unreadable).
Also, I think they basically don't scale, they might look nice for small expressions, but as soon as it becomes bigger it's no longer possible to use the list comprehension syntax.
So I think it's better to use map/select/filter_map directly.

You have some very good points! Nested comprehensions can get pretty gnarly. Still, I have heard anecdotally from a lot of professional developers that Python list comprehensions are useful to them. Julia inventors though well enough of them to include them in their new language, and a bunch of other languages have some variation as well. I tried to point out some of the benefits in my list above, but basically:

1. a versatile tool, a third (syntax)option alongside of enumerable methods for array generation
2. flexible and easy for beginners to understand
3. square brackets make the array return explicit

[sammomichael (Samuel Michael)]

ruby_list_comprehension gem documentation

https://github.com/SammoMichael/Ruby_List_Comprehension

ruby_list_comprehension gem page

https://rubygems.org/gems/ruby_list_comprehension
Here is a gem to give a better idea what I mean.

Instructions:
gem install ruby_list_comprehension

require 'ruby_list_comprehension'

l - ListComprehension.new

l[for x in 1..10 do x end] (optional map/filter condition)

l['for x in 1..10 do x + 5 || x - 3 if x != 3 end'] #=> [6, 7, 9, 10, 11, 12, 13, 14, 15]

l['for x in Set.new([1,2,2,3,4,5]) do x.to_s if x != 2 end] #=> ["1", "3", "4", "5"]

if you would like to try out a prototype I would appreciate feedback

[sammomichael (Samuel Michael)]

I also have added rudimentary support for nested comprehensions.

Build Matrix from nested loop

p $l[[for x in 1..10 do end], for x in 2..20 do x end] #=> does (1..10).map{(2..20).map{|x|x}

Flatten Matrix with nested loop

p $l[for x in 1..10 do for x in 2..20 do x end end] #=> does (1..10).map{(2..20).map{|x|x}.flatten

[sammomichael (Samuel Michael)]

sammomichael (Samuel Michael) wrote:

I also have added rudimentary support for nested comprehensions.

Build Matrix from nested loop

$l[[for x in 1..10 do end], for x in 2..20 do x end] #=> does (1..10).map{(2..20).map{|x|x}
$l[[for j in 1..5 do end], for i in 1..5 do end] #=> (1..5).map{(1..5).map(&:itself)}

Flatten Matrix with nested loop

$l[for x in 1..10 do for x in 2..20 do x end end] #=> does (1..10).flat_map{(2..20).map{|x|x} $l[for x in 1..10 do for x in 2..20 do x+2 end end]` #=> [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]

[emptyflask (Jon Roberts)]

In my opinion, the more useful types of list comprehensions are when multiple inputs are needed.
The Ruby syntax in example one looks best, but requires more array manipulation, even if Enumerable#filter_map replaces select.map. Example four is the kind of method we should be comparing with list comps.

# Nice syntax, but slow:
def one(a,b,c)
  a.product(b, c)
    .select {|x, y, z| x < y && y > z }
    .map(&:sum)
end

# Combine select and map in Ruby 2.7
def two(a,b,c)
  a.product(b, c)
    .filter_map {|x, y, z| x+y+z if x < y && y > z }
end

# Ugly but more efficient, requires a mutable result array:
def three(a,b,c)
  res = []
  a.product(b, c) do |x, y, z|
    res << x+y+z if x < y && y > z
  end
  res
end

# About the same performance as three, slightly cleaner:
def four(a,b,c)
  [].tap do |res|
    a.product(b, c) do |x, y, z|
      res << x+y+z if x < y && y > z
    end
  end
end

#       user     system      total        real
#  1.566321   0.100999   1.667320 (  1.667341)
#  0.884384   0.016958   0.901342 (  0.901343)
#  0.708852   0.005994   0.714846 (  0.714862)
#  0.714095   0.000005   0.714100 (  0.714138)

-- List comprehension (Haskell syntax):
five a b c = [ x+y+z | x <- a,
                       y <- b,
                       z <- c,
                       x < y, y > z ]