Feature #11817: map.parallel

• Status: Open
• Assignee: -
• Target version: -

[eike.rb (Eike Dierks)]

I'd like to suggest a new method: #parallel

That method should take care of executing workloads in parallel across multiple cores.

I believe that #map can sometimes be distributed to many simultaneous threads.

So we would spread out the map blocks into parallel processing,
and then collect the results in order.

I believe this would need some new infrastructure like apple did with great_central,
so I'm talking along these lines.
(Aka we only want to open up as many threads as there are cores)

This would not help much with a simple map (because we need to collect and serialize the results in the end)
but it could be cool for more complex mappings.

Maybe this should have been a rails topic first,
because there it might make a lot of sense to render partials in parallel,
this could really speed up rails response rendering ;-)

But I believe this is a topic for ruby in general.
Given that we now go to really many cores,
it should be at the heart of ruby to use that.

For a start we could just declare #parallel as -> return self.
and than improve on it.

A naive implementation would just open a new thread for every parallel map block,
but that would not work well -- it actually would be very slow.

So we actually need a thread management,
along the lines of great central.

I believe this should be built into the very core libs of the ruby language.

I know this is hard stuff, but it needs to be done.
Let's go parallel from the heart of the language.

---

Please let us discuss if adding the #parallel method would be a good idea.
Let's come up with some implementations to find the problems.

I'm explicitly referring to map#parallel here.
In rails that might be cache.parallel (if you followed)

for some math tasks it might also be
parallel do {}

[matz (Yukihiro Matsumoto)]

Show me your imaginary code example, so we can understand your concrete intention.
There's lot of issues around concurrency in Ruby. Simply adding map.parallel may not solve the problem.
But we are interested in your idea.

Matz.