Project

General

Profile

Actions

Feature #3187

closed

Allow dynamic Fiber stack size

Added by mperham (Mike Perham) almost 14 years ago. Updated over 2 years ago.

Status:
Rejected
Target version:
-
[ruby-core:29734]

Description

=begin
I'd like a way to increase the size of the Fiber stack dynamically so when my program starts, I can set it to whatever value I need for the code I'm running. 4KB is too easy to run into problems when running recursive code but settling on any arbitrary static value seems pointless to me.
=end


Files

fiber_stacksize.patch (5.89 KB) fiber_stacksize.patch nagachika (Tomoyuki Chikanaga), 10/19/2011 11:26 PM

Related issues 1 (1 open0 closed)

Related to Ruby master - Feature #6694: Thread.new without block.Assignedko1 (Koichi Sasada)Actions
Actions #1

Updated by mperham (Mike Perham) almost 14 years ago

=begin
To be clear, I'm not asking to extend the stack of a currently executing Fiber, just a one-time configuration when my program starts. Something like this:

 require 'fiber'
 Fiber.stack_size = 16 * 1024

 Fiber.new do
   parse_some_big_xml
 end.resume

=end

Actions #2

Updated by mame (Yusuke Endoh) almost 14 years ago

  • Target version set to 2.0.0

=begin

=end

Updated by rupert (Robert Pankowecki) over 12 years ago

I would also welcome such improvement.

Updated by nagachika (Tomoyuki Chikanaga) over 12 years ago

Hi,

I've written a patch. This patch adds some methods Fiber.default_vm_stacksize, Fiber.default_vm_stacksize=, Fiber#vm_stacksize and add an optional Hash argument to Fiber#initialize, and tests for them.
You can specify default VM stack size of Fiber created afterward and/or specify individually when create a Fiber.

ex)
Fiber.default_vm_stacksize = 16 * 1024

Fiber.new do
do_something
end

or

Fiber.new(:vm_stacksize => 16 * 1024) do
do_simething
end

Note that the size of VM stack is number of Objects and memsize of stack is vm_stacksize * sizeof(VALUE) bytes.
Also note that this patch enable to change only VM stack size, but not machine stack size. I think when Fiber is implemented based on makecontext/swapcontext (FIBER_USE_NATIVE=1), machine stack size (default: 64KB) can be configurable. I wonder if procedures like `parse_some_big_xml' in Mike's example also need larger machine stack size? Does anyone have such a testcase?

Updated by kosaki (Motohiro KOSAKI) over 12 years ago

Note that the size of VM stack is number of Objects and memsize of stack is vm_stacksize * sizeof(VALUE) bytes.
Also note that this patch enable to change only VM stack size, but not machine stack size. I think when Fiber is implemented based on makecontext/swapcontext (FIBER_USE_NATIVE

Updated by nagachika (Tomoyuki Chikanaga) over 12 years ago

I don't think we should export vm_stack. It's purely implementation
detail. Just expose "stack size"
knob and it should change both vm-stack and machine-stack size.
You're right. It's better to provide more abstract way to tune memory usage of Fiber.
How about like the following?

Fiber.stacksize = 2.0 # => twice the size of default stack size

or maybe we have to examine how VM/machine stack are consumed.

And even tough we provide such a unified interface, I think more low-level,
implementation specific methods are convenient when users need fine-tuned parameter set for
their applications. Is it a bad idea?
for example, ObjectSpace.count_objects etc.. tightly depend on MRI implementation,
and are useful just because of it.

Updated by spatulasnout (B Kelly) over 12 years ago

Tomoyuki Chikanaga wrote:

I've written a patch. This patch adds some methods Fiber.default_vm_stacksize, Fiber.default_vm_stacksize=, Fiber#vm_stacksize and add an optional Hash argument to Fiber#initialize, and tests for them.
You can specify default VM stack size of Fiber created afterward and/or specify individually when create a Fiber.

ex)
Fiber.default_vm_stacksize = 16 * 1024

Fiber.new do
do_something
end

or

Fiber.new(:vm_stacksize => 16 * 1024) do
do_simething
end

Very nice!

My application uses fibers extensively, and it began to exceed the
default fiber stack during recursive traversal of relatively shallow
trees. (Not surprising, given the 4K default stack size.)

I had patched cont.c locally as follows:

#define FIBER_STACK_SIZE_SCALE 8 /* need more fiber stack space */

#define FIBER_MACHINE_STACK_ALLOCATION_SIZE (0x10000 *
FIBER_STACK_SIZE_SCALE)

#define FIBER_VM_STACK_SIZE ((4 * 1024) * FIBER_STACK_SIZE_SCALE)

Note that the size of VM stack is number of Objects and memsize of stack is vm_stacksize * sizeof(VALUE) bytes.
Also note that this patch enable to change only VM stack size, but not machine stack size. I think when Fiber is implemented based on makecontext/swapcontext (FIBER_USE_NATIVE=1), machine stack size (default: 64KB) can be configurable. I wonder if procedures like `parse_some_big_xml' in Mike's example also need larger machine stack size? Does anyone have such a testcase?

Can anyone comment on the relationship between the VM stack size and the
machine stack size?

I scaled them both equally to be "safe".

But I don't know how they are related, so I'm not sure if the
corresponding increase to the machine stack was necessary?

Thanks,

Bill

Updated by ko1 (Koichi Sasada) over 12 years ago

Hi,

I agree with this proposal. Also add same parameter setting feature for
Thread.

However, I can't make good API to specify stack (VM and machine) size
(and other parameters if there are).

There are several proposals.

Type 1: Thread creating argument (ex: Thread.new(stack_size: 1024))
Type 2: Thread global parameter (ex: Thread.stack_size = 1024)

I think Type 2 is not good (to set default value is okay. However it
conflicts other usage. Typically thread-unsafe).

I also think Type 1 has also problem. If you want to pass keyword
argument "stack_size" to fiber or thread, it should be conflict.

My idea is creating new Thread (Fiber) class with new parameter:

  MyThread = Thread.new_template(stack_size: 1024)
  MyThread.new{
    ...
  }

# Of course "new_template" is bad name.

Any ideas?

--
// SASADA Koichi at atdot dot net

Updated by headius (Charles Nutter) over 12 years ago

Quick comments.

  • JVM can specify per-thread stack size but always in bytes. Any API that
    exposes stack size should be abstract rather than require a knowledge of
    stack frame sizes from impl to impl. The "stack size factor" version is
    pretty good.

  • I agree it would be nice to specify per-thread stack sizes too. We must
    do this for JRuby on Android, and have to use JRuby- specific mechanisms
    for it.

  • Charlie (mobile)

Updated by nahi (Hiroshi Nakamura) about 12 years ago

  • Assignee set to ko1 (Koichi Sasada)

How about;

t = Thread.new(stack_size: 1024)
t.stack_size = 1024
t[:initial_tls_hash] = {}
t.run {
  ...
}
Actions #11

Updated by shyouhei (Shyouhei Urabe) about 12 years ago

  • Status changed from Open to Assigned

Updated by ko1 (Koichi Sasada) over 11 years ago

Any other idea about it? It's only API design.

I'm not sure nahi-san's idea is good for Ruby or not.
I feel that it is too different from current style.

Updated by ko1 (Koichi Sasada) over 11 years ago

I make another ticket about it:
https://bugs.ruby-lang.org/issues/6694

Thanks,
Koichi

--
// SASADA Koichi at atdot dot net

Updated by ko1 (Koichi Sasada) over 11 years ago

I'm considering it because no progress on https://bugs.ruby-lang.org/issues/6694 (sorry, it is naming issue, I think).

Can I add an environment variable (such as RUBY_FIBER_MACHINE_STACK_SIZE) to avoid this issue temporarily?

Updated by mame (Yusuke Endoh) over 11 years ago

  • Target version changed from 2.0.0 to 2.6

Updated by ko1 (Koichi Sasada) over 10 years ago

  • Status changed from Assigned to Feedback

Ruby 2.0 already has
RUBY_VM_FIBER_VM_STACK_SIZE
RUBY_FIBER_MACHINE_STACK_SIZE

is it enough?

Updated by jaredbeck (Jared Beck) about 9 years ago

Ruby 2.0 already has
RUBY_VM_FIBER_VM_STACK_SIZE
RUBY_FIBER_MACHINE_STACK_SIZE

Are these environment variables to configure the stack size? Is there documentation on usage? (I mean, what are the units? bytes, kb?) Is there a way to read the default value programmatically in ruby?

Updated by headius (Charles Nutter) almost 8 years ago

FYI, it appears at least one Ruby implementation has implemented this unilaterally: https://github.com/rubinius/rubinius/commit/c26139a03132661202f30c778ac9e7bc489959d4

We'd also like to support this feature in JRuby, but we'd prefer to go through official channels.

I do not believe env vars are sufficient because you may have different libraries that want to tune their threads/fibers to different sizes.

Actions #19

Updated by shyouhei (Shyouhei Urabe) over 7 years ago

Updated by shyouhei (Shyouhei Urabe) over 7 years ago

We looked at this issue at yesterday's developer meeting.

The (potential) problem here is the way Rubinius implements stack size accidentally breaks compatibility of how Thread.new works. In the current API all the argument passed to this method are forwarded to its block. As far as I read the patch pointed by Charles it seems Rubinius chose to break here and let it eat the keyword arguments. This might work -- given Rubinius lives without any serious problem around it -- but does change the way it works in the MRI.

Another approach would be to separate a thread creation; then inject arguments to it; then finally kick it to run. This is what ko1 proposed in issue #6694.

Updated by jjyr (Jinyang Jiang) about 6 years ago

How about

Thread.with_configure(stack_size: 1024).new(a: 1, b: 2){}
# or
Thread::Config.new(stack_size: 1024).start(a: 1, b: 2){}
# alias start new

ko1 (Koichi Sasada) wrote:

Hi,

I agree with this proposal. Also add same parameter setting feature for
Thread.

However, I can't make good API to specify stack (VM and machine) size
(and other parameters if there are).

There are several proposals.

Type 1: Thread creating argument (ex: Thread.new(stack_size: 1024))
Type 2: Thread global parameter (ex: Thread.stack_size = 1024)

I think Type 2 is not good (to set default value is okay. However it
conflicts other usage. Typically thread-unsafe).

I also think Type 1 has also problem. If you want to pass keyword
argument "stack_size" to fiber or thread, it should be conflict.

My idea is creating new Thread (Fiber) class with new parameter:

  MyThread = Thread.new_template(stack_size: 1024)
  MyThread.new{
    ...
  }

# Of course "new_template" is bad name.

Any ideas?

--
// SASADA Koichi at atdot dot net

Updated by nerdrew (Andrew Lazarus) over 2 years ago

Is this request still being considered?

Updated by mperham (Mike Perham) over 2 years ago

nerdrew (Andrew Lazarus) wrote in #note-22:

Is this request still being considered?

I consider this issue (and really any decade-old issue) irrelevant. Please close.

Actions #24

Updated by ko1 (Koichi Sasada) over 2 years ago

  • Status changed from Feedback to Rejected
Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0