Ruby - Bug #15245: Heap buffer overflow (write of size 8) in vm.inc

wanabe (_ wanabe)
2018-10-23T23:49:24Z

It may be due to peephole_optimization.

I went into an infinite loop without peephole_optimization.
It is an expected behaviour.

$ ruby -ve 'iseq = RubyVM::InstructionSequence.compile("%w(1) || 2 while 3", peephole_optimization: false); puts iseq.disasm; iseq.eval'
ruby 2.6.0dev (2018-10-24 trunk 65341) [x86_64-linux]
<compiled>:1: warning: literal in condition
== disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(1,18)> (catch: FALSE)
== catch table
| catch type: break  st: 0006 ed: 0015 sp: 0000 cont: 0015
| catch type: next   st: 0006 ed: 0015 sp: 0000 cont: 0003
| catch type: redo   st: 0006 ed: 0015 sp: 0000 cont: 0006
|------------------------------------------------------------------------
0000 jump                         12                                  (   1)[Li]
0002 putnil
0003 pop
0004 jump                         12
0006 putstring                    "1"
0008 newarray                     1
0010 branchif                     12
0012 jump                         6
0014 putnil
0015 leave
^CTraceback (most recent call last):
2: from -e:1:in ` '
	1: from -e:1:in `eval'
<compiled>:1:in `<compiled>': Interrupt

And I got SEGV with peephole_optimization.

$ ruby -ve 'iseq = RubyVM::InstructionSequence.compile("%w(1) || 2 while 3", peephole_optimization: true); puts iseq.disasm; iseq.eval'
ruby 2.6.0dev (2018-10-24 trunk 65341) [x86_64-linux]
<compiled>:1: warning: literal in condition
== disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(1,18)> (catch: FALSE)
== catch table
| catch type: break  st: 0004 ed: 0009 sp: 0000 cont: 0009
| catch type: next   st: 0004 ed: 0009 sp: 0000 cont: 0003
| catch type: redo   st: 0004 ed: 0009 sp: 0000 cont: 0004
|------------------------------------------------------------------------
0000 jump                         4                                   (   1)[Li]
0002 putnil
0003 pop
0004 putstring                    "1"
0006 jump                         4
0008 putnil
0009 leave
<compiled>:1: [BUG] Segmentation fault at 0x000055d6d4924000
ruby 2.6.0dev (2018-10-24 trunk 65341) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0004 p:0004 s:-5775390691329 e:000013 TOP    <compiled>:1 [FINISH]
c:0003 p:---- s:0011 e:000010 CFUNC  :eval
c:0002 p:0033 s:0007 E:002428 EVAL   -e:1 [FINISH]
c:0001 p:0000 s:0003 E:000dd0 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
-e:1:in ` '
-e:1:in `eval'
<compiled>:1:in `<compiled>'

-- Machine register context ------------------------------------------------
 RIP: 0x000055d6d307dd52 RBP: 0xfffffffffffffffc RSP: 0x00007ffcda768900
 RAX: 0x000055d6d3fa3b38 RBX: 0x000055d6d439f600 RCX: 0x000055d6d4924000
 RDX: 0x000055d6d3fa3b38 RDI: 0x000055d6d3fa3b38 RSI: 0x000055d6d4924008
  R8: 0x0000000000003f98  R9: 0x000055d6d3f49140 R10: 0x0000000000000b58
 R11: 0x000000000000016c R12: 0x00007fdc5584bfa0 R13: 0x000055d6d404e530
 R14: 0x000055d6d439f600 R15: 0x00007fdc5584bf30 EFL: 0x0000000000010202

-- C level backtrace information -------------------------------------------
malloc_consolidate(): invalid chunk size