Bug #5202: RubyLex very slow when lexing large files

• Status: Closed
• Assignee: keiju (Keiju Ishitsuka)
• Target version: -
• ruby -v: ruby 1.9.2p290 (2011-07-09) [i386-mingw32]

[ryanmelt (Ryan Melton)]

RubyLex exponentially slower when lexing large files based on the size of the file. This is because of a call to @readed.reverse inside of the get_readed and ungetc methods. The following updated versions of these methods greatly improve performance by using rindex instead of reverse.index.

def get_readed
if idx = @readed.rindex("\n")
@base_char_no = @readed.size - (idx + 1)
else
@base_char_no += @readed.size
end

readed = @readed.join("")
@readed = []
readed

end

def ungetc(c = nil)
if @here_readed.empty?
c2 = @readed.pop
else
c2 = @here_readed.pop
end
c = c2 unless c
@rests.unshift c #c =
@seek -= 1
if c == "\n"
@line_no -= 1
if idx = @readed.rindex("\n")
@char_no = idx + 1
else
@char_no = @base_char_no + @readed.size
end
else
@char_no -= 1
end
end

[naruse (Yui NARUSE)]

• Status changed from Open to Assigned
• Assignee set to keiju (Keiju Ishitsuka)

[keiju (Keiju Ishitsuka)]

• Status changed from Assigned to Closed
• % Done changed from 0 to 100

This issue was solved with changeset r38609.
Ryan, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

• lib/irb/ruby-lex.rb: improve RubyLex performance for large files
  [Bug #5202]. Patch by ryanmelt.