Feature #21827: Deprecating Ripper

• Status: Open
• Assignee: -
• Target version: -

[Eregon (Benoit Daloze)]

I think it is time to consider deprecating Ripper.
Matz has agreed that going forward the official parser API for Ruby will be the Prism API so it's clear the official Ruby parsing API is the Prism API.

Ripper while being currently supported on CRuby, JRuby and TruffleRuby (partially) has a significant maintenance overhead.
For CRuby, it roughly doubles the number of lines in parse.y and adds lots of #ifdef RIPPER/#ifndef RIPPER which makes it really hard to follow.
For JRuby it's many thousands of lines of code to maintain just for that: https://github.com/jruby/jruby/tree/master/core/src/main/java/org/jruby/ext/ripper
For TruffleRuby, ripper is by far the most difficult C extension to support as it relies on many internal functions. When an internal function is used by ripper is missing it kills the entire process. On every update of RUBY_VERSION in TruffleRuby, the biggest amount of time is spent dealing with Ripper, which feels as a result like a major hack.

It could be possible to implement Ripper based on Prism, in fact there is some support for that, but the fact the Ripper API is rather unstable means it's hard to be compatible, and currently the translation layer relies on original ripper for some things.

It's also worth noting that Ripper.sexp is 2.75x slower than Prism.parse, while providing a much worse API (widely recognized as such, e.g. Ripper.sexp("def m = 42") vs Prism.parse("def m = 42"), lots of nil with unclear meaning in Ripper).
So from a new code perspective there seems to be very little value to use Ripper.

Ripper has lexing via Ripper.lex, which can be replaced by Prism::lex_compat.

I think we should start to deprecate Ripper.

The advantages of deprecating it are:

• Make it clear that this API should not be used in new code
• Encourage the usages to migrate to the Prism API, which is stable, officially supported, designed with many people to be usable and convenient for every parsing/tooling usage, etc.
• Improve portability of Ruby code between Ruby implementations (given Ripper is not fully compatible between Ruby implementations)

I propose to deprecate Ripper in 4.1.
I'm not sure when it would be appropriate to remove it given it's a more established API, maybe in 4.2, 4.3?

[Eregon (Benoit Daloze)]

• Tracker changed from Bug to Feature
• Backport deleted (3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN)

[st0012 (Stan Lo)]

Ripper is at least used in these default/bundled gems:

• irb
• rdoc
• rbs
• power_assert
• syntax_suggect

While I agree that we should deprecate it, I also would like to avoid showing the deprecation message when users use these gems in the next Ruby release.
Can you contact related maintainers to understand how hard it'd be for the project to migrate off Ripper?

Wrt irb and rdoc, which I maintain:

• @tompng (tomoya ishida) and I already decided to migrate to Prism-based parser/tokenizer for RDoc this year. So we should be good there
• irb uses Ripper quite extensively and doesn't just call Ripper.lex. So it'll probably take some work to migrate

[kddnewton (Kevin Newton)]

I don't think we're going to be able to deprecate in 4.1, Ripper is still very much used in the ecosystem. I think we can do the work over the next couple of years to remove it, but 4.1 is quite a tight timeline.