Feature #11475: AST transforms

• Status: Open
• Assignee: -
• Target version: -

[Txus (Josep M. Bach)]

Even though Ruby is in many ways an acceptable Lisp, extending the language itself is a hard endeavour. By using reflection/meta-programming and taking advantage of its rich grammar, it's certainly possible to bend the language in extreme ways (believe me, I've done it), but there are a couple of serious limitations:

• Everything must be done at run-time, incurring in necessary overhead, sometimes unacceptably so.
• The resulting code must be valid Ruby not only syntactically, but also semantically. That forbids the introduction of new semantic constructs by users themselves.

Now, some people might argue in favour of a special DSL for compile-time macro-building, possibly augmenting the grammar and introduce all sorts of new tricks to do that. But I think that wouldn't be the most general solution, nor the most desirable. The alternative: a simple, object-oriented AST transforms API.

By building an AST transforms API and making it available to users, I argue that we could kill many birds with one stone:

• A rich macro system (or many!) could be implemented as a simple library, with no changes to core ruby.
• Some micro-optimizations could be implemented as a library, which may find their way into core ruby later.
• Experimental, new semantic constructs could be made available to users through simple libraries, and when time has proven their usefulness, maybe they could be implemented natively, if needed.

I'd be up for the task if there is enough interest -- what are people's thoughts on this?

[nobu (Nobuyoshi Nakada)]

• Project changed from 14 to Ruby master

Do you have any ideas for the API?

[Txus (Josep M. Bach)]

Nobuyoshi Nakada wrote:

Do you have any ideas for the API?

My API idea is loosely based on the Rubinius' transform API, but without explicitly generating bytecode in the transform (that wouldn't be very portable, and it makes transforms hard to compose). Instead, the #transform method returns a new, immutable AST node. Here's how a user would implement their own transform:

class MyTransform
  def matches?(ast_node)
    # return boolean indicating whether this particular AST node is targeted by our transform
  end

  def transform(ast_node)
    # return new, transformed ast node
  end
end

Compiler.add_transform(MyTransform.new)

Note that the AST nodes should be simple immutable values, such as the ones from whitequark's ast gem.

The semantics of Compiler.add_transform would be that, from that moment on, the parser stage of the compiler applies the transform. That means any subsequent load, require or eval calls would run the transform.

A side benefit of that is that transforms could be stacked in load order, and refine the AST progressively. Or another option is to have explicit order, a la Rack middleware, and be able to decide which transforms run before which.