Feature #20345
Updated by katei (Yuta Saito) 8 months ago
## Motivation
Today, CRuby runs on many platforms. But not all platforms are capable of running build tools (e.g. WebAssembly/WASI), so cross-target compilation against extensions libraries is essential for those platforms.
We currently have 3 major mkmf users (`extconf.rb` consumers in in other words):
1. CRuby build system
2. rake-compiler
3. RubyGems
[1] CRuby build system and [2] rake-compiler have their bespoke tricks to support cross compilation but [3] does not support cross compilation yet. So we are going to support cross-compilation in RubyGems to unlock the use of gems including non-precompiled extension libraries.
However, introducing the same tricks to RubyGems to support cross compilation as well as the other two is not ideal and cannot handle some edge cases properly.
Therefore, this proposal aims to add cross-compilation support in mkmf itself and remove the need for special tricks in mkmf users.
Note that cross-compilation here includes:
- Cross *platform* compilation: Build extension libraries for platform A on platform B.
- Cross *ruby version* compilation: Build extension libraries for Ruby X with running mkmf.rb bundled with Ruby X on Ruby Y.
## Existing Solutions
We currently have two solutions to cross-compile extension libraries, but both solutions are based on faking `rbconfig`.
### CRuby build system
CRuby build system is capable for cross-compiling extension libraries for cross-platform and cross ruby version.
The key trick here is that CRuby build system generates <platform>-fake.rb that fakes `RUBY_` constants like `RUBY_PLATFORM` and loads just built `rbconfig` describing Ruby version X for platform A and prevents loading `rbconfig` for Ruby version Y for platform B.
As a result, this fakes the global `RbConfig` constant and mkmk generates Makefile using the faked `RbConfig`.
### rake-compiler
rake-compiler also fakes `RbConfig` as well as CRuby build system does. One of the notable tricks here is that the faking script loads `resolv`, which expects the original `RUBY_PLATFORM`, at first and fake RbConfig after that.
```ruby
# From https://github.com/rake-compiler/rake-compiler/blob/7357f9e917dae79350687782c22596a036693405/lib/rake/extensiontask.rb#L559-L563
# Pre-load resolver library before faking, in order to avoid error
# "cannot load such file -- win32/resolv" when it is required later on.
# See also: https://github.com/tjschuck/rake-compiler-dev-box/issues/5
require 'resolv'
require 'rbconfig'
```
This has been introduced as a workaround but this indicates that the faking method cannot be generally applied.
## Problems
Based on insights from the existing solutions, the problems here are:
1. There is no way to tell the target `RbConfig` to `mkmf` without polluting the global `RbConfig` constant.
2. There is no public API to retrieve the deployment target info, so existing `extconf.rb` assumes `::RbConfig` is the one.
## Proposal
I propose adding those interfaces to `mkmf`:
1. `--target-rbconfig` option to override the RbConfig used for generating Makefiles without replacing the global top-level `RbConfig` module.
2. `MakeMakefile::RbConfig` constant to access the RbConfig for the target platform.
By default, it's an alias of top-level `RbConfig`. If `--target-rbconfig` is given, it points to the specified `RbConfig` definition.
```console=
$ ruby extconf.rb --target-rbconfig=path/to/rbconfig.rb --target-rbconfig path/to/rbconfig.rb
```
```ruby
require "mkmf"
system(
"./libyaml/configure",
# Before:
# "--host=#{RbConfig::CONFIG['host']}",
"--host=#{MakeMakefile::RbConfig::CONFIG['host']}",
...
)
# Before:
# case RUBY_PLATFORM
case MakeMakefile::RbConfig::CONFIG['platform']
when /mswin|mingw|bccwin/
...
when /linux/
...
end
create_makefile("psych")
```
Extension library authors who want to support cross-compilation just need to replace their use of some constants in `extconf.rb` that assume the config describes the deployment target. Here is the list of faked constant variables and corresponding representations compatible with cross-compilation.
| Before | After (to make the ext x-compile ready) |
|:------|:-----|
|`RbConfig` | `MakeMakefile::RbConfig` |
|`RUBY_PLATFORM` | `MakeMakefile::RbConfig::CONFIG["platform"]` |
|`RUBY_VERSION` | `MakeMakefile::RbConfig::expand("$(MAJOR).$(MINOR).$(TEENY)")` |
|`RUBY_DESCRIPTION` | No corresponding config entry |
## Compatibility
This is a completely additive change, so I expect there is no compatibility issues for existing `extconf.rb`.
Note that migrating `RbConfig` to `MakeMakefile::RbConfig` does not break existing faked `RbConfig` based cross-compilation because `MakeMakefile::RbConfig` is an alias of `::RbConfig` by default and it's the faked config describing the deployment target in this scenario.
Also extension library authors who want to support cross-compilation and want to keep build with older Ruby before this change can include the following snippet at the beginning of `extconf.rb`:
```ruby=
MakeMakefile::RbConfig ||= RbConfig
```
## Implementation
Literally a few lines of changes: https://github.com/kateinoigakukun/ruby/commit/9f3090c26ae1e5712dee702c19ba7a50695dd86a
## Evaluation
I ported nokogiri gem, which has [1k lines of `extconf.rb`](https://github.com/sparklemotion/nokogiri/blob/v1.16.3/ext/nokogiri/extconf.rb) and several platform specific branches, to WebAssembly/WASI with this change, and the new API was enough to satisfy the cross-compilation scenario.