Bug #19683: ruby-3.3.0-preview1 does not build with BSD make without --with-baseruby

• Status: Closed
• Assignee: -
• Target version: 3.3
• ruby -v: ruby 3.3.0preview1 (2023-05-12 master a1b01e7701) [x86_64-openbsd7.3]
• Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN

[jeremyevans0 (Jeremy Evans)]

ruby-3.3.0-preview1 does not build with BSD make without --with-baseruby. However, it builds fine with GNU make and when using BSD make with --with-baseruby. Previous Ruby tarball releases have built fine with BSD make without --with-baseruby, so this issue has been introduced recently.

When building with BSD make without --with-baseruby, building fails after generating RI format with:

/bin/sh:  false: not found
*** Error 127 in . (exts.mk:94 'ruby': @make   UPDATE_LIBRARIES=no EXTENCS=dmyenc.o BASERUBY=echo\ executable\ host\ ruby\ is\ required.\ \ ...)
*** Error 2 in /home/jeremy/local/ruby-3.3.0-preview1 (Makefile:948 'build-ext': @make -f exts.mk    libdir="/usr/local/lib" LIBRUBY_EXTS=./...)

[jeremyevans0 (Jeremy Evans)]

This is still a problem in 3.3.0-preview2. I've been using the following bad hack to work around it:

Index: common.mk
--- common.mk.orig
+++ common.mk
@@ -378,10 +378,10 @@ ext/configure-ext.mk: $(PREP) all-incs $(MKFILES) $(RB
 configure-ext: $(EXTS_MK)

 build-ext: $(EXTS_MK)
-       $(Q)$(MAKE) -f $(EXTS_MK) $(mflags) libdir="$(libdir)" LIBRUBY_EXTS=$(LIBRUBY_EXTS) \
+       -$(Q)$(MAKE) -f $(EXTS_MK) $(mflags) libdir="$(libdir)" LIBRUBY_EXTS=$(LIBRUBY_EXTS) \
            EXTENCS="$(ENCOBJS)" BASERUBY="$(BASERUBY)" MINIRUBY="$(MINIRUBY)" \
            UPDATE_LIBRARIES=no $(EXTSTATIC)
-       $(Q)$(MAKE) $(EXTS_NOTE)
+       -$(Q)$(MAKE) $(EXTS_NOTE)

 exts-note: $(EXTS_MK)
        $(Q)$(MAKE) $(EXTS_NOTE)

[jeremyevans0 (Jeremy Evans)]

• Target version set to 3.3

This was still a problem in 3.3.0-preview3, and is still a problem in 3.3.0-rc1. It should definitely be fixed before the final release of 3.3.0.

[jeremyevans0 (Jeremy Evans)]

I traced this issue to a1b01e7701f9fc370f8dff777aad6d39a2c5a3e3 (lrama), because that introduced the use of BASERUBY, which is what currently breaks this:

--- a/common.mk
+++ b/common.mk
@@ -298,7 +298,8 @@ configure-ext: $(EXTS_MK)
 
 build-ext: $(EXTS_MK)
        $(Q)$(MAKE) -f $(EXTS_MK) $(mflags) libdir="$(libdir)" LIBRUBY_EXTS=$(LIBRUBY_EXTS) \
-           EXTENCS="$(ENCOBJS)" MINIRUBY="$(MINIRUBY)" UPDATE_LIBRARIES=no $(EXTSTATIC)
+           EXTENCS="$(ENCOBJS)" BASERUBY="$(BASERUBY)" MINIRUBY="$(MINIRUBY)" \
+           UPDATE_LIBRARIES=no $(EXTSTATIC)
        $(Q)$(MAKE) $(EXTS_NOTE)
 
 exts-note: $(EXTS_MK)
@@ -1208,8 +1209,8 @@ $(srcdir)/ext/ripper/ripper.c: $(srcdir)/ext/ripper/tools/preproc.rb $(srcdir)/p
        $(Q) $(CHDIR) $(@D) && \
        $(CAT_DEPEND) depend | \
        $(exec) $(MAKE) -f - $(mflags) \
-               Q=$(Q) ECHO=$(ECHO) RM="$(RM1)" BISON="$(YACC)" top_srcdir=../.. srcdir=. VPATH=../.. \
-               RUBY="$(BASERUBY)" PATH_SEPARATOR="$(PATH_SEPARATOR)" LANG=C
+               Q=$(Q) ECHO=$(ECHO) RM="$(RM1)" top_srcdir=../.. srcdir=. VPATH=../.. \
+               RUBY="$(BASERUBY)" BASERUBY="$(BASERUBY)" PATH_SEPARATOR="$(PATH_SEPARATOR)" LANG=C

This isn't necessarily specific to BASERUBY, though. The underlying issue is that exts.mk (created from templates/exts.mk.tmpl) uses $(MFLAGS):

ruby:
	$(Q)$(MAKE) $(MFLAGS) $(SUBMAKEOPTS) $@

This doesn't work as the ; inside BASERUBY (defined as echo executable host ruby is required. use --with-baseruby option.; false) is not quoted. I'm not sure how to get it quoted correctly (neither the Q or QL modifiers work), I'm checking with the OpenBSD make maintainer.

One possible way to workaround the issue would be to ship a tool/missing-baseruby file that contained:

#!/bin/sh
echo "executable host ruby is required.  use --with-baseruby option."
exit 1

And use that as BASERUBY.

However, I think the best fix would be to not use BASERUBY here. This is after miniruby and ruby are built, we should be able to use the built ruby instead of BASERUBY when building extensions, unless I'm missing something.

[jeremyevans0 (Jeremy Evans)]

I tried the following patch and I was able to build master (which requires BASERUBY) and 3.3.0-rc1 (which does not) with it using OpenBSD make:

diff --git a/common.mk b/common.mk
index 0125645e37..6c637e0936 100644
--- a/common.mk
+++ b/common.mk
@@ -395,8 +395,7 @@ configure-ext: $(EXTS_MK)

 build-ext: $(EXTS_MK)
        $(Q)$(MAKE) -f $(EXTS_MK) $(mflags) libdir="$(libdir)" LIBRUBY_EXTS=$(LIBRUBY_EXTS) \
-           EXTENCS="$(ENCOBJS)" BASERUBY="$(BASERUBY)" MINIRUBY="$(MINIRUBY)" \
-           UPDATE_LIBRARIES=no $(EXTSTATIC)
+           EXTENCS="$(ENCOBJS)" MINIRUBY="$(MINIRUBY)" UPDATE_LIBRARIES=no $(EXTSTATIC)
        $(Q)$(MAKE) $(EXTS_NOTE)

 exts-note: $(EXTS_MK)

It passed on CI except on Windows when compiling with Visual Studio. It failed on Windows due to ripper. ext/ripper/depend uses BASERUBY:

BISON = $(BASERUBY) $(top_srcdir)/tool/lrama/exe/lrama

This cannot be easily switched to RUBY, because lrama requires strscan, and strscan is currently not built until after ripper. Even if you work around that to build strscan before ripper, RUBY here does not allow loading the strscan extension library:

../../miniruby -I'../..' -I'../.././lib' -I'../../.ext/x86_64-openbsd7.4' -I'../../.ext/common' ../.././tool/lrama/exe/lrama -t -v -oripper.c - ripper.y < ripper.y
/home/jeremy/tmp/ruby/tool/lrama/lib/lrama/lexer.rb:1:in `require': this executable file can't load extension libraries (LoadError)

For Ruby 3.3.0, the tool/missing-baseruby approach I mentioned earlier appears to work. I submitted a pull request for that approach: https://github.com/ruby/ruby/pull/9299

Longer term alternative possible solutions to avoid need for BASERUBY in build-ext task for use by ripper:

• Change lrama so it doesn't need strscan and can use RUBY instead of BASERUBY
• Create ripper.y.c when parse.y.c is created, at top level
• Maybe ripper will be removed or replaced by prism and will not need lrama?

• Status changed from Open to Closed

Applied in changeset git|7dca6b53a995a9782026f726c291344b5b8e0ab2.

---

Add tool/missing-baseruby.bat, used when BASERUBY not available

Previously, the embedded semicolon in BASERUBY if BASERUBY is
not available breaks tarball builds without BASERUBY when using
OpenBSD make, due to the inability to escape MFLAGS correctly.
This moves the same BASERUBY code into a separate file, avoiding
the MFLAGS quoting issue.

BASERUBY must be passed to build-ext because it is required
by ripper since the introduction of lrama.

Fixes [Bug #19683]

Co-authored-by: Nobuyoshi Nakada [e-mail]