Feature #17845: Windows Ruby - ucrt build?

• Status: Closed
• Assignee: -
• Target version: 3.1

[MSP-Greg (Greg L)]

Currently, Windows Ruby is normally compiled two ways.

The first, mswin, is compiled using Microsoft's current Visual C compiler, and links to the universal runtime (ucrt).

The second, mingw, is compiled using MinGW gcc. This links to msvcrt, an older version of Microsoft's Visual C runtime.

Previously, all the MSYS2 MinGW packages linked to msvcrt. The MSYS2 project is now releasing build tools and packages linking to ucrt.

MSYS2 has provided ruby packages, and GitHub user @Biswa96 contributed https://github.com/msys2/MINGW-packages/pull/8518, allowing Ruby to compile with ucrt.

I tried the patch with ruby-loco, and it builds. There are some test issues, haven't had a chance to look at them yet.

Normally, using exe/dll/so files together that use different versions of the runtime is not a good idea. Building extension gems with ucrt may provide gems that can be used with mswin, but can't be compiled due to issues with Visual C (vs gcc). Also, packages from the vcpkg project may be compatible with Ruby ucrt.

The reason for this post is that Windows Ruby built with ucrt is essentially another platform, but the build 'looks like' a mingw build.

This is really a third platform choice. What should it be called? Things like CONFIG['RUBY_SO_NAME'], RUBY_PLATFORM, etc?

[xtkoba (Tee KOBAYASHI)]

I wonder if there is a standard target triplet for MinGW + UCRT. I came up with x86_64-w64-mingw32-ucrt, but when googled it hits only one page. I suppose something like x64-mingw32-ucrt is acceptable for target_os in configure, as there is a preceding example for Android: aarch64-linux-android vs aarch64-linux (GNU/Linux), although I have no idea whether it is sufficient to just suffix target_os.

[MSP-Greg (Greg L)]

MinGW msvcrt packages have a prefix of mingw-w64-x86_64,
MinGW  ucrt  packages have a prefix of mingw-w64-ucrt-x86_64.

[larskanis (Lars Kanis)]

I wonder if there is a standard target triplet for MinGW + UCRT.

The compiler triplet for MINGW with UCRT is still 'x86_64-w64-mingw32'. This is defined in MSYS2 here. It is not distinct from MSVCRT based MINGW because it differs only in a library (msvcrt.dll vs. ucrtbase.dll) and the dependent libraries are not part of the compiler triplet.

I did some testing with the patch used in MSYS2. It doesn't establish any new platform strings, so that RbConfig is mostly the same with MSVCRT and UCRT. Extension libraries linked to MSVCRT usually load into a UCRT based ruby at the first attempt, but they fail later on. FFI fails when using long double values and Nokogiri fails in some test case while calling free() of msvcrt.dll. These are expected issues, when mixing MSVCRT and UCRT into one process.

So, I would propose a patch similar to this rather than the one in MSYS2. This sets

• CONFIG['arch'] = 'x64-mingw32' # equal in MSVCRT and UCRT
• CONFIG['sitearch'] = 'x64-ucrt' # vs. 'x64-msvcrt' in MSVCRT
• CONFIG['RUBY_SO_NAME'] = 'x64-ucrt-ruby310' # vs. 'x64-msvcrt-ruby310' in MSVCRT
• RUBY_PLATFORM = 'x64-mingw32' # equal in MSVCRT and UCRT

I'm uncertain if the use of UCRT should be reflected in RUBY_PLATFORM. In any case it should still contain "mingw" since I've seen a lot of code checking for Windows by RUBY_PLATFORM=~/mingw|mswin/.

Similarly Rubygems should use a different platform string for UCRT and MSVCRT based gems, since binary extensions are incompatible. This is still missing in the patch above. Maybe 'x64-mingw32-ucrt' as rubygems platform or 'x64-mingwucrt' or the platform of MSWIN?

My rough idea is to switch to UCRT with RubyInstaller-3.1. That means that all current releases keep using MSVCRT and that there will be no MSVCRT based RubyInstaller-3.1.

[MSP-Greg (Greg L)]

@larskanis (Lars Kanis)

Sorry, I meant to ping you immediately, but got distracted. Also, I saw your changes to ridk, but didn't see your patch file.

Anyway, I agree that there needs to be some way to distinguish between mingw and ucrt builds. And RubyGems/Bundler will need changes.

I haven't looked at runtime dependencies for a while. Question - wouldn't some (or most) extension gems compiled with ucrt be compatible with mswin builds, since there isn't a MSVC runtime mismatch?

I'll see what can be done to get ucrt usable with GH Actions...

[xtkoba (Tee KOBAYASHI)]

The test TestEnv#test_huge_value showed a difference between msvcrt and ucrt for x64-mingw32:

  1) Failure:
TestEnv#test_huge_value [C:/CIFS/ruby-trunk/test/ruby/test_env.rb:472]:
Exception raised:
<#<Errno::EINVAL: Invalid argument - ruby_setenv(foo)>>
Backtrace:
  C:/CIFS/ruby-trunk/test/ruby/test_env.rb:472:in `[]='

The test code is shown below. It seems that x64-mingw32 + ucrt behaves here as if /mswin/ =~ RUBY_PLATFORM and not like msvcrt. To make this test work correctly we need a way to distinguish between the two.

  def test_huge_value
    huge_value = "bar" * 40960
    ENV["foo"] = "bar"
    if /mswin/ =~ RUBY_PLATFORM
      assert_raise(Errno::EINVAL) { ENV["foo"] = huge_value }
      assert_equal("bar", ENV["foo"])
    else
      assert_nothing_raised { ENV["foo"] = huge_value }
      assert_equal(huge_value, ENV["foo"])
    end
  end

[naruse (Yui NARUSE)]

• Tracker changed from Misc to Feature
• Target version set to 3.1

[xtkoba (Tee KOBAYASHI)]

• File ruby-mingw32-ucrt.patch ruby-mingw32-ucrt.patch added

The attached patch is essentially a cleanup of that by @Biswa96 from GitHub. With this ./configure rb_cv_msvcrt=ucrt140 should work for Mingw-w64 with the default crt being UCRT [1].

[1] https://sourceforge.net/p/mingw-w64/mailman/message/36621319/

It would be desirable that the configure script is improved so that the checking for mingw32 runtime DLL can find the conftest executable linked against UCRT.

[xtkoba (Tee KOBAYASHI)]

• File ruby-configure-mingw32-ucrt.patch ruby-configure-mingw32-ucrt.patch added

A patch is attached for the configure script to set rb_cv_msvcrt=ucrt and RT_VER=140 when UCRT is used.

[xtkoba (Tee KOBAYASHI)]

• File ruby-configure-coroutine-mingw32.patch ruby-configure-coroutine-mingw32.patch added

Pattern matching for target_os in configure script should be permissive if we consider suffixing something onto "mingw32".

[xtkoba (Tee KOBAYASHI)]

• File ruby-i386-mingw32-ucrt-undef.patch ruby-i386-mingw32-ucrt-undef.patch added

_WIN64 is not defined on i386-mingw32 + ucrt.

[xtkoba (Tee KOBAYASHI)]

• File ruby-date-mingw32-ucrt-timezone.patch ruby-date-mingw32-ucrt-timezone.patch added

Or should extconf.rb be improved to notice deprecation?

../../../ext/date/date_core.c:7803:17: warning: 'timezone' is deprecated: Only provided for source compatibility; this variable might not always be accurate when linking to UCRT. [-Wdeprecated-declarations]
    of = (long)-timezone;
                ^
/usr/x86_64-w64-mingw32_ucrt/x86_64-w64-mingw32/include/time.h:257:32: note: 'timezone' has been explicitly marked deprecated here
  _CRTIMP extern long timezone __MINGW_ATTRIB_DEPRECATED_UCRT;
                               ^
/usr/x86_64-w64-mingw32_ucrt/x86_64-w64-mingw32/include/time.h:249:5: note: expanded from macro '__MINGW_ATTRIB_DEPRECATED_UCRT'
    __MINGW_ATTRIB_DEPRECATED_MSG( \
    ^
/usr/x86_64-w64-mingw32_ucrt/x86_64-w64-mingw32/include/_mingw.h:182:58: note: expanded from macro '__MINGW_ATTRIB_DEPRECATED_MSG'
#define __MINGW_ATTRIB_DEPRECATED_MSG(x) __attribute__ ((__deprecated__(x)))
                                                         ^

[xtkoba (Tee KOBAYASHI)]

A separate ticket for -Wparentheses warning issue: #17946

[larskanis (Lars Kanis)]

Thank you @xtkoba (Tee KOBAYASHI) for working on this! I replaced my previous patches by the above patches in rubyinstaller2-packages.

One thing I noticed is that you disabled the call to _CrtSetReportMode(_CRT_ASSERT, 0) in MINGW-UCRT. Actually gcc provides the function, if crtdbg.h is included. However I don't know if it makes any functional difference to call it or not.

Or should extconf.rb be improved to notice deprecation?

IMHO it's OK to use defines and to keep extconf.rb untouched.

[xtkoba (Tee KOBAYASHI)]

On Mingw-w64 _CrtSetReportMode is defined in crtdbg.h as

#define _CrtSetReportMode(t,f) ((int)0)

thus I believe the definition in win32/win32.c makes no difference although it may cause a -Wmacro-redefined warning.

[larskanis (Lars Kanis)]

The rubyinstaller-head nightly build is compiled with MINGW-UCRT since a few days. So all projects that use ruby-setup with version "head" or Appveyor with rubyinstaller-head download run on it now. Rubyinstaller-head already triggered a small issue in ffi and fxruby but other projects seems to run smooth.

The experience with rubyinstaller-head strengthens my plan to switch to UCRT with RubyInstaller-3.1.

Regarding CI runs: ruby-setup with version "mingw" is still on MSVCRT and https://github.com/MSP-Greg/setup-ruby-pkgs doesn't support UCRT builds so far.

@MSP-Greg (Greg L): Would you mind to add support to setup-ruby-pkgs? Do you want to switch your ruby-logo build to UCRT as well or add it as another architecture?

[MSP-Greg (Greg L)]

@larkanis

Do you want to switch your ruby-logo build to UCRT as well or add it as another architecture?

I think adding as another architecture would be best? Maybe tag it as ruby-ucrt?

Would you mind to add support to setup-ruby-pkgs?

I'll open an issue about adding the ucrt build tools to the Actions images so a full download isn't required.

Also, there's a lot of patches here, will they be added to ruby/ruby?

Re setup-ruby-pkgs, maybe the best solution is to install ucrt packages matching the listed mingw package when the build is ruby-head (or head)?

Not sure whether to discuss here or in a setup-ruby-pkgs issue. I think this affects setup-ruby, as it adds the tools paths to ENV['Path']... @Eregon (Benoit Daloze)?

Sorry, I've been following this, but didn't expect you'd convert your build so quickly. I'm fine with it though...

[larskanis (Lars Kanis)]

I opened a pull request to ruby including the above patches: https://github.com/ruby/ruby/pull/4599

One remaining issue is about the gem platform. It is currently:

• Gem::Platform.local.to_s => "x64-mingw32" # equal in MSVCRT and UCRT

BUT gems with C-extension aren't compatible between MINGW-MSVCRT and MINGW-UCRT. Loading fails due to different libruby names (due to ruby-configure-mingw32-ucrt.patch above). If libruby would be named equally, then C extensions would often work to some extend, but can fail with a segfault or deliver wrong results.

On the other hand the MSWIN platform isn't compatible to MINGW-UCRT. In theory gems compiled for MINGW-UCRT should run flawlessly on MSWIN, since they both share the same C runtime. But extensions compiled with MSWIN link to x64-vcruntime140-ruby310.dll, so that either loading of the DLL fails or the entry addresses are different leading to error incompatible library version. The other way around of using MSWIN gems in MINGW-UCRT doesn't work due to the additional dependency to vcruntime140.dll, which isn't part of Windows nor MSYS2.

More details to the different Windows compiler platforms are here: https://www.msys2.org/docs/environments/

Since all three Windows platforms aren't compatible they should have distinct Rubygems platform strings. I propose x64-mingw32-ucrt which fits into the naming scheme of MSWIN x64-mswin64-140 and the proposed x86_64-linux-musl for Apline Linux. Or should we use the chance to drop the misleading "32" and call it x64-mingw-ucrt?

[larskanis (Lars Kanis)]

Thinking a bit more about the ABI incompatibilities convinced me more about the need to differentiate the ruby platforms on Windows. So I added another patch to https://github.com/ruby/ruby/pull/4599 which sets CONFIG['arch'] and RUBY_PLATFORM to "x64-mingw-ucrt".

This allows easy differentiation between ABI incompatible platforms like MSWIN64 and MSVCRT-based MINGW32 and it also implicates a distinct rubygem platform which is also "x64-mingw-ucrt". See also the table in the pull request.

Although the term "mingw32" is the OS-part even for 64 bit systems, the "32" is misleading and confusing for many users. Therefore the new platform string drops the "32" from the OS part to just "mingw". This conforms to the common practice of windows platform testing per RUBY_PLATFORM=~/mswin|mingw/ .

Binaries are available here: https://github.com/oneclick/rubyinstaller2/releases/tag/rubyinstaller-head

I did some testing with several rails apps on rubyinstaller-head (x64-mingw-ucrt) and it did not show any regressions. There is only one issue with bundler: x64-mingw-ucrt is treated as platform "ruby" and "mri" in the Gemfile. This is probably conform to the understanding of most users, but it is against the specification of bundlers platform argument which classifies "mri" as NOT Windows.

[larskanis (Lars Kanis)]

IMHO the PR https://github.com/ruby/ruby/pull/4599 is ready to be merged. Are the platform strings OK as summarized in the PR? Are there any changes to be done to get it in?

The bundler issue described above is rather peripheral. I think it's best to discuss it with the bundler team directly.

[MSP-Greg (Greg L)]

GitHub Actions has added a Windows-2022 platform, and it only has MSYS2 packages installed, which provide most of the bash shell functionality. The MSYS2 packages do not include the MinGW build tool packages. This will require changes to ruby/setup-ruby to support the platform.

Rather than having every Windows Actions CI job pull build tool packages from the MSYS2 site, the idea of running an Actions CRON job (maybe once or twice daily) in another repo to gather the build tools, and package them in a zip (7z) file as a release. ruby/setup-ruby would download the appropriate file (mingw64 or ucrt64) and install the tools, which provides much faster CI. Personally, I think packaging all the tools needed to built Ruby would be best, and would be similar to Ubuntu & macOS. This install concept was actually used in Actions before MSYS2 was added to the images.

This would also mean that MSP-Greg/setup-ruby-pkgs wouldn't be needed for updates to build tools, as it's currently used for in many repos.

I'm mentioning this as the work needed to support Windows-2022 in Actions is extensive, and moving forward with Ruby ucrt64 builds would be helpful so that the Windows-2022 support could be done in one pass...

[MSP-Greg (Greg L)]

I said:

'This would also mean that MSP-Greg/setup-ruby-pkgs wouldn't be needed for updates to build tools, as it's currently used for in many repos.'

This would only be true if repos only run CI on the Window-2022 platform.

[MSP-Greg (Greg L)]

This can be closed.

[jeremyevans0 (Jeremy Evans)]

• Status changed from Open to Closed