Bug #18061: Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found

• Status: Open
• Assignee: -
• Target version: -
• Backport: 2.6: UNKNOWN, 2.7: REQUIRED, 3.0: REQUIRED

[jaruga (Jun Aruga)]

I found an issue in our company's internal test called "execshield" by a security tool annobin - annocheck command [1][2].

Hardened: libruby.so.2.7.4: FAIL: property-note test because no .note.gnu.property section found

Here is the reproducer on the upstream latest master, commit is 5f2987d6c2ae9ace3178ac3e1bbb4ac7079101eb,

$ autoconf
$ ./configure --enable-shared
$ make

$ ls libruby.so.3.1.0
libruby.so.3.1.0*

If you are using Red Hat based Linux distro, it's easy to install by the RPM package like this.

$ sudo dnf -y install annobin-annocheck

$ sudo yum -y install annobin-annocheck

Then

$ annocheck libruby.so.3.1.0

If you are using other Linux distros such as Ubuntu, you can use it by a container I prepared.

Prepare the following Dockerfile.

$ cat Dockerfile 
FROM docker.io/fedora:34

RUN cat /etc/fedora-release
RUN dnf -y install annobin-annocheck
WORKDIR /work

Then build the container image with the Dockerfile and run the annocheck command for the libruby.so.3.1.0 on your host environment. The -v is an option for bind mount between host and container environment.

$ docker build --rm -t fedora-annocheck .

$ docker run --rm -t -v $(pwd):/work fedora-annocheck annocheck /work/libruby.so.3.1.0
annocheck: Version 9.79.
Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now 
Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage 
Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information 
Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found 
Hardened: Rerun annocheck with --verbose to see more information on the tests.

The message Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found is what I found in our internal test. For other FAIL messages, maybe it can be fixed by changing how to build.

Asking a colleague, I was told that the coroutine/*/Context.S files such as coroutine/x86/Context.S cause the failure. Do you have any idea how to fix this? Thanks.

• [1] https://sourceware.org/annobin/
• [2] You can see man annocheck or https://www.mankier.com/1/annocheck .

[nobu (Nobuyoshi Nakada)]

• File 0001-Add-.note.gnu.property-sections.patch 0001-Add-.note.gnu.property-sections.patch added

Does it work just by adding an empty section?

[jaruga (Jun Aruga)]

Thanks for the patch! Let me check it.

[jaruga (Jun Aruga)]

I applied the patch for the Ruby 3.0.2 in RHEL 8, built. But I still see the failed message.

annocheck: Version 9.65.
Hardened: libruby.so.3.0.2: FAIL: property-note test because no .note.gnu.property section found
Hardened: Rerun annocheck with --verbose to see more information on the tests.

I also applied the patch to the Ruby on master branch 1cb5a669d39fe741388be7ac45fc88c336b097cf . But the message is same.

$ autoconf
$ ./configure --enable-shared
$ make

$ annocheck libruby.so.3.1.0
annocheck: Version 9.79.
Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now 
Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage 
Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information 
Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found 
Hardened: Rerun annocheck with --verbose to see more information on the tests.

So, I don't think that adding the empty section works.

[jaruga (Jun Aruga)]

By the way, I found a case generating ".note.gnu.property section" in the OpenSSL project.
https://github.com/openssl/openssl/commit/51994e505dbb1cd0dd76869ec962e2948b77b585

The commit message explains how they verified the change. So in the case of Ruby, here is a way to verify?

I also applied the patch to the Ruby on master branch 1cb5a669d39fe741388be7ac45fc88c336b097cf .

On the above commit, I see an error because there is no .note.gnu.property non-empty section?

$ autoconf

$ CC="gcc -Wl,-z,cet-report=error -fcf-protection" ./configure --enable-shared

$ make
...
/bin/ld: coroutine/amd64/Context.o: error: missing IBT and SHSTK properties
/bin/ld: probes.o: error: missing IBT and SHSTK properties
collect2: error: ld returned 1 exit status
make: *** [Makefile:277: miniruby] Error 1

$ echo $?
2

[jaruga (Jun Aruga)]

It seems we can check if the .note.gnu.property exists in a binary file by the command readelf -S --wide <file>. The binary file ruby has it. But the file libruby.so.3.1.0 doesn't it.

$ readelf -S --wide ruby
There are 40 section headers, starting at offset 0x36038:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .interp           PROGBITS        0000000000400318 000318 00001c 00   A  0   0  1
  [ 2] .note.gnu.property NOTE            0000000000400338 000338 000020 00   A  0   0  8
...

$ readelf -S --wide libruby.so.3.1.0
There are 42 section headers, starting at offset 0xc15a38:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .note.gnu.build-id NOTE            0000000000000270 000270 000024 00   A  0   0  4
  [ 2] .gnu.hash         GNU_HASH        0000000000000298 000298 0033e4 00   A  3   0  8
...

[nobu (Nobuyoshi Nakada)]

• File 0001-Add-.note.gnu.property-sections.patch 0001-Add-.note.gnu.property-sections.patch added

That seems the same as a section generated by GCC.

[jaruga (Jun Aruga)]

Thanks for the patch! Let me check it.

[ioquatix (Samuel Williams)]

On another side, we have an Intel CET [2], [3]:
"Control-flow Enforcement Technology (CET) provides the following capabilities to defend
against ROP/JOP style control-flow subversion attacks:
- Shadow Stack - return address protection to defend against Return Oriented Programming,
- Indirect branch tracking - free branch protection to defend against Jump/Call Oriented Programming.​"

Intel CET uses following processor-specific program property types in .note.gnu.property [3, p85, p87]:

* GNU_PROPERTY_X86_FEATURE_1_IBT This indicates that all executable sections are compatible
with IBT (see Section 13.1.1) when endbr64 instruction starts each valid target where an indirect
branch instruction can land. 8
* GNU_PROPERTY_X86_FEATURE_1_SHSTK This indicates that all executable sections are compatible
with SHSTK (see Section 13.1.2) where return address popped from shadow stack always matches
return address popped from normal stack.

We should confirm that coroutine stack swapping is compatible with GNU_PROPERTY_X86_FEATURE_1_SHSTK.

[jaruga (Jun Aruga)]

Thanks for the patch! Let me check it.

0001-Add-.note.gnu.property-sections.patch (3.69 KB)

I tried the 2nd patch on an older master commit 1cb5a669d39fe741388be7ac45fc88c336b097cf , as I couldn't build on the latest master commit due to #18072 . But I still see similar or same result.

$ gcc --version
gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

With -Wl,-z,cet-report=error

$ autoconf
$ CC="gcc -Wl,-z,cet-report=error -fcf-protection" ./configure --enable-shared
$ make
...
vm_call_iseq_optimized.inc updated
generating vmtc.inc
compiling vm.c
compiling vm_backtrace.c
compiling vm_dump.c
compiling vm_sync.c
making trace_point.rbinc
compiling vm_trace.c
assembling coroutine/amd64/Context.S
processing probes in object files
compiling enc/ascii.c
compiling enc/us_ascii.c
compiling enc/unicode.c
compiling enc/utf_8.c
generating enc/trans/newline.c ...
compiling enc/trans/newline.c
compiling ./missing/setproctitle.c
compiling ./missing/strlcat.c
compiling ./missing/strlcpy.c
compiling addr2line.c
compiling dmyenc.c
linking miniruby
/bin/ld: probes.o: error: missing IBT and SHSTK properties
collect2: error: ld returned 1 exit status
make: *** [Makefile:277: miniruby] Error 1

Check by annocheck and readelf.

$ autoconf
$ ./configure --enable-shared
$ make

$ annocheck libruby.so.3.1.0
annocheck: Version 9.79.
Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now 
Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage 
Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information 
Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found 
Hardened: Rerun annocheck with --verbose to see more information on the tests.

$ readelf -S --wide libruby.so.3.1.0
  => No .note.gnu.property section.

[jaruga (Jun Aruga)]

• Backport changed from 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN to 2.6: UNKNOWN, 2.7: REQUIRED, 3.0: REQUIRED

[ioquatix (Samuel Williams)]

I investigated this issue today.

It doesn't appear to just be a matter of adding a few properties, this actually involves correctly implementing an intel-specific shadow stack.

Assembly coroutine backend and x86 CET support (in QEMU): https://lists.sr.ht/~philmd/qemu/patches/4691

My understanding is we'd need to implement the shadow stack handling in the coroutine code. Or maybe we can just specify that it's not supported and that's also okay?

I think in the future, it's more likely that C compilers will provide native coroutine functions - in this situation maybe we can just use those instead of our native implementations and this problem will go away.

(1) Can we fix this issue without introducing shadow stacks?
(2) Is that sufficient to pass the tests above?

By the way, the error shows an issue with probes.o and while I see the issue also exists for the coroutine implementation, the fix provided by @nobu (Nobuyoshi Nakada) at least hides the error - whether the actual shadow stack is working is anyone's guess - actual programs may just crash on coroutine swap.

See this branch with Nobu's patch: https://github.com/ruby/ruby/pull/5895

From https://www.phoronix.com/scan.php?page=news_item&px=Intel-CET-IBT-SHSTK-glibc:

Landing yesterday in Glibc for Intel's Control-flow Enforcement Technology (CET) were the instructions for Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).

These Intel CET bits for the GNU C Library amount to a fair amount of code being added. The commit message explains some of the CET steps taken. The Control-flow Enforcement Technology behavior can be changed for SHSTK/IBT at run-time through the "GLIBC_TUNABLES" environment variable.

CET aims to fend off return-oriented programming (ROP) and call-jump-oriented programming (COP/JOP) attacks via a shadow stack to keep track of expected return addresses and to raise faults when the return addresses don't match up with what's found in the shadow stack. The Indirect branch tracking is for the stopping of jump/call oriented attacks. CET has yet to appear in released Intel processors while this security feature has already been squared away in the GCC compiler and other areas of the software stack and is now getting setup within Glibc.

which links to https://sourceware.org/git/?p=glibc.git;a=commit;h=f753fa7dea3367bd3eb7b543103ff8cda182a3fa which has a lot more details.

[jaruga (Jun Aruga)]

@ioquatix (Samuel Williams) Thank you for your investigation! I prepared make test-annocheck command for your convenient, and for us to test this case on CI. You can pick up my commit from https://github.com/ruby/ruby/pull/5900 .

[jaruga (Jun Aruga)]

• File config-pie.log config-pie.log added

I am testing Samuel's PR: https://github.com/ruby/ruby/pull/5895.

To pass the security tool, annocheck with the PR, I am trying to build with minimal compiler flags.

I am using the following gcc on my local.

$ gcc --version
gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The following command works. The number of failures by annobin is reduced from 5 to 2 by the flags.

$ ./configure --enable-shared --with-gcc="gcc -fcf-protection -Wl,-z,now"

But the following command doesn't work with -Wl,-pie.

$ ./configure --enable-shared --with-gcc="gcc -fcf-protection -Wl,-z,now -Wl,-pie"
checking for ruby... /bin/ruby
tool/config.guess already exists
tool/config.sub already exists
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for ld... ld
checking for gcc-ar... gcc-ar
checking for g++... g++
checking for gcc-nm... gcc-nm
checking for gcc-ranlib... gcc-ranlib
checking for gcc... gcc -fcf-protection -Wl,-z,now -Wl,-pie
checking whether the C compiler works... no
configure: error: in `/home/jaruga/git/ruby/ruby':
configure: error: C compiler cannot create executables
See `config.log' for more details

I would upload the config.log (config-pie.log). Do you know what's wrong?

[jaruga (Jun Aruga)]

@ioquatix (Samuel Williams) My PR above was merged. Now after you rebase your PR on the latest master branch, you can test your PR with make test-annocheck on your local or test with CI with the following modification. Thanks.

$ git diff
diff --git a/.github/workflows/compilers.yml b/.github/workflows/compilers.yml
index ddb53cadb8..8ae6e40b9e 100644
--- a/.github/workflows/compilers.yml
+++ b/.github/workflows/compilers.yml
@@ -87,7 +87,7 @@ jobs:
               # https://bugs.ruby-lang.org/issues/18061
               # https://sourceware.org/annobin/annobin.html/Test-pie.html
               # https://sourceware.org/annobin/annobin.html/Test-notes.html
-              TEST_ANNOCHECK_OPTS: "--skip-pie --skip-notes"
+              TEST_ANNOCHECK_OPTS: "--skip-pie"
             check: true
           - { key: default_cc, name: clang-15,  value: clang-15,  container: clang-15 }
           - { key: default_cc, name: clang-14,  value: clang-14,  container: clang-14 }

[jaruga (Jun Aruga)]

@nobu (Nobuyoshi Nakada) what is your intent for this commit, https://github.com/ruby/ruby/commit/a7577dbfd3ea53cccf7aaf94208069784ad17791 ? What issue did you see?

Before the commit, you see the annocheck was executed in make test-annocheck using Fedora container in Ubuntu focal container on gcc-11 annocheck case on Ubuntu host in .github/workflows/compilers.yml. The log is here. But after the commit, the annocheck is not executed any more in the make test-annocheck in the gcc-11 annocheck case on the CI any more. The log is here.

[jaruga (Jun Aruga)]

On the current latest master f2dc97294086c31604a45ee350c4fa5d51762f45, here is the result on my Fedora 35. The Fedora container and annocheck are not executed.

$ git clean -fdx
$ ./autogen.sh
$ ./configure --enable-shared --with-gcc="gcc -fcf-protection -Wl,-z,now"
$ make
$ TEST_ANNOCHECK_OPTS="--skip-pie --skip-notes" make test-annocheck
make: Nothing to be done for 'test-annocheck'.

Here is the test on the previous commit of the commit a7577dbfd3ea53cccf7aaf94208069784ad17791 above. The Fedora container and annocheck can be executed multiple times after make.

$ git clean -fdx
$ git checkout 641c3830df8177b38fcfca33d3a0ece8a2c1e7d2
$ ./autogen.sh
$ ./configure --enable-shared --with-gcc="gcc -fcf-protection -Wl,-z,now"
$ make

$ TEST_ANNOCHECK_OPTS="--skip-pie --skip-notes" make test-annocheck
...
+ /bin/docker run --rm -t -v /home/jaruga/git/ruby/ruby:/work ruby-fedora-annocheck annocheck --verbose --skip-pie --skip-notes ruby
...
Hardened: ruby: Overall: PASS.

$ TEST_ANNOCHECK_OPTS="--skip-pie --skip-notes" make test-annocheck
...
+ /bin/docker run --rm -t -v /home/jaruga/git/ruby/ruby:/work ruby-fedora-annocheck annocheck --verbose --skip-pie --skip-notes ruby
...
Hardened: ruby: Overall: PASS.

[nobu (Nobuyoshi Nakada)]

jaruga (Jun Aruga) wrote in #note-15:

@nobu (Nobuyoshi Nakada) what is your intent for this commit, https://github.com/ruby/ruby/commit/a7577dbfd3ea53cccf7aaf94208069784ad17791 ? What issue did you see?

TEST_RUNNABLE is a flag for native/cross compiling.
Since annocheck works on Fedora Docker image, it is unrelated to the host OS, and it may work even when cross compiling if Docker works.

[jaruga (Jun Aruga)]

nobu (Nobuyoshi Nakada) wrote in #note-17:

jaruga (Jun Aruga) wrote in #note-15:

@nobu (Nobuyoshi Nakada) what is your intent for this commit, https://github.com/ruby/ruby/commit/a7577dbfd3ea53cccf7aaf94208069784ad17791 ? What issue did you see?

TEST_RUNNABLE is a flag for native/cross compiling.
Since annocheck works on Fedora Docker image, it is unrelated to the host OS, and it may work even when cross compiling if Docker works.

I see. Thanks for explaining it, and thanks for fixing the issue on the commit https://github.com/ruby/ruby/commit/2411f0ad8cf608d6aae221ce4d803b5c5271795a . I confirmed it works after the commit on the CI (the log is here) and my local Fedora. But as you said, it may work on Windows, Mac, *BSD too where Docker is installed. If it works there, maybe the current target linux-test-annocheck is not enough.

[nobu (Nobuyoshi Nakada)]

jaruga (Jun Aruga) wrote in #note-18:

But as you said, it may work on Windows, Mac, *BSD too where Docker is installed. If it works there, maybe the current target linux-test-annocheck is not enough.

Isn't Docker always Linux, regardless hosts?

[jaruga (Jun Aruga)]

Isn't Docker always Linux, regardless hosts?

Yes, Docker where annocheck is running in is always Linux (= docker.io/fedora:latest container). But the target_os (= "linux" on host os: Fedora and Ubuntu) in common.mk is maybe not "linux" on host: Windows, Mac and *BSD, right? I will try to test make annocheck on Mac.

https://github.com/ruby/ruby/blob/11af23ee923858fbcec18f2d1365296fdfe24dc2/common.mk#L1450-L1453

test-annocheck: $(target_os)-test-annocheck
linux-test-annocheck: $(PROGRAM)
	$(tooldir)/test-annocheck.sh $(PROGRAM)
$(target_os)-test-annocheck: PHONY

[jaruga (Jun Aruga)]

I will try to test make annocheck on Mac.

I sent the PR here. It's working in progress.
https://github.com/ruby/ruby/pull/5916

[jaruga (Jun Aruga)]

Here is a minimal reproducer to see the property-note test failure. The ./configure --enable-shared --with-gcc="gcc -fcf-protection -Wl,-z,now" above was wrong. The -Wl,-z,now should not be in `--with-gcc="..". I checked the reproducer on my Fedora 36.

$ ./autogen.sh
$ ./configure --enable-shared --with-gcc="gcc -fcf-protection" LDFLAGS=-Wl,-z,now
$ make
$ make test-annocheck
...
Hardened: ruby: MAYB: test: notes because not all of the .text section is covered by notes
...
Hardened: ruby: FAIL: pie test because not built with '-Wl,-pie'
...
Hardened: ruby: Overall: FAIL.

$ TEST_ANNOCHECK_OPTS="--skip-pie --skip-notes" make test-annocheck
...
Hardened: ruby: Overall: PASS.

[jaruga (Jun Aruga)]

@ioquatix (Samuel Williams) (Samuel Williams) @nobu (Nobuyoshi Nakada), possibly Samuel already fixed the property-note issue failed by annocheck correctly on the master branch. Sorry for my mistake.

I tested Ruby on the (relatively) latest master branch 78425d7e74887b57ee15e6b8933bd3878db6a888. And when I built with all the build flags used to build Fedora's Ruby RPM package, the annocheck passed for the ruby binary.

This repository is my experiment, and the details: https://github.com/junaruga/ruby-annocheck-test/
I built with the flags below in the build_with_fedora_build_flags.sh.

https://github.com/junaruga/ruby-annocheck-test/blob/main/build_with_fedora_build_flags.sh

CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' \
CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' \
LDFLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 ' \
./configure \
    --enable-shared \
    --enable-mkmf-verbose 2>&1 | tee configure.log

For the gcc -specs=file options I used, I put the files in the https://github.com/junaruga/ruby-annocheck-test/tree/main/gcc_specs.
I will try to understand what gcc flags fixed the 2 failures (https://bugs.ruby-lang.org/issues/18061#note-22), then update the CI .github/workflows/compilers.yml annocheck case with the minimal flags.

[ioquatix (Samuel Williams)]

While it was passing, it's now failing:

Previously Passing

https://github.com/ruby/ruby/runs/6956677552?check_suite_focus=true#step:20:54

================================================================================
 Package                 Architecture Version               Repository     Size
================================================================================
Installing:
 annobin-annocheck       x86_64       10.73-1.fc36          updates       165 k
Installing dependencies:
 annobin-docs            noarch       10.73-1.fc36          updates        90 k
 cpio                    x86_64       2.13-12.fc36          fedora        270 k
Transaction Summary
================================================================================

... snip ...

annocheck: Version 10.73.
Hardened: ruby: PASS: optimization test 
Hardened: ruby: PASS: pic test 
Hardened: ruby: PASS: stack-prot test 
Hardened: ruby: PASS: cf-protection test because correct flags found in .note.gnu.property note 
Hardened: ruby: PASS: writable-got test 
Hardened: ruby: PASS: dynamic-segment test 
Hardened: ruby: PASS: bind-now test 
Hardened: ruby: PASS: run-path test because the DT_RUNPATH dynamic tag is present and correct 
Hardened: ruby: PASS: entry test 
Hardened: ruby: PASS: gnu-stack test because stack segment exists with the correct permissions 
Hardened: ruby: PASS: gnu-relro test 
Hardened: ruby: skip: branch-protection test because not an AArch64 binary 
Hardened: ruby: skip: dynamic-tags test because AArch64 specific 
Hardened: ruby: skip: fortify test because no C/C++ compiled code found 
Hardened: ruby: skip: glibcxx-assertions test because source language not C++ 
Hardened: ruby: skip: go-revision test because no GO compiled code found 
Hardened: ruby: PASS: instrumentation test 
Hardened: ruby: skip: lto test because not compiled C/C++ code 
Hardened: ruby: skip: only-go test because no GO compiled code found 
Hardened: ruby: PASS: production test 
Hardened: ruby: PASS: property-note test because CET enabled property note found 
Hardened: ruby: PASS: rwx-seg test 
Hardened: ruby: PASS: short-enums test 
Hardened: ruby: skip: stack-clash test because no C/C++ compiled code found 
Hardened: ruby: skip: stack-realign test because not a 32-bit i686 executable 
Hardened: ruby: PASS: textrel test 
Hardened: ruby: PASS: threads test 
Hardened: ruby: PASS: unicode test 
Hardened: ruby: skip: warnings test because no C/C++ compiled code found 
Hardened: ruby: Overall: PASS.

Now Failing

https://github.com/ruby/ruby/runs/6959694909?check_suite_focus=true#step:20:53

================================================================================
 Package                 Architecture Version               Repository     Size
================================================================================
Installing:
 annobin-annocheck       x86_64       10.76-1.fc36          updates       167 k
Installing dependencies:
 annobin-docs            noarch       10.76-1.fc36          updates        91 k
 cpio                    x86_64       2.13-12.fc36          fedora        270 k
Transaction Summary
================================================================================

... snip ...

annocheck: Version 10.76.
Hardened: ruby: PASS: optimization test 
Hardened: ruby: PASS: pic test 
Hardened: ruby: PASS: stack-prot test 
Hardened: ruby: PASS: cf-protection test because correct flags found in .note.gnu.property note 
Hardened: ruby: PASS: writable-got test 
Hardened: ruby: PASS: dynamic-segment test 
Hardened: ruby: PASS: bind-now test 
Hardened: ruby: PASS: run-path test because the DT_RUNPATH dynamic tag is present and correct 
Hardened: ruby: PASS: entry test 
Hardened: ruby: PASS: gnu-stack test because stack segment exists with the correct permissions 
Hardened: ruby: PASS: gnu-relro test 
Hardened: ruby: MAYB: test: gaps because no notes found 
Hardened: ruby: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-gaps.html
Hardened: ruby: skip: branch-protection test because not an AArch64 binary 
Hardened: ruby: skip: dynamic-tags test because AArch64 specific 
Hardened: ruby: skip: fortify test because no C/C++ compiled code found 
Hardened: ruby: skip: glibcxx-assertions test because source language not C++ 
Hardened: ruby: skip: go-revision test because no GO compiled code found 
Hardened: ruby: PASS: instrumentation test 
Hardened: ruby: skip: lto test because not compiled C/C++ code 
Hardened: ruby: skip: only-go test because no GO compiled code found 
Hardened: ruby: PASS: production test 
Hardened: ruby: PASS: property-note test because CET enabled property note found 
Hardened: ruby: PASS: rwx-seg test 
Hardened: ruby: PASS: short-enums test 
Hardened: ruby: skip: stack-clash test because no C/C++ compiled code found 
Hardened: ruby: skip: stack-realign test because not a 32-bit i686 executable 
Hardened: ruby: PASS: textrel test 
Hardened: ruby: PASS: threads test 
Hardened: ruby: PASS: unicode test 
Hardened: ruby: skip: warnings test because no C/C++ compiled code found 
Hardened: ruby: Overall: FAIL (due to MAYB results).

I haven't touched the implementation except for my PR. I don't think this is a matter of just adding notes, I think the coroutine implementation needs to maintain a shadow stack for the purpose of return value tracking etc.

I don't know enough about the relevant technologies to implement this right now. We can't just enable the flags without the corresponding implementation - with CET enabled, I'd imagine it would crash.

I did implement ASAN which has a similar shadow stack. With that in mind, it might be possible to follow a similar implementation. But this is not something I'm very familiar with.

[ioquatix (Samuel Williams)]

I started messing around with the implementation again.

But my computer is too old, it looks like I need 11th gen Intel CPU to test CET :(

Is anyone able to help test the PR if I update it?

[ioquatix (Samuel Williams)]

In my latest PR, I try to detect control-flow protections:

   * with thread:         pthread
   * with coroutine:      amd64
   * with c-f protection: no

(not supported on my CPU).

Then, I've eye-balled an implementation. Not sure if it works, maybe it will in GH Actions?

[jaruga (Jun Aruga)]

ioquatix (Samuel Williams) wrote in #note-24:

While it was passing, it's now failing:

Thanks for the report. I will take a look and fix the CI. Maybe it comes from the annocheck version update from 10.73 to 10.76. My guess is that the annocheck changed something between the versions. Maybe the current workaround is to add another skip option to TEST_ANNOCHECK_OPTS: "--skip-pie --skip-notes" in the .github/workflows/compilers.yml.

https://github.com/ruby/ruby/runs/6956677552?check_suite_focus=true#step:20:108

annocheck: Version 10.73.

https://github.com/ruby/ruby/runs/6959694909?check_suite_focus=true#step:20:109

annocheck: Version 10.76.

Comparing the 2 commits, I don't see any issues causing this failure.

$ git diff aeab4058784c86df47a455ffdb08714b337d0209..150bd6e2427a8a6178aa5f8ebc39ce6e59601e1f

[ioquatix (Samuel Williams)]

https://lwn.net/Articles/885220/

I was studying this and the related LKML discussion. I don't think this problem is solved any time soon. I don't think coroutine support using ret is compatible yet, unless some how we can acess wrss instruction. I need to do some more investigation and probably get hold of a compatible CPU.

[jaruga (Jun Aruga)]

We opened the 2 PRs for that now :) @mame (Yusuke Endoh) You can merge your PR. I will add my PR later with some comments.
https://github.com/ruby/ruby/pull/6043
https://github.com/ruby/ruby/pull/6044

[jaruga (Jun Aruga)]

jaruga (Jun Aruga) wrote in #note-29:

We opened the 2 PRs for that now :) @mame (Yusuke Endoh) You can merge your PR. I will add my PR later with some comments.
https://github.com/ruby/ruby/pull/6043
https://github.com/ruby/ruby/pull/6044

The PR was merged to the master branch.
https://github.com/ruby/ruby/commit/f8b6d4f4d1b0eb0b3220fc83f157bd33ddcc5d4e

[jaruga (Jun Aruga)]

Maybe I found a way to fix annocheck gaps and notes tests on Ubuntu focal on CI.
Here is the PR: https://github.com/ruby/ruby/pull/6045.

[jaruga (Jun Aruga)]

ioquatix (Samuel Williams) wrote in #note-25:

I started messing around with the implementation again.

But my computer is too old, it looks like I need 11th gen Intel CPU to test CET :(

Is anyone able to help test the PR if I update it?

I have my own Framework Laptop first generation with the 11th gen Intel CPU (11th Gen Intel i7-1165G7) with Fedora 36 and gcc 12. I can help to test it. :)

[jaruga (Jun Aruga)]

I got useful info from the helpful annocheck upstream maintainer to fix the annocheck gaps and pie tests.

Bug 29269 - annocheck: gaps test: question for the flag: -Wa,--generate-missing-build-notes=yes .
https://sourceware.org/bugzilla/show_bug.cgi?id=29269#c2

As a reference to understand the content, here are the specs files used by the gcc -specs=file options. I want to see that we add minimal flags without using specs files into our annocheck CI case.

$ cat redhat-annobin-cc1 
*cc1_options:
+ %{!-fno-use-annobin:%{!iplugindir*:%:find-plugindir()} -fplugin=annobin}

$ cat redhat-hardened-cc1 
*cc1_options:
+ %{!r:%{!fpie:%{!fPIE:%{!fpic:%{!fPIC:%{!fno-pic:-fPIE}}}}}}

$ cat redhat-hardened-ld 
*self_spec:
+ %{!static:%{!shared:%{!r:-pie}}}

[ioquatix (Samuel Williams)]

• Related to Bug #20029: coroutine/arm64/Context.S does not support PAC/BTI added