Bug #18373: Bundled gem (RBS, debug) extensions are not properly built - Ruby master - Ruby Issue Tracking System

Actions

Updated by vo.x (Vit Ondruch) almost 3 years ago

I have downgraded RBS to 1.6.2, but the same issue suffers (not surprisingly) debug:

Building native extensions. This could take a while...
/builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:95:in `run': ERROR: Failed to build gem native extension. (Gem::Ext::BuildError)

    current directory: /builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/share/ruby/gems/gems/debug-1.3.4/ext/debug
/builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/bin/ruby --disable\\=gems -I/builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/lib64/ruby -I /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib -r ./siteconf20211201-24843-jbuqiz.rb extconf.rb
creating Makefile

current directory: /builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/share/ruby/gems/gems/debug-1.3.4/ext/debug
make DESTDIR\\= clean
make[1]: Entering directory '/builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/share/ruby/gems/gems/debug-1.3.4/ext/debug'
rm -f 
rm -fr debug.so false *.o  *.bak mkmf.log .*.time
make[1]: Leaving directory '/builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/share/ruby/gems/gems/debug-1.3.4/ext/debug'

current directory: /builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/share/ruby/gems/gems/debug-1.3.4/ext/debug
make DESTDIR\\=
make[1]: Entering directory '/builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/share/ruby/gems/gems/debug-1.3.4/ext/debug'
make[1]: *** No rule to make target '/include/x86_64-linux/ruby/config.h', needed by 'debug.o'.  Stop.
make[1]: Leaving directory '/builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/share/ruby/gems/gems/debug-1.3.4/ext/debug'

make failed, exit code 2

Gem files will remain installed in /builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/share/ruby/gems/gems/debug-1.3.4 for inspection.
Results logged to /builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211201gita84dc9d80d.fc36.x86_64/usr/share/ruby/gems/extensions/x86_64-linux/3.1.0/debug-1.3.4/gem_make.out
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:44:in `block in make'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:36:in `each'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:36:in `make'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/ext_conf_builder.rb:63:in `block in build'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/tempfile.rb:317:in `open'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/ext_conf_builder.rb:26:in `build'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:161:in `build_extension'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:195:in `block in build_extensions'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:192:in `each'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:192:in `build_extensions'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/installer.rb:837:in `build_extensions'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/installer.rb:318:in `install'
	from ./tool/rbinstall.rb:895:in `block in install'
	from ./tool/rbinstall.rb:719:in `no_write'
	from ./tool/rbinstall.rb:895:in `install'
	from ./tool/rbinstall.rb:1062:in `block (2 levels) in <main>'
	from ./tool/rbinstall.rb:1037:in `foreach'
	from ./tool/rbinstall.rb:1037:in `block in <main>'
	from ./tool/rbinstall.rb:1121:in `block in <main>'
	from ./tool/rbinstall.rb:1118:in `each'
	from ./tool/rbinstall.rb:1118:in `<main>'
/builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:95:in `run': make failed, exit code 2 (Gem::InstallError)
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:44:in `block in make'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:36:in `each'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:36:in `make'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/ext_conf_builder.rb:63:in `block in build'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/tempfile.rb:317:in `open'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/ext_conf_builder.rb:26:in `build'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:161:in `build_extension'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:195:in `block in build_extensions'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:192:in `each'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/ext/builder.rb:192:in `build_extensions'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/installer.rb:837:in `build_extensions'
	from /builddir/build/BUILD/ruby-3.1.0-a84dc9d80d/lib/rubygems/installer.rb:318:in `install'
	from ./tool/rbinstall.rb:895:in `block in install'
	from ./tool/rbinstall.rb:719:in `no_write'
	from ./tool/rbinstall.rb:895:in `install'
	from ./tool/rbinstall.rb:1062:in `block (2 levels) in <main>'
	from ./tool/rbinstall.rb:1037:in `foreach'
	from ./tool/rbinstall.rb:1037:in `block in <main>'
	from ./tool/rbinstall.rb:1121:in `block in <main>'
	from ./tool/rbinstall.rb:1118:in `each'
	from ./tool/rbinstall.rb:1118:in `<main>'
make: *** [uncommon.mk:383: do-install-all] Error 1

Actions

Copy link

#2 [ruby-core:106421]

Updated by xtkoba (Tee KOBAYASHI) almost 3 years ago

FYI, in my environment ruby-3.1.0-preview1 can be cross-installed without error for aarch64-linux-android target (with build host being x86_64-linux-gnu).

Some observation:

$ grep 'config\.h' /var/tmp/build/arm64-v8a/lib/ruby/gems/3.1.0/gems/rbs-1.7.0.beta.5/ext/rbs_extension/Makefile
ruby_headers = $(hdrdir)/ruby.h $(hdrdir)/ruby/backward.h $(hdrdir)/ruby/ruby.h $(hdrdir)/ruby/defines.h $(hdrdir)/ruby/missing.h $(hdrdir)/ruby/intern.h $(hdrdir)/ruby/st.h $(hdrdir)/ruby/subst.h $(arch_hdrdir)/ruby/config.h

$ grep '^arch_hdrdir' /var/tmp/build/arm64-v8a/lib/ruby/gems/3.1.0/gems/rbs-1.7.0.beta.5/ext/rbs_extension/Makefile
arch_hdrdir = /var/tmp/ruby-3.1.0-preview1/.build-android-aarch64/.ext/include/aarch64-linux-android

So I guess in your environment arch_hdrdir in Makefile for gem extension is incorrectly defined as /include/x86_64-linux, though I don't know why.

Actions

Copy link

#3 [ruby-core:106426]

Updated by ko1 (Koichi Sasada) almost 3 years ago

What make command do you use?

Actions

Copy link

#4 [ruby-core:106461]

Updated by jaruga (Jun Aruga) almost 3 years ago

Can you provide the full log of the build.log showing the build error as an attachment of this ticket or somewhere?

Actions

Copy link

#5 [ruby-core:106539]

Updated by vo.x (Vit Ondruch) almost 3 years ago

File Makefile Makefile added

xtkoba (Tee KOBAYASHI) wrote in #note-2:

$ grep '^arch_hdrdir' /var/tmp/build/arm64-v8a/lib/ruby/gems/3.1.0/gems/rbs-1.7.0.beta.5/ext/rbs_extension/Makefile
arch_hdrdir = /var/tmp/ruby-3.1.0-preview1/.build-android-aarch64/.ext/include/aarch64-linux-android
So I guess in your environment `arch_hdrdir` in `Makefile` for gem extension is incorrectly defined as `/include/x86_64-linux`, though I don't know why.

This is good tip:

$ grep '^arch_hdrdir' /builddir/build/BUILDROOT/ruby-3.1.0-0.1.20211207git26303c31f0.fc36.x86_64/usr/share/ruby/gems/gems/rbs-1.8.0/ext/rbs_extension/Makefile 
arch_hdrdir = $(extout)/include/$(arch)

And this must come from here. Obviously, the condition is not met, because we are using --with-rubyhdrdir=/usr/include --with-rubyarchhdrdir=/usr/include configuration options on Fedora:

$ ./miniruby -r./rbconfig -e 'puts RbConfig::CONFIG["rubyhdrdir"]'
/usr/include

This is not unreasonable I believe. However, I am not sure what this branch is supposed to do. On top of that, the $(extout) is not set. I think that ideally, we would like to pass the first condition and the configuration options should be prefixed with $(DESTDIR).

Actions

Copy link

#6 [ruby-core:106633]

Updated by darix (Marcus Rückert) almost 3 years ago

this only breaks when you run "make install DESTDIR=/some/path"

but it works nicely without. so this is a bug in how DESTDIR is passed
to the rubygems extensions build.

darix

--
Always remember:
Never accept the world as it appears to be.
Dare to see it for what it could be.
The world can always use more heroes.

Actions

Copy link

#7 [ruby-core:106714]

Updated by vo.x (Vit Ondruch) almost 3 years ago

I am not sure what to really blame. The place where it goes quite wrong is actually mkmf, somewhere around here:

  if not $extmk and File.exist?(($hdrdir = RbConfig::CONFIG["rubyhdrdir"]) + "/ruby/ruby.h")
    $topdir = $hdrdir
    $top_srcdir = $hdrdir
    $arch_hdrdir = RbConfig::CONFIG["rubyarchhdrdir"]
  elsif File.exist?(($hdrdir = ($top_srcdir ||= topdir) + "/include")  + "/ruby.h")
    $topdir ||= RbConfig::CONFIG["topdir"]
    $arch_hdrdir = "$(extout)/include/$(arch)"
  else

The first branch is used for regular cases when Ruby is already installed. The second branch is used for building extensions during build of Ruby.

Unfortunately, installing gems and building their extensions is non of these. So there should be probably created some middle path. Maybe if the mkmf could be convinced that the extension is nothing different to building extensions which are part of Ruby ...

But I think that it might make difference, if there is already installed Ruby on the system, which is not my case.

Actions

Copy link

#8 [ruby-core:106718]

Updated by vo.x (Vit Ondruch) almost 3 years ago

I was able to use this little hack to compile the extensions:

find .bundle -name extconf.rb -exec \
  sed -i \
    -e '/create_makefile/i \$arch_hdrdir = "$(hdrdir)/../.ext/include/$(arch)"' \
    -e '/create_makefile/i \$DLDFLAGS << " -L#{$top_srcdir}"' {} \;

Actions

Copy link

#9

Updated by vo.x (Vit Ondruch) almost 3 years ago

Subject changed from RBS build failure: '/include/x86_64-linux/ruby/config.h', needed by 'constants.o'. to Bundle gem (RBS, debug) extension are not properly built

Actions

Copy link

#10

Updated by vo.x (Vit Ondruch) almost 3 years ago

Subject changed from Bundle gem (RBS, debug) extension are not properly built to Bundled gem (RBS, debug) extensions are not properly built

Actions

Copy link

#11 [ruby-core:106723]

Updated by vo.x (Vit Ondruch) almost 3 years ago

BTW, this causes also different issues as far as I can tell:

https://github.com/ruby/ruby/pull/5206

Actions

Copy link

#12 [ruby-core:106740]

Updated by mame (Yusuke Endoh) almost 3 years ago

Please answer to https://bugs.ruby-lang.org/issues/18373#note-3. I couldn't reproduce the issue by the following commands

$ cd /tmp
$ git clone https://github.com/ruby/ruby.git
$ cd ruby
$ ./autogen.sh
$ ./configure
$ make
$ make install DESTDIR=/tmp/local/

A small and self-contained procedure to reproduce this issue is needed to make progress.

Actions

Copy link

#13 [ruby-core:106743]

Updated by vo.x (Vit Ondruch) almost 3 years ago

mame (Yusuke Endoh) wrote in #note-12:

Please answer to https://bugs.ruby-lang.org/issues/18373#note-3. I couldn't reproduce the issue by the following commands
$ cd /tmp
$ git clone https://github.com/ruby/ruby.git

This ^^ is one issue. The starting point should be something like:

curl -OL https://cache.ruby-lang.org/pub/ruby/snapshot/snapshot-master.tar.xz

or https://cache.ruby-lang.org/pub/ruby/3.1/ruby-3.1.0-preview1.tar.xz and make sure no Ruby is installed on the system. I think this is the main issue and it does not really matter on the specific make command. But I'll try to come up with simple reproducer.

Actions

Copy link

#14 [ruby-core:106747]

Updated by vo.x (Vit Ondruch) almost 3 years ago

So the basic scenarios seem to work as expected (unexpectedly for me, sorry). But this is the failing scenario, where I am using expanded tarball:

$ history
    1  cd /builddir/build/BUILD/ruby-3.1.0-74b58dd690/
    2  autoconf 
    3  ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info
    4  make
    5  make install DESTDIR=/tmp/local/
    6  history

$ ruby -v
sh: ruby: command not found

Please note that these ./configure options are standard Fedora configuration options and they make the difference. With plain ./configure, the example would pass just fine. Also, as @darix (Marcus Rückert) pointed out, the plain make install also succeeds. But that is probably not really surprising, because in that case Ruby can already by found on the expected location prior gems are installed.

Actions

Copy link

#15 [ruby-core:106753]

Updated by ko1 (Koichi Sasada) almost 3 years ago

Note that we can reproduce the issue with --includedir configure option like that: ./configure --includedir=/usr/include && make -j && make install DESTDIR=/tmp/foo also on Ubuntu with git repo.
@nobu (Nobuyoshi Nakada) is checking.

Actions

Copy link

#16

Updated by nobu (Nobuyoshi Nakada) almost 3 years ago

Related to Bug #7860: Passing --libdir to ./configure causes Gem.ruby to point to an incorrect ruby interpreter path added
Related to Bug #9160: configureに--with-rubylibprefixを指定するとrbconfigのprefixが空文字になる added

Actions

Copy link

#17 [ruby-core:106754]

Updated by nobu (Nobuyoshi Nakada) almost 3 years ago

Does this fix it?
https://github.com/ruby/ruby/pull/5318

Actions

Copy link

#18 [ruby-core:106757]

Updated by vo.x (Vit Ondruch) almost 3 years ago

File Makefile Makefile added

nobu (Nobuyoshi Nakada) wrote in #note-17:

Does this fix it?

No, unfortunately. Please see the attached Makefile. I think that the culprit is still the arch_hdrdir = $(extout)/include/$(arch)

Actions

Copy link

#19 [ruby-core:106760]

Updated by nobu (Nobuyoshi Nakada) almost 3 years ago

vo.x (Vit Ondruch) wrote in #note-18:

No, unfortunately. Please see the attached Makefile. I think that the culprit is still the arch_hdrdir = $(extout)/include/$(arch)

Yes, it's the problem.
Could you show rbconfig.rb file too?

Actions

Copy link

#20

Updated by vo.x (Vit Ondruch) almost 3 years ago

File rbconfig.rb rbconfig.rb added

Actions

Copy link

#21 [ruby-core:106762]

Updated by vo.x (Vit Ondruch) almost 3 years ago

nobu (Nobuyoshi Nakada) wrote in #note-19:

Could you show rbconfig.rb file too?

Done

Actions

Copy link

#22

Updated by nobu (Nobuyoshi Nakada) almost 3 years ago

Status changed from Open to Closed

Applied in changeset git|c87c027f18c005460746a74c07cd80ee355b16e4.

configure.in: unexpand exec_prefix in includedir

Replace exec_prefix in includedir as well as bindir, libdir, and
so on. [Bug #18373]

Actions

Copy link

#23 [ruby-core:106806]

Updated by nobu (Nobuyoshi Nakada) almost 3 years ago

Status changed from Closed to Open

As the patch seems to fix one situation at least anyway, committed for now.

Actions

Copy link

#24 [ruby-core:106814]

Updated by darix (Marcus Rückert) almost 3 years ago

On Fri, 24 Dec 2021 11:44:21 +0000 (UTC)
"nobu (Nobuyoshi Nakada)" noreply@ruby-lang.org wrote:

Issue #18373 has been updated by nobu (Nobuyoshi Nakada).

Status changed from Closed to Open

As the patch seems to fix one situation at least anyway, committed
for now.

I just tested the latest snapshot (2021-12-25 01:00 AM) and it still
doesn't work in my packaging environment.

Actions

Copy link

#25 [ruby-core:108067]

Updated by vo.x (Vit Ondruch) over 2 years ago

This is even more tricky then I initially thought, at least in the RPM context. The issue that RPM assumes there is "build" step, where everything is build and later "install" step, where everything is move into right location. Unfortunately, the way the files are currently installed mixes the two steps together, because the install steps builds the bundled gem binary extensions after the gem is installed. That in turn causes troubles with debug info stripping (which went through unnoticed for a while due to change in grep and therefore bug in RPM itself).

I starting to think that the right approach would be to build the binary extensions along the other Ruby modules and let the installer to only install the files.

Actions

Copy link

#26 [ruby-core:108070]

Updated by vo.x (Vit Ondruch) over 2 years ago

Ah, now I found this git|bac6e8ca5d8f6bc45984d12ddad55d3d876d4324, so it used to be the case. I'll try to take a look what would the rbinstall.rb need to install also the binary extensions.

Actions

Copy link

#27 [ruby-core:108146]

Updated by vo.x (Vit Ondruch) over 2 years ago

vo.x (Vit Ondruch) wrote in #note-25:

I have put together https://github.com/ruby/ruby/pull/5743 which should address this particular issue.

Actions

Copy link

#28 [ruby-core:108155]

Updated by vo.x (Vit Ondruch) over 2 years ago

Not sure what $installed_list is good for, but the extensions as build currently are very likely not listed there. Just one another flaw of the current design.

Actions

Copy link

#29 [ruby-core:108171]

Updated by vo.x (Vit Ondruch) over 2 years ago

vo.x (Vit Ondruch) wrote in #note-27:

vo.x (Vit Ondruch) wrote in #note-25:

I have put together https://github.com/ruby/ruby/pull/5743 which should address this particular issue.

I have updated the PR ^^ in a way that it seems to completely address my original issue and I am going to use it in Fedora package.

Actions

Copy link

#30

Updated by vo.x (Vit Ondruch) over 2 years ago

Backport changed from 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN to 2.6: DONTNEED, 2.7: DONTNEED, 3.0: DONTNEED, 3.1: REQUIRED

Actions

Copy link

#31 [ruby-core:108232]

Updated by vo.x (Vit Ondruch) over 2 years ago

Status changed from Open to Closed

This seems to be addressed by PR#5774. Thx @nobu (Nobuyoshi Nakada).

For the backport, please note that with this changes, the .gemspec files for expanded gems in tarball are newly located in .bundle/specifications, but this should be just implementation detail.

Actions

Copy link

#32 [ruby-core:108233]

Updated by vo.x (Vit Ondruch) over 2 years ago

Assignee set to nobu (Nobuyoshi Nakada)

Actions

Copy link

#33

Updated by jeremyevans0 (Jeremy Evans) over 2 years ago

Has duplicate Bug #18592: RPATH of bundled native extensions is incorrect added

Actions

Copy link

#34 [ruby-core:108997]

Updated by nagachika (Tomoyuki Chikanaga) over 2 years ago

Backport changed from 2.6: DONTNEED, 2.7: DONTNEED, 3.0: DONTNEED, 3.1: REQUIRED to 2.6: DONTNEED, 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONE

ruby_3_1 d56b2e8820efd40840443ce3a5d16cff32a090c7 merged revision(s) 5c1b76a3a55afeb07116bbd3492303c6b6cd890d.

Makefile (9.3 KB) Makefile		vo.x (Vit Ondruch), 12/07/2021 05:52 PM
Makefile (8.24 KB) Makefile		vo.x (Vit Ondruch), 12/21/2021 11:13 AM
rbconfig.rb (13.8 KB) rbconfig.rb		vo.x (Vit Ondruch), 12/21/2021 02:37 PM

Project

General

Profile

Ruby » Ruby master

Custom queries

Bug #18373

Bundled gem (RBS, debug) extensions are not properly built

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by xtkoba (Tee KOBAYASHI) almost 3 years ago

Updated by ko1 (Koichi Sasada) almost 3 years ago

Updated by jaruga (Jun Aruga) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by darix (Marcus Rückert) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by mame (Yusuke Endoh) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by ko1 (Koichi Sasada) almost 3 years ago

Updated by nobu (Nobuyoshi Nakada) almost 3 years ago

Updated by nobu (Nobuyoshi Nakada) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by nobu (Nobuyoshi Nakada) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by vo.x (Vit Ondruch) almost 3 years ago

Updated by nobu (Nobuyoshi Nakada) almost 3 years ago

Updated by nobu (Nobuyoshi Nakada) almost 3 years ago

Updated by darix (Marcus Rückert) almost 3 years ago

Updated by vo.x (Vit Ondruch) over 2 years ago

Updated by vo.x (Vit Ondruch) over 2 years ago

Updated by vo.x (Vit Ondruch) over 2 years ago

Updated by vo.x (Vit Ondruch) over 2 years ago

Updated by vo.x (Vit Ondruch) over 2 years ago

Updated by vo.x (Vit Ondruch) over 2 years ago

Updated by vo.x (Vit Ondruch) over 2 years ago

Updated by vo.x (Vit Ondruch) over 2 years ago

Updated by jeremyevans0 (Jeremy Evans) over 2 years ago

Updated by nagachika (Tomoyuki Chikanaga) over 2 years ago