Project

General

Profile

Feature #5999

Optimize for faster loading time and win32/file.c refactoring

Added by h.shirosaki (Hiroshi Shirosaki) over 7 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Target version:
[ruby-core:42480]

Description

This was originally discussed at [ruby-core:41799] Best way to separate implementation specific code?.

Ruby startup time (loading time) is slow when loading a lot of libraries such as Rails. Some works were done to improve performance such as #3924, but startup is still not fast. Especially ruby on Windows seems much slower than on Unix like OS.
This work is an effort to improve startup time on Windows.

I have updated the patch with help from Luis Lavena. This patch improves loading time a little.

https://gist.github.com/1605263#file_0001_move_rb_file_load_ok_to_win32_file.c.patch

If load_path and loaded_features are large numbers such as rails app, file_load_ok() is called many times against non-existing path.

On Windows, GetFileAttributes() is faster than open(). And open() performance is affected by AntiVirus software's realtime scan.
I confirmed this with NOD32. Microsoft Security Essentials doesn't affect so badly as NOD32. It seems to depend on kinds of antivirus softwares.

Benchmark:
https://gist.github.com/1605263#gistcomment-75650
https://gist.github.com/1448652

Avoiding to call open() against non-existing path by using GetFileAttributes() seems to improve performance.
I used CreateFile() instead of open() because CreateFile() seems a little faster.

Here are rails startup benchmarks.

On Windows 7:

200ms improvement
https://gist.github.com/1605263#gistcomment-82137

On Windows XP:

800ms improvement with NOD32 OFF
1800ms improvement with NOD32 ON
https://gist.github.com/1605263#gistcomment-75644

This fix would be helpful for windows users. I guess most windows users use antivirus softwares.

I confirmed make, make test are OK. make test-all result was same as before patch at the following revision.
ruby 2.0.0dev (2012-02-09 trunk 34511) [i386-mingw32]

At [ruby-core:41831], Matz agreed with replacing file_load_ok.

This is the first work in win32/file.c. The following works (optimize expand_path etc.) into win32/file.c will be done by Luis.
This patch changed configure and Makefile. I also confirmed make with nmake(mswin32).

Is this acceptable way?

Associated revisions

Revision 4b1e9f0c
Added by shirosaki over 7 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[ruby-core:42480] [Feature #5999]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34849 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 34849
Added by shirosaki over 7 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[ruby-core:42480] [Feature #5999]

Revision 34849
Added by shirosaki over 7 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[ruby-core:42480] [Feature #5999]

Revision 34849
Added by shirosaki over 7 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[ruby-core:42480] [Feature #5999]

Revision 34849
Added by shirosaki over 7 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[ruby-core:42480] [Feature #5999]

Revision 34849
Added by shirosaki over 7 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[ruby-core:42480] [Feature #5999]

Revision 34849
Added by shirosaki over 7 years ago

  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[ruby-core:42480] [Feature #5999]

Revision 2c400078
Added by usa (Usaku NAKAMURA) over 6 years ago

merge revision(s) 34849,34853,34854,34855,34859,34862,35384,35385,36811,36812,36850,36907,36908: [Backport #7174]

    * Makefile.in (PLATFORM_DIR): add a variable for `win32` directory.

    * Makefile.in (clean-platform): add new target.
      It cleans `win32` directory.

    * common.mk (clean): add a dependency for `win32` directory.

    * common.mk (distclean): ditto.

    * common.mk (distclean-platform): add new target.
      It cleans `win32` directory.

    * common.mk ($(PLATFORM_D)): add new target to make `win32` directory.

    * common.mk (win32/win32.$(OBJEXT)): move win32.o into `win32`
      directory.

    * common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

    * configure.in: move win32.o into `win32` directory and add
      win32/file.o to MISSING.

    * file.c (file_load_ok, rb_file_load_ok): replace static
      file_load_ok() with public rb_file_load_ok().
      It's to link Windows implementation in win32/file.c.

    * file.c (rb_find_file_ext_safe): ditto.

    * file.c (rb_find_file_safe): ditto.

    * win32/file.c (rb_file_load_ok): new file. Add Windows specific
      optimized implementation of rb_file_load_ok(). We created a
      separated file to avoid too many #ifdef macro which is unreadable.

    * win32/Makefile.sub (PLATFORM_DIR): add a variable for `win32`
      directory.

    * win32/Makefile.sub (MISSING): move win32.obj into `win32`
      directory and add win32/file.obj to MISSING.

    * win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
      It's because miniruby doesn't exist when making `win32` directory.

    * win32/Makefile.sub (clean-platform): add new target to clean `win32`
      directory.

    * win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
      win32/file.c to build properly.

    * win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
     `win32` directory.
      Patch created with Luis Lavena.
      [ruby-core:42480] [Feature #5999]

    * win32/Makefile.sub (MAKEDIRS): use mkdir of cmd.exe instead of ruby.
      [Bug #6103] [ruby-core:43012]

    * win32/README.win32: added a notice about command extension of cmd.exe.

    * win32/makedirs.bat: new command to make intermediate
      directories, and not to report any errors if the directory
      already exists.

    * win32/Makefile.sub (MAKEDIRS): enable command extensions.

    * win32/file.c (INVALID_FILE_ATTRIBUTES): define for old SDK.

    * configure.in (mingw): add shlwapi to the list of dependency
      libs for Windows.

    * win32/Makefile.sub (EXTSOLIBS): ditto.

    * internal.h: declare internal functions rb_w32_init_file,
      rb_file_expand_path_internal and rb_file_expand_path_fast.

    * file.c (Init_File): invoke Windows initialization rb_w32_init_file

    * win32/file.c (rb_file_load_path_internal): new function.
      Windows-specific implementation that replaces file_expand_path.
      [Bug #6836][ruby-core:46996]

    * win32/file.c (rb_w32_init_file): new function. Initialize codepage
      cache for faster conversion encodings lookup.

    * file.c (file_expand_path): rename to rb_file_expand_path_internal.
      Conditionally exclude from Windows.

    * file.c (rb_file_expand_path_fast): new function. delegates to
      rb_file_expand_path_internal without performing a hit to the
      filesystem.

    * file.c (file_expand_path_1): use rb_file_expand_path_internal without
      path expansion (used by require).

    * file.c (rb_find_file_ext_safe): ditto.

    * file.c (rb_find_file_safe): ditto.

    * load.c (rb_get_expanded_load_path): use rb_file_expand_path_fast.

    * load.c (rb_feature_provided): ditto.

    * file.c (rb_file_expand_path): use rb_file_expand_path_internal with
      path expansion.

    * file.c (rb_file_absolute_path): ditto.

    * test/ruby/test_file_exhaustive.rb: new tests to exercise
      rb_file_expand_path_internal implementation and compliance with
      existing behaviors.

    * test/ruby/test_file_exhaustive.rb: fix test introduced in r36811 for
      posix environments where HOME is not defined.  [ruby-core:47322]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_1_9_3@37321 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

History

Updated by luislavena (Luis Lavena) over 7 years ago

  • Category set to core
  • Assignee set to usa (Usaku NAKAMURA)
  • Target version set to 2.0.0

Updated by luislavena (Luis Lavena) over 7 years ago

Usa,

Do you approve this be merged?

Thank you.

Updated by jonforums (Jon Forums) over 7 years ago

As there's been no negative feedback in the past 17 days, is this patch acceptable as-is and can it now be committed to trunk?

Updated by usa (Usaku NAKAMURA) over 7 years ago

  • Status changed from Open to Assigned
  • Assignee changed from usa (Usaku NAKAMURA) to h.shirosaki (Hiroshi Shirosaki)

I don't object.
Shirosaki-san, please commit it.

#5

Updated by Anonymous over 7 years ago

  • Status changed from Assigned to Closed
  • % Done changed from 0 to 100

This issue was solved with changeset r34849.
Hiroshi, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


  • Makefile.in (PLATFORM_DIR): add a variable for win32 directory.
  • Makefile.in (clean-platform): add new target.
    It cleans win32 directory.

  • common.mk (clean): add a dependency for win32 directory.

  • common.mk (distclean): ditto.

  • common.mk (distclean-platform): add new target.
    It cleans win32 directory.

  • common.mk ($(PLATFORM_D)): add new target to make win32 directory.

  • common.mk (win32/win32.$(OBJEXT)): move win32.o into win32
    directory.

  • common.mk (win32/file.$(OBJEXT)): add new target for win32/file.c.

  • configure.in: move win32.o into win32 directory and add
    win32/file.o to MISSING.

  • file.c (file_load_ok, rb_file_load_ok): replace static
    file_load_ok() with public rb_file_load_ok().
    It's to link Windows implementation in win32/file.c.

  • file.c (rb_find_file_ext_safe): ditto.

  • file.c (rb_find_file_safe): ditto.

  • win32/file.c (rb_file_load_ok): new file. Add Windows specific
    optimized implementation of rb_file_load_ok(). We created a
    separated file to avoid too many #ifdef macro which is unreadable.

  • win32/Makefile.sub (PLATFORM_DIR): add a variable for win32
    directory.

  • win32/Makefile.sub (MISSING): move win32.obj into win32
    directory and add win32/file.obj to MISSING.

  • win32/Makefile.sub (MAKEDIRS): replace MINIRUBY with BASERUBY.
    It's because miniruby doesn't exist when making win32 directory.

  • win32/Makefile.sub (clean-platform): add new target to clean win32
    directory.

  • win32/Makefile.sub ({$(srcdir)}.c{}.obj): make it not match
    win32/file.c to build properly.

  • win32/Makefile.sub (win32/win32.$(OBJEXT)): move win32.obj into
    win32 directory.

Patch created with Luis Lavena.
[ruby-core:42480] [Feature #5999]

Updated by h.shirosaki (Hiroshi Shirosaki) over 7 years ago

I don't object.
Shirosaki-san, please commit it.

Thank you. I committed it at r34849.

--
Hiroshi Shirosaki

Also available in: Atom PDF