Feature #14109
Updated by esparta (Espartaco Palma) over 6 years ago
Dir.children is available since Feature #11302. FileUtils uses
Dir.each on an internal method encapsulated on a private class
Entry_#entry, having no '.' neither '..' entries would make
now superfluous a chained reject filtering.
This change can improve the performance of these FileUtils
methods when the provided path covers thousands of files or
directories:
* chmod_R
* chown_R
* remove_entry
* remove_entry_secure
* rm_r
* remove_dir
* copy_entry
Related: Feature #13896
Some profiling I did using 50,000 files on a given folder, using this code:
~~~ ruby
require_relative 'fileutils'
FileUtils.chmod_R 0777, 'benchmark'
~~~
Before the patch:
~~~ text
28.69 15.05 15.05 200004 0.08 0.10 FileUtils::Entry_#join
10.68 20.65 5.60 100005 0.06 0.26 FileUtils::Entry_#entries
6.75 24.19 3.54 100004 0.04 0.11 FileUtils::Entry_#lstat
5.68 27.17 2.98 150007 0.02 0.13 FileUtils::Entry_#path
4.97 29.78 2.61 50002 0.05 0.44 FileUtils::Entry_#chmod
4.97 29.78 2.61 50002 0.05 0.44 FileUtils::Entry_#chmod
4.80 32.29 2.52 50004 0.05 2.73 FileUtils.chmod_R
4.33 34.56 2.27 1 2268.74 52382.08 FileUtils::Entry_#preorder_traverse
4.18 36.76 2.19 450012 0.00 0.00 String#==
3.73 38.71 1.96 100004 0.02 0.13 FileUtils::Entry_#lstat!
3.56 40.58 1.87 400008 0.00 0.00 BasicObject#!
3.35 42.33 1.76 50002 0.04 0.08 FileUtils::Entry_#directory?
3.27 44.05 1.72 50002 0.03 0.25 FileUtils::Entry_#symlink?
2.14 45.17 1.12 150003 0.01 0.01 File.join
2.01 46.23 1.05 50002 0.02 0.03 Class#new
1.89 47.22 0.99 50002 0.02 0.02 FileUtils.fu_mode
1.47 47.99 0.77 4 192.12 2755.69 Array#map
1.40 48.72 0.73 1 733.87 2821.06 Array#reject
0.92 49.20 0.48 50002 0.01 0.01 File.lstat
0.85 49.65 0.44 100004 0.00 0.00 FileUtils::Entry_#dereference?
0.82 50.07 0.43 50002 0.01 0.01 File.chmod
0.57 50.37 0.30 50006 0.01 0.01 File.path
0.56 50.67 0.29 50002 0.01 0.01 FileUtils::Entry_#initialize
0.49 50.92 0.26 50002 0.01 0.01 File::Stat#directory?
0.49 51.18 0.25 50002 0.01 0.01 File::Stat#symlink?
0.47 51.43 0.25 50003 0.00 0.00 Array#pop
0.47 51.67 0.24 50001 0.00 0.00 FileUtils::Entry_#rel
0.46 51.91 0.24 50001 0.00 0.00 Kernel#untaint
0.43 52.14 0.23 50002 0.00 0.00 Kernel#is_a?
0.43 52.36 0.23 50001 0.00 0.00 FileUtils::Entry_#prefix
0.04 52.38 0.02 1 18.88 18.98 Dir.entries
0.03 52.40 0.01 1 13.28 59.69 Kernel#require_relative
0.02 52.41 0.01 320 0.04 0.09 nil#
0.01 52.41 0.01 86 0.08 0.08 Module#module_eval
0.01 52.42 0.01 180 0.03 0.10 FileUtils.collect_method
0.01 52.42 0.00 4 0.67 2.51 Array#select
0.01 52.42 0.00 44 0.06 0.15 Array#map!
0.01 52.43 0.00 3 0.88 1.08 Module#public
0.00 52.43 0.00 2 1.12 2.61 Kernel#require
0.00 52.43 0.00 5 0.39 10481.37 Array#each
0.00 52.43 0.00 262 0.01 0.01 Module#method_added
0.00 52.43 0.00 61 0.02 0.03 Module#module_function
0.00 52.44 0.00 226 0.00 0.00 BasicObject#singleton_method_added
0.00 52.44 0.00 176 0.00 0.00 Hash#[]
0.00 52.44 0.00 176 0.00 0.00 Array#include?
0.00 52.44 0.00 168 0.00 0.00 Symbol#==
0.00 52.44 0.00 17 0.03 0.06 FileUtils.private_module_function
0.00 52.44 0.00 22 0.02 0.03 Module#alias_method
0.00 52.44 0.00 86 0.00 0.00 Integer#+
0.00 52.44 0.00 1 0.33 0.33 Array#reverse
0.00 52.44 0.00 44 0.01 0.01 UnboundMethod#parameters
0.00 52.44 0.00 44 0.01 0.01 Hash#[]=
0.00 52.44 0.00 7 0.03 0.05 Module#include
0.00 52.44 0.00 44 0.01 0.01 Module#instance_method
0.00 52.44 0.00 44 0.00 0.00 Array#compact!
0.00 52.44 0.00 1 0.19 0.19 Array#concat
0.00 52.44 0.00 44 0.00 0.00 Symbol#to_s
0.00 52.44 0.00 4 0.04 0.07 Kernel#extend
0.00 52.44 0.00 28 0.00 0.00 String#intern
0.00 52.44 0.00 1 0.10 0.10 Dir.open
0.00 52.44 0.00 17 0.00 0.00 Module#private_class_method
0.00 52.44 0.00 4 0.02 0.02 Module#extend_object
0.00 52.44 0.00 7 0.01 0.01 Module#append_features
0.00 52.44 0.00 4 0.01 0.01 Hash#keys
0.00 52.44 0.00 1 0.05 0.07 MonitorMixin#mon_enter
0.00 52.44 0.00 2 0.03 0.06 FileUtils.fu_list
0.00 52.44 0.00 7 0.01 0.01 Module#private
0.00 52.44 0.00 2 0.03 0.03 Kernel#singleton_methods
0.00 52.44 0.00 1 0.05 0.05 Numeric#zero?
0.00 52.44 0.00 1 0.04 12.84 Enumerable#inject
0.00 52.44 0.00 1 0.04 0.04 Module#private_instance_methods
0.00 52.44 0.00 4 0.01 0.01 IO#set_encoding
0.00 52.44 0.00 1 0.03 0.06 MonitorMixin#mon_exit
0.00 52.44 0.00 7 0.00 0.00 Module#included
0.00 52.44 0.00 2 0.01 0.01 Array#-
0.00 52.44 0.00 1 0.03 0.07 Numeric#nonzero?
0.00 52.44 0.00 1 0.02 0.02 Kernel#methods
0.00 52.44 0.00 1 0.02 0.03 FileUtils::StreamUtils_#fu_windows?
0.00 52.44 0.00 4 0.01 0.01 Module#extended
0.00 52.44 0.00 1 0.02 0.02 MonitorMixin#mon_check_owner
0.00 52.44 0.00 1 0.02 0.02 Array#&
0.00 52.44 0.00 3 0.00 0.00 Class#inherited
0.00 52.44 0.00 3 0.00 0.00 Thread.current
0.00 52.44 0.00 1 0.01 0.01 Regexp#=~
0.00 52.44 0.00 1 0.01 0.01 TracePoint#enable
0.00 52.44 0.00 1 0.01 0.01 Gem::Specification.unresolved_deps
0.00 52.44 0.00 1 0.01 0.01 Array#flatten
0.00 52.44 0.00 1 0.01 0.01 Gem.find_unresolved_default_spec
0.00 52.44 0.00 1 0.01 0.01 Kernel#respond_to?
0.00 52.44 0.00 1 0.01 0.01 TracePoint#disable
0.00 52.44 0.00 1 0.01 0.01 Thread::Mutex#unlock
0.00 52.44 0.00 1 0.00 0.00 Thread::Mutex#lock
0.00 52.44 0.00 1 0.00 52442.37 #toplevel
~~~
After the patch
~~~ text
30.07 14.76 14.76 200004 0.07 0.10 FileUtils::Entry_#join
7.87 18.62 3.86 50002 0.08 0.41 FileUtils::Entry_#entries
7.20 22.15 3.53 100004 0.04 0.11 FileUtils::Entry_#lstat
6.09 25.14 2.99 150007 0.02 0.12 FileUtils::Entry_#path
5.34 27.77 2.62 50002 0.05 0.43 FileUtils::Entry_#chmod
5.12 30.28 2.51 50004 0.05 2.59 FileUtils.chmod_R
4.64 32.56 2.28 1 2279.40 49019.52 FileUtils::Entry_#preorder_traverse
3.98 34.51 1.95 100004 0.02 0.13 FileUtils::Entry_#lstat!
3.72 36.34 1.83 400008 0.00 0.00 BasicObject#!
3.56 38.09 1.75 50002 0.03 0.08 FileUtils::Entry_#directory?
3.54 39.82 1.74 350007 0.00 0.00 String#==
3.47 41.53 1.70 50002 0.03 0.25 FileUtils::Entry_#symlink?
2.29 42.65 1.13 150003 0.01 0.01 File.join
2.05 43.66 1.01 50002 0.02 0.03 Class#new
2.01 44.65 0.99 50002 0.02 0.02 FileUtils.fu_mode
1.51 45.39 0.74 4 184.89 2664.27 Array#map
1.01 45.88 0.50 50002 0.01 0.01 File.lstat
0.90 46.32 0.44 50002 0.01 0.01 File.chmod
0.88 46.75 0.43 100004 0.00 0.00 FileUtils::Entry_#dereference?
0.60 47.05 0.29 50006 0.01 0.01 File.path
0.57 47.33 0.28 50002 0.01 0.01 FileUtils::Entry_#initialize
0.53 47.58 0.26 50002 0.01 0.01 File::Stat#symlink?
0.53 47.84 0.26 50003 0.01 0.01 Array#pop
0.51 48.09 0.25 50002 0.01 0.01 File::Stat#directory?
0.49 48.33 0.24 50001 0.00 0.00 FileUtils::Entry_#rel
0.48 48.57 0.24 50001 0.00 0.00 Kernel#untaint
0.45 48.79 0.22 50002 0.00 0.00 Kernel#is_a?
0.43 49.00 0.21 50001 0.00 0.00 FileUtils::Entry_#prefix
0.04 49.02 0.02 1 18.47 18.49 Dir.children
0.03 49.03 0.01 1 13.31 59.77 Kernel#require_relative
0.02 49.04 0.01 320 0.03 0.08 nil#
0.01 49.05 0.01 86 0.08 0.08 Module#module_eval
0.01 49.06 0.01 180 0.03 0.10 FileUtils.collect_method
0.01 49.06 0.00 2 1.29 3.07 Kernel#require
0.01 49.06 0.00 4 0.64 2.52 Array#select
0.01 49.06 0.00 44 0.06 0.14 Array#map!
0.00 49.07 0.00 3 0.70 1.10 Module#public
0.00 49.07 0.00 5 0.41 9808.82 Array#each
0.00 49.07 0.00 226 0.01 0.01 BasicObject#singleton_method_added
0.00 49.07 0.00 262 0.01 0.01 Module#method_added
0.00 49.07 0.00 61 0.02 0.02 Module#module_function
0.00 49.07 0.00 44 0.02 0.02 Module#instance_method
0.00 49.07 0.00 176 0.00 0.00 Array#include?
0.00 49.07 0.00 176 0.00 0.00 Hash#[]
0.00 49.08 0.00 168 0.00 0.00 Symbol#==
0.00 49.08 0.00 17 0.03 0.06 FileUtils.private_module_function
0.00 49.08 0.00 22 0.02 0.03 Module#alias_method
0.00 49.08 0.00 86 0.00 0.00 Integer#+
0.00 49.08 0.00 44 0.01 0.01 UnboundMethod#parameters
0.00 49.08 0.00 7 0.03 0.04 Module#include
0.00 49.08 0.00 44 0.01 0.01 Hash#[]=
0.00 49.08 0.00 1 0.20 0.20 Array#reverse
0.00 49.08 0.00 44 0.00 0.00 Array#compact!
0.00 49.08 0.00 44 0.00 0.00 Symbol#to_s
0.00 49.08 0.00 1 0.16 0.16 Array#concat
0.00 49.08 0.00 4 0.04 0.06 Kernel#extend
0.00 49.08 0.00 28 0.00 0.00 String#intern
0.00 49.08 0.00 17 0.00 0.00 Module#private_class_method
0.00 49.08 0.00 2 0.03 0.03 Kernel#singleton_methods
0.00 49.08 0.00 7 0.01 0.01 Module#private
0.00 49.08 0.00 1 0.05 0.07 MonitorMixin#mon_enter
0.00 49.08 0.00 2 0.03 0.05 FileUtils.fu_list
0.00 49.08 0.00 7 0.01 0.01 Module#append_features
0.00 49.08 0.00 4 0.01 0.01 Module#extend_object
0.00 49.08 0.00 1 0.04 0.04 Module#private_instance_methods
0.00 49.08 0.00 4 0.01 0.01 IO#set_encoding
0.00 49.08 0.00 7 0.00 0.00 Module#included
0.00 49.08 0.00 1 0.03 0.06 MonitorMixin#mon_exit
0.00 49.08 0.00 2 0.01 0.01 Array#-
0.00 49.08 0.00 1 0.03 0.03 Kernel#methods
0.00 49.08 0.00 1 0.02 0.03 Numeric#nonzero?
0.00 49.08 0.00 1 0.02 0.03 FileUtils::StreamUtils_#fu_windows?
0.00 49.08 0.00 1 0.02 12.57 Enumerable#inject
0.00 49.08 0.00 1 0.02 0.02 Dir.open
0.00 49.08 0.00 4 0.00 0.00 Module#extended
0.00 49.08 0.00 1 0.02 0.02 MonitorMixin#mon_check_owner
0.00 49.08 0.00 4 0.00 0.00 Hash#keys
0.00 49.08 0.00 1 0.02 0.02 Array#&
0.00 49.08 0.00 3 0.01 0.01 Class#inherited
0.00 49.08 0.00 3 0.00 0.00 Thread.current
0.00 49.08 0.00 1 0.01 0.01 TracePoint#enable
0.00 49.08 0.00 1 0.01 0.01 Regexp#=~
0.00 49.08 0.00 1 0.01 0.01 TracePoint#disable
0.00 49.08 0.00 1 0.01 0.01 Gem.find_unresolved_default_spec
0.00 49.08 0.00 1 0.01 0.01 Array#flatten
0.00 49.08 0.00 1 0.01 0.01 Gem::Specification.unresolved_deps
0.00 49.08 0.00 1 0.01 0.01 Numeric#zero?
0.00 49.08 0.00 1 0.01 0.01 Thread::Mutex#lock
0.00 49.08 0.00 1 0.01 0.01 Thread::Mutex#unlock
0.00 49.08 0.00 1 0.01 0.01 Kernel#respond_to?
0.00 49.08 0.00 1 0.00 49079.91 #toplevel
~~~
PR at GitHub: https://github.com/ruby/ruby/pull/1754