YJIT Benchmarks

Details for Benchmarks at 2021-12-09 07:10:44 GMT

YJIT metrics from the yjit-bench suite

Overall YJIT is 31.6% faster than interpreted CRuby!
On Railsbench specifically, YJIT is 28.7% faster than CRuby!

Performance on Headline Benchmarks

Speed of each Ruby implementation relative to the baseline CRuby measurement. Higher is better.

Memory Usage on Headline Benchmarks

0.0 2.0 4.0 6.0 8.0 CRuby 3.1.0dev MJIT3.0 MJIT YJIT 3.1.0dev activerecord hexapdf liquid-render mail psych-load railsbench geomean*
Memory usage of each Ruby implementation relative to the baseline CRuby measurement. Lower is better.

Performance on Other Benchmarks

0.0 0.5 1.0 1.5 2.0 No JIT MJIT3.0 MJIT YJIT binarytrees erubi erubi_rails fannkuchredux jekyll lee nbody optcarrot rubykon
Speed of each Ruby implementation relative to the baseline CRuby measurement. Higher is better.

Memory Usage on Other Benchmarks

0.0 2.0 4.0 6.0 8.0 10.0 CRuby 3.1.0dev MJIT3.0 MJIT YJIT 3.1.0dev binarytrees erubi erubi_rails fannkuchredux jekyll lee nbody optcarrot rubykon geomean*
Memory usage of each Ruby implementation relative to the baseline CRuby measurement. Lower is better.

Performance on MicroBenchmarks

0.0 2.0 4.0 6.0 8.0 No JIT MJIT3.0 MJIT YJIT 30k_ifelse 30k_methods cfunc_itself fib getivar keyword_args respond_to setivar
Speed of each Ruby implementation relative to the baseline CRuby measurement. Higher is better.

Memory Usage on MicroBenchmarks

0.0 2.0 4.0 6.0 8.0 10.0 CRuby 3.1.0dev MJIT3.0 MJIT YJIT 3.1.0dev 30k_ifelse 30k_methods cfunc_itself fib getivar keyword_args respond_to setivar geomean*
Memory usage of each Ruby implementation relative to the baseline CRuby measurement. Lower is better.

Want Raw Graphs and CSV?

Benchmarks Speed Details

bench No JIT (ms) No JIT RSD MJIT3.0 (ms) MJIT3.0 RSD MJIT (ms) MJIT RSD YJIT (ms) YJIT RSD MJIT3.0 spd MJIT3.0 spd RSD MJIT spd MJIT spd RSD YJIT spd YJIT spd RSD % in YJIT
activerecord 143.7 1.03% 157.9 0.75% 174.4 5.12% 106.3 0.43% 0.91x 1.28% 0.82x 5.22% 1.35x 1.12% 81.84%
hexapdf 2798.9 0.60% 2591.0 1.03% 2163.6 1.25% 1.08x 1.19% 1.29x 1.39% 74.81%
liquid-render 171.2 1.29% 159.5 2.24% 157.0 4.42% 115.4 1.82% 1.07x 2.59% 1.09x 4.61% 1.48x 2.23% 84.87%
mail 160.7 0.23% 147.3 0.64% 160.3 1.79% 142.0 0.32% 1.09x 0.68% 1.00x 1.80% 1.13x 0.39% 98.33%
psych-load 2225.8 0.96% 1932.3 0.68% 1817.6 0.19% 1617.2 0.25% 1.15x 1.18% 1.22x 0.98% 1.38x 1.00% 82.03%
railsbench 2676.4 0.95% 2881.2 1.22% 3053.3 1.66% 2079.4 1.69% 0.93x 1.54% 0.88x 1.91% 1.29x 1.94% 81.71%
binarytrees 381.5 0.15% 229.5 0.12% 245.4 0.12% 291.1 0.10% 1.66x 0.19% 1.55x 0.19% 1.31x 0.18% 84.27%
erubi 409.8 1.12% 398.8 0.88% 384.3 0.87% 421.4 1.28% 1.03x 1.42% 1.07x 1.41% 0.97x 1.70% 5.62%
erubi_rails 39.0 8.65% 49.2 5.07% 42.5 8.18% 33.4 3.63% 0.79x 10.03% 0.92x 11.90% 1.17x 9.39% 82.86%
fannkuchredux 5639.8 0.04% 3948.0 0.50% 5653.1 0.06% 5627.9 0.05% 1.43x 0.50% 1.00x 0.07% 1.00x 0.06% 0.01%
jekyll 8370.6 1.47% 7955.2 2.69% 8276.5 4.10% 7376.3 2.85% 1.05x 3.07% 1.01x 4.36% 1.13x 3.21% 82.34%
lee 1022.1 1.07% 920.5 0.89% 828.7 1.17% 753.8 0.07% 1.11x 1.39% 1.23x 1.59% 1.36x 1.07% 99.93%
nbody 104.1 0.06% 60.6 0.58% 58.5 0.03% 71.7 0.06% 1.72x 0.58% 1.78x 0.07% 1.45x 0.09% 100.00%
optcarrot 5169.7 0.43% 2181.2 0.59% 2185.3 0.66% 3075.7 0.57% 2.37x 0.73% 2.37x 0.79% 1.68x 0.72% 96.13%
rubykon 10555.2 0.39% 6471.3 0.47% 6793.6 0.43% 5511.2 0.33% 1.63x 0.61% 1.55x 0.58% 1.92x 0.51% 99.68%
30k_ifelse 2338.0 0.12% 2168.0 0.24% 4102.7 4.98% 318.2 0.11% 1.08x 0.27% 0.57x 4.98% 7.35x 0.17% 100.00%
30k_methods 5949.7 0.06% 6063.5 2.73% 7148.2 0.04% 737.1 0.02% 0.98x 2.73% 0.83x 0.07% 8.07x 0.06% 100.00%
cfunc_itself 97.3 0.17% 59.8 0.15% 66.6 0.11% 39.8 0.40% 1.63x 0.23% 1.46x 0.20% 2.44x 0.43% 100.00%
fib 210.4 0.06% 59.8 0.06% 62.5 0.13% 52.1 0.18% 3.52x 0.08% 3.37x 0.14% 4.04x 0.19% 100.00%
getivar 87.6 0.22% 90.3 0.18% 88.8 0.11% 35.5 0.02% 0.97x 0.28% 0.99x 0.25% 2.47x 0.22% 98.93%
keyword_args 242.1 0.05% 184.5 0.05% 197.5 0.03% 46.3 0.18% 1.31x 0.07% 1.23x 0.06% 5.23x 0.19% 100.00%
respond_to 229.0 0.71% 184.0 0.55% 196.0 0.24% 154.9 0.37% 1.24x 0.90% 1.17x 0.75% 1.48x 0.80% 100.00%
setivar 64.6 0.07% 68.2 0.10% 66.2 0.09% 39.5 0.48% 0.95x 0.12% 0.98x 0.12% 1.63x 0.49% 98.40%

RSD is relative standard deviation - the standard deviation divided by the mean, expressed as a percentage.
% in YJIT is the percentage of instructions that complete in YJIT rather than exiting to the non-JITted interpreter. YJIT performs better when this is higher.
Speedup is relative to interpreted CRuby. So an "MJIT speedup" of 1.21x means MJIT runs at 1.21 times the iters/second of CRuby with JIT disabled.

You can find our benchmark code in the yjit-bench Github repo and the yjit-extra-benchmarks Github repo.
Our benchmark-runner and reporting code is in the yjit-metrics Github repo.

Tested Ruby version for YJIT and No-JIT: ruby 3.1.0dev (2021-12-09T03:28:42Z master c5f68a9a24) +YJIT [x86_64-linux]
Tested Ruby version for Ruby latest MJIT: ruby 3.1.0dev (2021-12-09T03:28:42Z master c5f68a9a24) +JIT [x86_64-linux]
Tested Ruby version for Ruby 3.0 MJIT: ruby 3.0.0p0 (2020-12-25 revision 95aff21468) +JIT [x86_64-linux]
For many older releases we got much better results with released MJIT 3.0 than latest prerelease, so we compared against both.

Benchmark Memory Usage Details

bench CRuby 3.1.0dev mem (MiB) MJIT3.0 mem (MiB) MJIT mem (MiB) YJIT 3.1.0dev mem (MiB) Inline Code Outlined Code YJIT Mem overhead
activerecord 63 60 71 322 1 1 406.2%
hexapdf 299 331 661 2 1 121.2%
liquid-render 31 29 35 290 1 1 828.6%
mail 48 48 51 306 1 1 535.7%
psych-load 39 35 42 296 1 1 652.3%
railsbench 104 104 136 374 3 2 259.4%
binarytrees 29 31 29 285 1 1 867.0%
erubi 71 76 107 337 1 1 371.5%
erubi_rails 110 119 113 386 3 2 249.6%
fannkuchredux 23 21 23 280 1 1 1090.0%
jekyll 840 1460 2011 1708 2 2 103.3%
lee 35 32 37 289 1 1 707.5%
nbody 23 22 23 280 1 1 1086.2%
optcarrot 74 79 84 340 1 1 358.4%
rubykon 67 82 71 323 1 1 380.5%
30k_ifelse 63 62 190 348 6 5 446.1%
30k_methods 56 55 180 323 3 2 474.1%
cfunc_itself 23 22 23 280 1 1 1090.2%
fib 23 21 23 280 1 1 1092.8%
getivar 23 22 23 280 1 1 1084.2%
keyword_args 23 22 23 280 1 1 1083.3%
respond_to 23 22 23 280 1 1 1088.6%
setivar 23 22 23 280 1 1 1088.1%

Memory is shown in mebibytes (1024 * 1024 bytes.)

Older YJIT allocated an additional 256MiB for generated code. Current YJIT allocates executable memory on demand, so this overhead should no longer be present.

Number of Iterations and Warmups Tested

bench No JIT warmups No JIT iters MJIT3.0 warmups MJIT3.0 iters MJIT warmups MJIT iters YJIT warmups YJIT iters
activerecord 5 191 20 191 75 191 20 191
hexapdf 5 15 20 15 20 15
liquid-render 5 167 20 167 75 167 20 167
mail 5 141 20 141 75 141 20 141
psych-load 5 15 20 15 75 15 20 15
railsbench 5 15 20 15 75 15 20 15
binarytrees 5 87 20 87 75 87 20 87
erubi 5 52 20 52 75 52 20 52
erubi_rails 5 620 20 620 75 620 20 620
fannkuchredux 5 15 20 15 51 15 20 15
jekyll 5 15 20 15 36 15 20 15
lee 5 26 20 26 75 26 20 26
nbody 5 343 20 343 75 343 20 343
optcarrot 5 15 20 15 75 15 20 15
rubykon 5 15 20 15 44 15 20 15
30k_ifelse 5 62 20 62 75 62 20 62
30k_methods 5 27 20 27 41 27 20 27
cfunc_itself 5 491 20 491 75 491 20 491
fib 5 383 20 383 75 383 20 383
getivar 5 562 20 562 75 562 20 562
keyword_args 5 434 20 434 75 434 20 434
respond_to 5 131 20 131 75 131 20 131
setivar 5 505 20 505 75 505 20 505

Different Ruby configurations want different amounts of warmup. With no JIT, CRuby needs hardly any. YJIT and MJIT 3.0 both warm up quite quickly, while MJIT in 3.1 often slows down for a time as it compiles, after an unpredictable delay.

Benchmark YJIT Stats

Hover your cursor over the benchmark names for descriptions of each benchmark.

bench Exit Report Inline Outlined Comp iSeqs Comp Blocks Inval Inval Ratio Bind Alloc Bind Set Const Bumps
activerecord (click) 990122 756996 185 1194 0 0% 0 0 0
hexapdf (click) 1096362 893267 595 7997 614 7% 0 0 321
liquid-render (click) 494405 390718 143 1462 95 6% 0 0 1
mail (click) 853802 638390 325 5026 124 2% 0 0 39
psych-load (click) 312993 246919 57 436 1 0% 0 0 0
railsbench (click) 2514658 1905712 1295 9234 263 2% 0 0 26
binarytrees (click) 164467 127410 9 49 0 0% 0 0 0
erubi (click) 290849 229356 8 35 0 0% 0 0 0
erubi_rails (click) 2445794 1765614 260 1938 17 0% 0 0 3
fannkuchredux (click) 180467 139420 6 203 0 0% 0 0 0
jekyll (click) 1879210 1513266 336 3781 186 4% 0 0 0
lee (click) 355178 280423 44 562 67 11% 0 0 12
nbody (click) 173875 134611 8 149 0 0% 0 0 0
optcarrot (click) 502963 440300 196 3567 20 0% 0 0 0
rubykon (click) 305971 247587 138 1525 1 0% 0 0 0
30k_ifelse (click) 5553651 4347734 9261 57795 0 0% 0 0 0
30k_methods (click) 2169907 1656853 5780 19358 0 0% 0 0 0
cfunc_itself (click) 163123 126039 6 40 0 0% 0 0 0
fib (click) 161587 125647 6 29 0 0% 0 0 0
getivar (click) 163315 128158 6 56 0 0% 0 0 0
keyword_args (click) 164147 126793 7 42 0 0% 0 0 0
respond_to (click) 165171 127493 6 55 0 0% 0 0 0
setivar (click) 163827 127011 6 29 0 0% 0 0 0

YJIT stats correspond to the YJIT stats exit report.

Note: currently, all stats are collected on x86_64, not ARM.

Raw JSON data files

All graphs and table data in this page comes from processing these data files, which come from benchmark runs.