YJIT Benchmarks

Details for Benchmarks at 2022-09-05 20:13:00 GMT

YJIT metrics from the yjit-bench suite

Overall YJIT is 35.2% faster than interpreted CRuby!
On Railsbench specifically, YJIT is 34.2% faster than CRuby!

Performance on Headline Benchmarks

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 No JIT MJIT YJIT activerecord hexapdf liquid-render mail psych-load railsbench
Speed of each Ruby implementation relative to the baseline CRuby measurement. Higher is better.

Memory Usage on Headline Benchmarks

0.0 0.2 0.4 0.6 0.8 1.0 1.2 CRuby 3.2.0dev MJIT YJIT 3.2.0dev activerecord hexapdf liquid-render mail psych-load railsbench geomean*
Memory usage of each Ruby implementation relative to the baseline CRuby measurement. Lower is better.

Performance on Other Benchmarks

0.0 0.5 1.0 1.5 2.0 No JIT MJIT YJIT binarytrees chunky_png erubi erubi_rails fannkuchredux lee nbody optcarrot rubykon
Speed of each Ruby implementation relative to the baseline CRuby measurement. Higher is better.

Memory Usage on Other Benchmarks

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 CRuby 3.2.0dev MJIT YJIT 3.2.0dev binarytrees chunky_png erubi erubi_rails fannkuchredux lee nbody optcarrot rubykon geomean*
Memory usage of each Ruby implementation relative to the baseline CRuby measurement. Lower is better.

Performance on MicroBenchmarks

0.0 2.0 4.0 6.0 8.0 No JIT MJIT YJIT 30k_ifelse 30k_methods cfunc_itself fib getivar keyword_args respond_to setivar str_concat
Speed of each Ruby implementation relative to the baseline CRuby measurement. Higher is better.

Memory Usage on MicroBenchmarks

0.0 0.5 1.0 1.5 2.0 2.5 CRuby 3.2.0dev MJIT YJIT 3.2.0dev 30k_ifelse 30k_methods cfunc_itself fib getivar keyword_args respond_to setivar str_concat geomean*
Memory usage of each Ruby implementation relative to the baseline CRuby measurement. Lower is better.

Want Raw Graphs and CSV?

Benchmarks Speed Details

bench No JIT (ms) No JIT RSD MJIT (ms) MJIT RSD YJIT (ms) YJIT RSD MJIT spd MJIT spd RSD YJIT spd YJIT spd RSD % in YJIT
activerecord 151.2 0.52% 156.8 0.27% 101.8 0.67% 0.96x 0.59% 1.48x 0.85% 85.98%
hexapdf 3196.3 1.00% 2347.6 2.26% 1.36x 2.47% 83.68%
liquid-render 193.7 1.83% 168.6 1.19% 123.7 1.65% 1.15x 2.18% 1.57x 2.46% 85.90%
mail 175.5 0.14% 174.5 2.55% 156.7 0.24% 1.01x 2.56% 1.12x 0.28% 99.36%
psych-load 2426.4 0.05% 2132.5 0.03% 1891.2 0.02% 1.14x 0.06% 1.28x 0.05% 88.89%
railsbench 2780.7 0.59% 2762.4 0.61% 2072.7 1.13% 1.01x 0.85% 1.34x 1.28% 86.40%
binarytrees 479.7 0.04% 310.3 0.15% 372.0 0.06% 1.55x 0.15% 1.29x 0.07% 84.27%
chunky_png 948.2 0.05% 766.8 0.06% 654.1 0.15% 1.24x 0.08% 1.45x 0.16% 99.95%
erubi 375.5 0.60% 351.8 0.53% 329.0 0.63% 1.07x 0.80% 1.14x 0.86% 100.00%
erubi_rails 27.0 2.80% 26.6 2.46% 19.3 3.52% 1.01x 3.73% 1.40x 4.50% 86.41%
fannkuchredux 5416.2 0.11% 5393.3 0.08% 2355.9 0.08% 1.00x 0.13% 2.30x 0.14% 77.42%
lee 1245.3 0.68% 1053.9 0.57% 948.5 0.96% 1.18x 0.89% 1.31x 1.18% 99.97%
nbody 126.4 0.07% 76.2 0.06% 91.2 0.07% 1.66x 0.09% 1.39x 0.10% 100.00%
optcarrot 6281.3 0.43% 2686.2 0.62% 3449.7 0.61% 2.34x 0.76% 1.82x 0.75% 96.16%
rubykon 12309.5 0.43% 8801.5 0.90% 6798.8 0.60% 1.40x 1.00% 1.81x 0.74% 99.74%
30k_ifelse 2241.2 0.01% 4229.4 2.64% 366.2 0.09% 0.53x 2.64% 6.12x 0.09% 100.00%
30k_methods 6400.1 0.00% 12953.1 4.26% 786.1 0.02% 0.49x 4.26% 8.14x 0.02% 100.00%
cfunc_itself 105.0 0.11% 98.2 0.30% 44.7 0.59% 1.07x 0.32% 2.35x 0.60% 100.00%
fib 245.4 0.03% 86.1 0.03% 64.8 0.19% 2.85x 0.04% 3.79x 0.20% 100.00%
getivar 104.3 0.18% 104.3 0.25% 42.6 0.05% 1.00x 0.31% 2.45x 0.18% 98.73%
keyword_args 289.4 0.11% 224.0 0.07% 53.9 0.18% 1.29x 0.13% 5.37x 0.21% 100.00%
respond_to 274.4 0.19% 257.5 0.19% 187.9 0.22% 1.07x 0.27% 1.46x 0.29% 100.00%
setivar 75.3 0.09% 75.3 0.07% 43.3 0.16% 1.00x 0.11% 1.74x 0.18% 97.89%
str_concat 83.6 1.26% 53.1 7.17% 47.4 1.16% 1.58x 7.28% 1.76x 1.71% 99.72%

RSD is relative standard deviation - the standard deviation divided by the mean, expressed as a percentage.
% in YJIT is the percentage of instructions that complete in YJIT rather than exiting to the non-JITted interpreter. YJIT performs better when this is higher.
Speedup is relative to interpreted CRuby. So an "MJIT speedup" of 1.21x means MJIT runs at 1.21 times the iters/second of CRuby with JIT disabled.

You can find our benchmark code in the yjit-bench Github repo and the yjit-extra-benchmarks Github repo.
Our benchmark-runner and reporting code is in the yjit-metrics Github repo.

Tested Ruby version for YJIT and No-JIT: ruby 3.2.0dev (2022-09-05T15:39:37Z master 63ed61e322) +YJIT [x86_64-linux]
Tested Ruby version for Ruby latest MJIT: ruby 3.2.0dev (2022-09-05T15:39:37Z master 63ed61e322) +MJIT [x86_64-linux]

Benchmark Memory Usage Details

bench CRuby 3.2.0dev mem (MiB) MJIT mem (MiB) YJIT 3.2.0dev mem (MiB) Inline Code Outlined Code YJIT Mem overhead
activerecord 64 66 76 2 1 18.9%
hexapdf 285 383 2 1 34.4%
liquid-render 31 34 38 1 1 20.5%
mail 49 54 56 1 1 14.1%
psych-load 38 43 41 1 1 7.0%
railsbench 104 105 131 3 2 25.6%
binarytrees 30 32 33 1 1 8.2%
chunky_png 39 46 44 1 1 13.1%
erubi 80 113 93 1 1 15.5%
erubi_rails 103 103 122 2 2 18.0%
fannkuchredux 24 25 27 1 1 11.6%
lee 33 39 40 1 1 20.9%
nbody 24 25 27 1 1 11.2%
optcarrot 63 70 69 1 1 10.0%
rubykon 51 60 55 1 1 8.4%
30k_ifelse 66 147 124 6 5 86.4%
30k_methods 57 170 76 3 2 33.0%
cfunc_itself 24 25 27 1 1 10.6%
fib 24 26 27 1 1 9.9%
getivar 25 25 27 1 1 9.6%
keyword_args 25 26 27 1 1 10.2%
respond_to 24 25 27 1 1 9.9%
setivar 24 25 27 1 1 10.4%
str_concat 125 119 107 1 1 -14.9%

Memory is shown in mebibytes (1024 * 1024 bytes.)

Older YJIT allocated an additional 256MiB for generated code. Current YJIT allocates executable memory on demand, so this overhead should no longer be present.

Number of Iterations and Warmups Tested

bench No JIT warmups No JIT iters MJIT warmups MJIT iters YJIT warmups YJIT iters
activerecord 5 194 75 194 20 194
hexapdf 5 15 20 15
liquid-render 5 161 75 161 20 161
mail 5 125 75 125 20 125
psych-load 5 15 75 15 20 15
railsbench 5 15 75 15 20 15
binarytrees 5 67 75 67 20 67
chunky_png 5 30 75 30 20 30
erubi 5 62 75 62 20 62
erubi_rails 5 1031 75 1031 20 1031
fannkuchredux 5 15 56 15 20 15
lee 5 21 75 21 20 21
nbody 5 264 75 264 20 264
optcarrot 5 15 75 15 20 15
rubykon 5 15 33 15 20 15
30k_ifelse 5 54 72 54 20 54
30k_methods 5 25 22 25 20 25
cfunc_itself 5 448 75 448 20 448
fib 5 308 75 308 20 308
getivar 5 469 75 469 20 469
keyword_args 5 375 75 375 20 375
respond_to 5 103 75 103 20 103
setivar 5 379 75 379 20 379
str_concat 5 483 75 483 20 483

Different Ruby configurations want different amounts of warmup. With no JIT, CRuby needs hardly any. YJIT and MJIT 3.0 both warm up quite quickly, while MJIT in 3.1 often slows down for a time as it compiles, after an unpredictable delay.

Benchmark YJIT Stats

Hover your cursor over the benchmark names for descriptions of each benchmark.

bench Exit Report Inline Outlined Comp iSeqs Comp Blocks Inval Inval Ratio Bind Alloc Bind Set Const Bumps
activerecord (click) 1076471 780891 131 1267 0 0% 0 0 0
hexapdf (click) 1195511 924814 682 8022 9 0% 0 0 0
liquid-render (click) 534839 401873 150 1674 2 0% 0 0 0
mail (click) 957751 665333 374 5566 17 0% 0 0 0
psych-load (click) 360695 263576 68 482 1 0% 0 0 0
railsbench (click) 2759607 1967964 1436 10004 16 0% 0 0 0
binarytrees (click) 176183 127426 13 58 0 0% 0 0 0
chunky_png (click) 402807 290774 88 1007 0 0% 0 0 0
erubi (click) 344439 252507 10 79 0 0% 0 0 0
erubi_rails (click) 2045815 1432288 300 2026 4 0% 0 0 0
fannkuchredux (click) 190263 136909 10 196 0 0% 0 0 0
lee (click) 399159 290145 58 634 0 0% 0 0 0
nbody (click) 187191 136381 12 171 0 0% 0 0 0
optcarrot (click) 534839 441594 207 3588 20 0% 0 0 0
rubykon (click) 332087 256788 144 1583 1 0% 0 0 0
30k_ifelse (click) 5704311 4347894 9265 57804 0 0% 0 0 0
30k_methods (click) 2209847 1656681 5784 19361 0 0% 0 0 0
cfunc_itself (click) 175735 126524 10 49 0 0% 0 0 0
fib (click) 173111 125807 10 38 0 0% 0 0 0
getivar (click) 174583 127654 10 65 0 0% 0 0 0
keyword_args (click) 175479 126840 11 51 0 0% 0 0 0
respond_to (click) 176759 127275 10 64 0 0% 0 0 0
setivar (click) 174839 126714 10 38 0 0% 0 0 0
str_concat (click) 174455 126668 12 62 0 0% 0 0 0

YJIT stats correspond to the YJIT stats exit report.

Note: currently, all stats are collected on x86_64, not ARM.

Raw JSON data files

All graphs and table data in this page comes from processing these data files, which come from benchmark runs.