YJIT Benchmarks

Details for Benchmarks at 2022-05-28 19:12:53 GMT

YJIT metrics from the yjit-bench suite

¶Performance on Headline Benchmarks

X

Speed of each Ruby implementation relative to the baseline CRuby measurement. Higher is better.

¶Memory Usage on Headline Benchmarks

X

Memory usage of each Ruby implementation relative to the baseline CRuby measurement. Lower is better.

¶Performance on Other Benchmarks

X

Speed of each Ruby implementation relative to the baseline CRuby measurement. Higher is better.

¶Memory Usage on Other Benchmarks

X

Memory usage of each Ruby implementation relative to the baseline CRuby measurement. Lower is better.

¶Performance on MicroBenchmarks

X

Speed of each Ruby implementation relative to the baseline CRuby measurement. Higher is better.

¶Memory Usage on MicroBenchmarks

X

Memory usage of each Ruby implementation relative to the baseline CRuby measurement. Lower is better.

¶Want Raw Graphs and CSV?

• Headlining benchmarks (x86_64)
• All benchmarks (x86_64)
• CSV speed results (x86_64)

¶Benchmarks Speed Details

X

bench	No JIT (ms)	No JIT RSD	MJIT (ms)	MJIT RSD	YJIT (ms)	YJIT RSD	MJIT spd	MJIT spd RSD	YJIT spd	YJIT spd RSD	% in YJIT
activerecord	167.9	2.34%	202.3	5.47%	102.7	0.39%	0.83x	5.95%	1.63x	2.37%	85.54%
hexapdf	3202.4	0.97%			2417.2	2.35%			1.32x	2.54%	84.57%
liquid-render	212.0	1.19%	214.0	1.33%	142.6	1.65%	0.99x	1.79%	1.49x	2.04%	85.70%
mail	214.1	2.09%	204.3	3.32%	185.5	2.71%	1.05x	3.92%	1.15x	3.42%	99.32%
psych-load	2544.3	0.13%	2351.8	0.14%	2179.5	0.05%	1.08x	0.19%	1.17x	0.14%	88.45%
railsbench	3123.5	0.54%	3449.8	0.59%	2406.3	0.81%	0.91x	0.79%	1.30x	0.97%	86.19%
binarytrees	478.6	0.09%	307.3	0.32%	368.0	0.16%	1.56x	0.33%	1.30x	0.18%	84.27%
chunky_png	981.9	0.28%	791.5	0.63%	691.7	0.43%	1.24x	0.69%	1.42x	0.52%	99.95%
discourse	580.6	14.67%	611.0	13.93%	526.8	18.30%	0.95x	20.23%	1.10x	23.45%	73.51%
erubi	502.3	2.62%	454.6	1.08%	401.8	0.86%	1.10x	2.84%	1.25x	2.76%	100.00%
erubi_rails	32.0	39.33%	34.8	42.60%	24.9	56.01%	0.92x	57.98%	1.28x	68.44%	86.31%
fannkuchredux	8786.3	0.44%	3917.6	0.11%	9018.1	0.64%	2.24x	0.45%	0.97x	0.77%	0.02%
lee	1207.0	0.37%	967.9	0.35%	882.4	1.70%	1.25x	0.51%	1.37x	1.74%	99.97%
nbody	123.7	0.05%	72.7	0.11%	86.6	0.52%	1.70x	0.12%	1.43x	0.52%	100.00%
optcarrot	6092.0	0.43%	2563.4	0.63%	3524.4	0.62%	2.38x	0.76%	1.73x	0.75%	96.13%
rubykon	13625.2	1.81%	11507.9	1.13%	7835.6	3.26%	1.18x	2.13%	1.74x	3.73%	99.59%
30k_ifelse	2315.3	0.01%	3563.0	3.87%	363.7	0.18%	0.65x	3.87%	6.37x	0.18%	100.00%
30k_methods	6401.7	0.01%	10119.1	15.33%	882.1	0.05%	0.63x	15.33%	7.26x	0.06%	100.00%
cfunc_itself	107.3	0.22%	68.0	0.34%	47.6	0.97%	1.58x	0.41%	2.25x	1.00%	100.00%
fib	240.7	0.09%	86.1	0.03%	62.5	0.60%	2.80x	0.09%	3.85x	0.60%	100.00%
getivar	113.7	0.33%	26.3	70.73%	42.6	0.71%	4.32x	70.73%	2.67x	0.78%	99.21%
keyword_args	287.2	0.08%	220.0	0.25%	53.0	0.67%	1.31x	0.26%	5.42x	0.68%	100.00%
respond_to	279.8	0.19%	230.3	0.41%	190.0	0.25%	1.22x	0.45%	1.47x	0.32%	100.00%
setivar	86.7	0.26%	10.6	103.61%	51.2	0.74%	8.19x	103.61%	1.70x	0.78%	99.53%
str_concat	131.0	0.84%	100.9	1.10%	108.3	1.47%	1.30x	1.38%	1.21x	1.69%	99.68%

RSD is relative standard deviation - the standard deviation divided by the mean, expressed as a percentage.
% in YJIT is the percentage of instructions that complete in YJIT rather than exiting to the non-JITted interpreter. YJIT performs better when this is higher.
Speedup is relative to interpreted CRuby. So an "MJIT speedup" of 1.21x means MJIT runs at 1.21 times the iters/second of CRuby with JIT disabled.

You can find our benchmark code in the yjit-bench Github repo and the yjit-extra-benchmarks Github repo.
Our benchmark-runner and reporting code is in the yjit-metrics Github repo.

Tested Ruby version for YJIT and No-JIT: ruby 3.2.0dev (2022-05-28T10:22:54Z master 6e3295e554) +YJIT [x86_64-linux]
Tested Ruby version for Ruby latest MJIT: ruby 3.2.0dev (2022-05-28T10:22:54Z master 6e3295e554) +MJIT [x86_64-linux]

¶Benchmark Memory Usage Details

X

bench	CRuby 3.2.0dev mem (MiB)	MJIT mem (MiB)	YJIT 3.2.0dev mem (MiB)	Inline Code	Outlined Code	YJIT Mem overhead
activerecord	64	72	331	1	1	411.4%
hexapdf	269		613	2	1	127.3%
liquid-render	30	35	292	1	1	850.1%
mail	49	53	312	1	1	535.2%
psych-load	40	47	299	1	1	634.3%
railsbench	102	136	383	3	2	275.4%
binarytrees	31	33	288	1	1	829.5%
chunky_png	41	47	301	1	1	619.2%
discourse	389	404	706	6	5	81.6%
erubi	35	43	298	1	1	752.6%
erubi_rails	102	106	374	2	2	267.1%
fannkuchredux	25	28	283	1	1	1019.8%
lee	34	39	298	1	1	768.5%
nbody	25	26	282	1	1	1031.6%
optcarrot	58	67	320	1	1	445.7%
rubykon	50	80	310	1	1	510.8%
30k_ifelse	66	169	375	6	5	461.7%
30k_methods	57	184	331	3	2	474.8%
cfunc_itself	24	27	282	1	1	1044.0%
fib	24	27	282	1	1	1042.9%
getivar	25	27	282	1	1	1023.3%
keyword_args	24	28	282	1	1	1036.8%
respond_to	24	27	282	1	1	1034.3%
setivar	25	26	282	1	1	1031.4%
str_concat	112	93	354	1	1	216.8%

Memory is shown in mebibytes (1024 * 1024 bytes.)

Older YJIT allocated an additional 256MiB for generated code. Current YJIT allocates executable memory on demand, so this overhead should no longer be present.

¶Number of Iterations and Warmups Tested

Different Ruby configurations want different amounts of warmup. With no JIT, CRuby needs hardly any. YJIT and MJIT 3.0 both warm up quite quickly, while MJIT in 3.1 often slows down for a time as it compiles, after an unpredictable delay.

¶Benchmark YJIT Stats

Hover your cursor over the benchmark names for descriptions of each benchmark.

YJIT stats correspond to the YJIT stats exit report.

Note: currently, all stats are collected on x86_64, not ARM.

¶Raw JSON data files

All graphs and table data in this page comes from processing these data files, which come from benchmark runs.

• prod_ruby_no_jit
• prod_ruby_with_mjit
• prod_ruby_with_yjit
• yjit_stats