Benchmark #

Intuitions about which code is faster are often wrong. String concatenation with + or with <<? Array#include? or Set#include?? map then select or filter_map? Without real measurement, you’re just guessing — and guesses are often wrong. The Benchmark module in Ruby’s standard library provides the proper way to measure code execution time and compare several implementations head-to-head. It displays user, system, and real time, and normalizes the results for fair comparison. This article covers the entire Benchmark API, how to interpret the results, and — just as importantly — how to write valid benchmarks so the results truly reflect production performance.

Basic Benchmarking #

require "benchmark"

# Benchmark.measure — measure the time of one code block
result = Benchmark.measure do
  100_000.times { "hello" + " " + "world" }
end
puts result
#   0.089432   0.001234   0.090666 (  0.090789)
#   │          │          │         │
#   user time  sys time   total     real (wall clock)

# The displayed columns:
# user time  — CPU time for user-space code
# sys time   — CPU time for system calls
# total      — user + sys
# real time  — wall clock time, including I/O wait, GC, etc.

flowchart LR
    A["Benchmark.measure { code }"] --> B[Benchmark::Tms]
    B --> C["utime — user CPU time"]
    B --> D["stime — system CPU time"]
    B --> E["total — utime + stime"]
    B --> F["real — wall clock time"]

    G["Benchmark.bm { |x| x.report }"] --> H[Comparison table]
    I["Benchmark.bmbm { |x| x.report }"] --> J[Warm-up + table]
    K["Benchmark.realtime { code }"] --> L["Float — seconds"]

Benchmark.bm — Comparing Implementations #

Benchmark.bm is the most frequently used method — it displays the results of several implementations in an easy-to-read table format.

require "benchmark"

n = 100_000

Benchmark.bm do |x|
  x.report("string +:") { n.times { "hello" + " " + "world" } }
  x.report("string <<:") { n.times { +"hello" << " " << "world" } }
  x.report("interpolation:") { n.times { "hello #{"world"}" } }
end

#                   user     system      total        real
# string +:       0.089432   0.000000   0.089432 (  0.089789)
# string <<:      0.045123   0.000000   0.045123 (  0.045234)
# interpolation:  0.032456   0.000000   0.032456 (  0.032567)

Labels and Formatting #

require "benchmark"

n = 500_000

# Labels with a consistent column width
Benchmark.bm(20) do |x|   # 20 = label column width
  x.report("Array#include?:") do
    arr = (1..1000).to_a
    n.times { arr.include?(rand(1000)) }
  end

  x.report("Set#include?:") do
    require "set"
    set = Set.new(1..1000)
    n.times { set.include?(rand(1000)) }
  end

  x.report("Hash key lookup:") do
    hash = (1..1000).each_with_object({}) { |i, h| h[i] = true }
    n.times { hash.key?(rand(1000)) }
  end
end

#                         user     system      total        real
# Array#include?:        4.234512   0.000000   4.234512 (  4.235123)
# Set#include?:          0.156789   0.000000   0.156789 (  0.157234)
# Hash key lookup:       0.134567   0.000000   0.134567 (  0.135012)

Benchmark.bmbm — More Accurate Benchmarks #

bmbm runs the benchmark twice: first as a “rehearsal” (warm-up), then once more for the actual results. This matters because Ruby’s garbage collector, JIT compiler, and CPU caches can make the first run look slower than subsequent runs.

require "benchmark"

n = 200_000

Benchmark.bmbm do |x|
  x.report("map + select:") do
    (1..n).map { |i| i * 2 }.select { |i| i > n }
  end

  x.report("filter_map:") do
    (1..n).filter_map { |i| i * 2 if i * 2 > n }
  end

  x.report("lazy:") do
    (1..n).lazy.map { |i| i * 2 }.select { |i| i > n }.to_a
  end
end

# Rehearsal -----------------------------------------------
# map + select:    0.089432   0.002345   0.091777 (  0.092123)
# filter_map:      0.045678   0.001234   0.046912 (  0.047234)
# lazy:            0.012345   0.000567   0.012912 (  0.013123)
# -------------------------------------- total: 0.150601sec
#
#                  user     system      total        real
# map + select:    0.087123   0.001234   0.088357 (  0.088678)
# filter_map:      0.044567   0.001123   0.045690 (  0.045901)
# lazy:            0.011234   0.000456   0.011690 (  0.011901)

Use bmbm (not bm) when:
Comparing code that allocates large amounts of memory (because GC can trigger on the first run)
Benchmarking code that benefits from CPU caches (warm-up fills the cache before measuring)
Wanting more stable and reproducible results
For simple benchmarks without GC and cache concerns, bm is sufficient.

Benchmark.realtime #

For simple cases where you just need to know how long something takes (not a comparison), realtime returns a Float in seconds.

require "benchmark"

# realtime — returns wall clock time as a Float
elapsed = Benchmark.realtime do
  sleep(0.5)
  100_000.times { Math.sqrt(rand) }
end

puts "Finished in #{elapsed.round(3)} seconds"
# => Finished in 0.523 seconds

# Useful for production logging
def process_report(data)
  time = Benchmark.realtime do
    @result = compute_report(data)
  end

  logger.info "Report processed in #{(time * 1000).round(1)}ms"
  @result
end

# Or for simple inline profiling
[
  ["sort:", -> { (1..10_000).to_a.shuffle.sort }],
  ["sort_by:", -> { (1..10_000).to_a.shuffle.sort_by { |x| x } }],
  ["sort numeric:", -> { (1..10_000).to_a.shuffle.sort { |a, b| a <=> b } }]
].each do |label, code|
  t = Benchmark.realtime { 100.times { code.call } }
  puts "#{label.ljust(20)} #{(t * 1000).round(2)}ms"
end

Writing Valid Benchmarks #

Poorly written benchmarks give misleading results. Several important rules:

Use Enough Iterations #

require "benchmark"

# ANTI-PATTERN: too few iterations — noise is larger than the signal
Benchmark.bm do |x|
  x.report("too few:") { 10.times { "hello" + "world" } }
end
#   0.000012   0.000000   0.000012 (  0.000013)
# This number is meaningless — too small to measure accurately

# CORRECT: enough iterations for stable measurements (usually > 100ms per benchmark)
Benchmark.bm do |x|
  x.report("enough:") { 1_000_000.times { "hello" + "world" } }
end
#   0.456789   0.001234   0.458023 (  0.459012)
# This number is meaningful

Avoid GC Mid-Benchmark #

require "benchmark"

# ANTI-PATTERN: letting GC run randomly during the benchmark
Benchmark.bm do |x|
  x.report("with GC:") do
    100_000.times { [1, 2, 3].map { |n| n * 2 } }
    # GC can trigger at any time, adding variance
  end
end

# CORRECT: run GC before each benchmark for consistent conditions
require "gc"

Benchmark.bm do |x|
  GC.start; GC.compact if GC.respond_to?(:compact)
  x.report("clean:") do
    GC.disable
    100_000.times { [1, 2, 3].map { |n| n * 2 } }
    GC.enable
  end
end

Make Sure the Code Actually Executes #

require "benchmark"

# ANTI-PATTERN: the Ruby optimizer may remove code whose results are unused
Benchmark.bm do |x|
  x.report("could be optimized:") do
    100_000.times { 2 ** 32 }   # the result isn't stored, may be optimized away
  end
end

# CORRECT: store the result to ensure the code executes
Benchmark.bm do |x|
  x.report("definitely executed:") do
    result = nil
    100_000.times { result = 2 ** 32 }
    result   # make sure the result is used
  end
end

Benchmark Equal Conditions #

require "benchmark"

# ANTI-PATTERN: comparing unequal conditions
arr = (1..1000).to_a

Benchmark.bm do |x|
  # This creates a new Array every time — the Set creation overhead isn't measured
  x.report("Set (overhead included):") do
    set = Set.new(arr)   # Set creation is inside the benchmark!
    100_000.times { set.include?(rand(1000)) }
  end

  x.report("Array (fair):") do
    100_000.times { arr.include?(rand(1000)) }
  end
end

# CORRECT: separate setup from what's measured
require "set"
arr = (1..1000).to_a
set = Set.new(arr)   # setup OUTSIDE the benchmark

Benchmark.bm(15) do |x|
  x.report("Array include?:") { 100_000.times { arr.include?(rand(1000)) } }
  x.report("Set include?:") { 100_000.times { set.include?(rand(1000)) } }
end

Real Benchmark Examples #

String Building Comparison #

require "benchmark"

n = 200_000

Benchmark.bmbm(25) do |x|
  x.report("+ operator:") do
    n.times do
      result = ""
      result = result + "Hello" + ", " + "World" + "!"
    end
  end

  x.report("<< operator:") do
    n.times do
      result = +""
      result << "Hello" << ", " << "World" << "!"
    end
  end

  x.report("interpolation:") do
    n.times { "Hello, World!" }
  end

  x.report("Array join:") do
    n.times { ["Hello", ", ", "World", "!"].join }
  end

  x.report("format/sprintf:") do
    n.times { format("Hello, %s!", "World") }
  end
end

# Typical results:
#                           user     system      total        real
# + operator:           0.456789   0.000000   0.456789 (  0.457012)
# << operator:          0.123456   0.000000   0.123456 (  0.123678)
# interpolation:        0.089123   0.000000   0.089123 (  0.089345)
# Array join:           0.167890   0.000000   0.167890 (  0.168012)
# format/sprintf:       0.234567   0.000000   0.234567 (  0.234789)

Data Structure Comparison #

require "benchmark"
require "set"

n = 100_000
data = (1..10_000).to_a.shuffle

Benchmark.bmbm(20) do |x|
  arr = data.dup
  set = Set.new(data)
  hash = data.each_with_object({}) { |v, h| h[v] = true }

  x.report("Array include?:") do
    n.times { arr.include?(rand(10_000)) }
  end

  x.report("Set include?:") do
    n.times { set.include?(rand(10_000)) }
  end

  x.report("Hash key?:") do
    n.times { hash.key?(rand(10_000)) }
  end

  x.report("Array bsearch:") do
    sorted = arr.sort
    n.times { sorted.bsearch { |x| x >= rand(10_000) } }
  end
end

Benchmarks with Formatted Output #

require "benchmark"

def run_benchmark(title, n: 100_000, &block)
  puts "\n#{title}"
  puts "=" * 50
  puts "Iterations: #{n.to_s.reverse.gsub(/(\d{3})(?=\d)/, '\\1.').reverse}"

  Benchmark.bmbm(20, &block)
end

run_benchmark("Hash vs Array for lookup", n: 500_000) do |x|
  size = 10_000
  arr = (1..size).to_a
  hash = (1..size).each_with_object({}) { |i, h| h[i] = true }

  x.report("Array#include?:") { 500_000.times { arr.include?(rand(size)) } }
  x.report("Hash#key?:") { 500_000.times { hash.key?(rand(size)) } }
end

Interpreting the Results #

Understanding Benchmark output is half the job.

                  user     system      total        real
implementation_a:  0.234567  0.001234   0.235801 (  0.236012)
implementation_b:  0.123456  0.000567   0.124023 (  0.124234)

user time is the CPU time used by your Ruby code. This is the most relevant for comparing pure algorithms.

system time is the CPU time for system calls — I/O, OS memory allocation, and so on. High when the code does many I/O or memory operations.

total is user + system — the overall CPU time.

real time (wall clock time) is the actual elapsed time. It can be higher than total when there’s I/O wait, GC pauses, or context switches. For CPU-bound code, real ≈ total. For I/O-bound code, real » total.

# Interpretation tips:
# 1. Focus on "real" for practical production comparisons
# 2. If real >> total, there's significant I/O wait or GC
# 3. Variance across runs (run several times!) shows reliability
# 4. Relative comparisons are more meaningful than absolute numbers:
#    "Implementation B is 1.9x faster than A" is more meaningful than
#    "Implementation B takes 0.124 seconds"

# Calculating the speedup factor
time_a = Benchmark.realtime { 100_000.times { arr.include?(rand(10_000)) } }
time_b = Benchmark.realtime { 100_000.times { hash.key?(rand(10_000)) } }

speedup = time_a / time_b
puts "Hash is #{speedup.round(1)}x faster than Array for lookups"

Benchmarking in Production #

The Benchmark module suits development and profiling. For performance monitoring in production, a slightly different pattern is more appropriate.

# Logging execution time in production
require "benchmark"

class PerformanceLogger
  def self.measure(operation_name, threshold_ms: 100, &block)
    result = nil
    time = Benchmark.realtime { result = block.call }
    ms = (time * 1000).round(2)

    if ms > threshold_ms
      logger.warn "SLOW: #{operation_name} took #{ms}ms (threshold: #{threshold_ms}ms)"
    else
      logger.debug "#{operation_name}: #{ms}ms"
    end

    result
  end
end

# Usage
users = PerformanceLogger.measure("fetch_users", threshold_ms: 200) do
  User.where(active: true).includes(:profile).to_a
end

PerformanceLogger.measure("send_email_batch", threshold_ms: 5000) do
  users.each { |u| EmailService.send_welcome(u) }
end

Summary #

Benchmark.bmbm for serious comparisons — runs the benchmark twice (rehearsal + actual) to eliminate cold-start, GC, and CPU-cache effects; more accurate than bm for most cases.
Benchmark.realtime for single measurements — returns a Float in seconds; suitable for logging execution time in production or quick profiling.
Enough iterations for meaningful results — benchmarks should run at least 100ms–1s for stable results; too few iterations produce noise dominating the signal.
Setup outside the benchmark block — only measure what you want to compare; data structure initialization, connections, and other preparation must be outside the x.report block.
Compare equal conditions — make sure all implementations start from the same state (data size, state, memory) for a fair comparison.
Focus on relative comparisons — “2x faster” is more meaningful than “0.045 seconds”; absolute numbers change with hardware, but relative ratios are more stable.
“Real time” for production performance — user+sys shows CPU efficiency, but real time shows what users experience; for I/O-bound code the two can differ greatly.
Benchmark first, optimize later — don’t optimize before measuring; intuitions about bottlenecks are often wrong, and premature optimization makes code more complex without real benefit.

← Previous: Tempfile