Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Comparing fuzzer coverage and speed

compare_fuzzers runs repeatable A/B measurements against production-style libFuzzer binaries. It records libFuzzer coverage (cov), features (ft), executions per second, corpus growth, peak RSS, and wall/user/ system time. Complete logs and crash artifacts are retained, and any failed or unparseable run makes the command fail.

The examples use uv run to invoke the entry point from the managed project environment. Omit uv run after activating another environment where the project is installed.

Build the baseline and candidate with the same curl revision, compiler, sanitizer, and OSS-Fuzz settings. For example, use two OSS-Fuzz output directories produced with the address sanitizer and libFuzzer. Then run:

uv run compare_fuzzers \
  --baseline-dir /path/to/baseline/out/curl \
  --candidate-dir /path/to/candidate/out/curl \
  --seconds 30 \
  --repeats 5 \
  --cpu 0 \
  --output build/measurements/comparison.json

The defaults cover the fixed structured fast HTTP, deep HTTP, HTTPS, WS, WSS, TELNET, FTP, TFTP, API, and timing lanes. The deep HTTP target retains stateful authentication, redirect, upload, MIME, and result-probe work so the ordinary HTTP lane can concentrate its CPU budget on single-request URL and response parsing. The original mixed-semantics curl_fuzzer_proto target is retained for OSS-Fuzz corpus and testcase compatibility, but is not benchmarked by default because its variable scheme and timing policy obscures lane-level throughput. Repeat --target to choose a different same-name A/B set:

uv run compare_fuzzers \
  --baseline-dir /path/to/baseline/out/curl \
  --candidate-dir /path/to/candidate/out/curl \
  --target curl_fuzzer_proto_http \
  --target curl_fuzzer_proto_http_deep \
  --target curl_fuzzer_proto \
  --target fuzz_url

To compare differently named targets, use a target pair:

uv run compare_fuzzers \
  --baseline-dir /path/to/baseline/out/curl \
  --candidate-dir /path/to/candidate/out/curl \
  --target-pair curl_fuzzer_proto=curl_fuzzer_proto_http

The label in the report is, for example, curl_fuzzer_proto=>curl_fuzzer_proto_http. Each side of a mapped pair receives its own native corpus and seed archive. Same-name --target comparisons retain the stricter behavior of sharing one content-identical snapshot. --corpus TARGET=PATH overrides are keyed by the actual binary target name, so a mapped pair may supply one override for each side.

For a same-name target, the input snapshot combines the checked-in corpus with the baseline’s <target>_seed_corpus.zip, deduplicating by SHA-256. A mapped pair resolves the checked-in corpus and seed archive independently from its baseline and candidate layouts. Add a downloaded fleet corpus with --public-corpus-root ossfuzz_corpus. Override a target’s inputs completely with one or more --corpus TARGET=PATH arguments; PATH may be a directory, zip archive, or individual input.

Structured targets accept --public-corpus-root only when the root contains .structured-schema-owned-eligibility-v1. Create that marker only after the upstream structured corpora have been replaced and the local root has been refreshed as described in Corpus management. Explicit --corpus overrides remain the caller’s schema/revision assertion.

Proto corpus inheritance is limited to binaries that use the exact same generated profile, currently the GnuTLS and mbedTLS HTTPS variants reusing the OpenSSL HTTPS profile. The typed-option cutover does not treat compatibility or other profile corpora as interchangeable merely because retained protobuf tags can parse. Cross-profile corpora are not reused, and pre-cutover public corpora must be benchmarked with the revision and target that produced them.

Every run gets a fresh copy of one content-addressed snapshot. Repeat N uses the deterministic seed --seed-base + N - 1; baseline/candidate order alternates between repeats to reduce ordering bias. The JSON records corpus and binary hashes, exact managed arguments, host details, individual results, and medians.

Raw cov and ft values are instrumentation-specific coverage and feature counts. They remain recorded for each run, but their cross-build deltas default to null: matching target names, compiler versions, and curl revisions do not prove that two binaries have the same coverage sites and feature semantics. Pass --compare-libfuzzer-counters only when those are known to be identical. Mapped targets always report these deltas as null. Use the source coverage comparison below for coverage parity across different binaries.

Compare curl source coverage

Raw libFuzzer cov and ft counters include each harness and cannot prove source coverage parity between different binaries. The comparison helper can additionally replay the fixed native corpus through standalone LLVM coverage builds and compare curl-relative function and code-region identities.

Create an instrumented build in each source worktree with codecoverage.sh. The default build layout puts binaries under build-coverage/ and curl sources under build-coverage/curl/src/curl_external, which the helper detects:

(cd /path/to/baseline-worktree && ./codecoverage.sh)
(cd /path/to/candidate-worktree && ./codecoverage.sh)

uv run compare_fuzzers \
  --baseline-dir /path/to/baseline-libfuzzer-out \
  --candidate-dir /path/to/candidate-libfuzzer-out \
  --target-pair curl_fuzzer_proto=curl_fuzzer_proto_http \
  --coverage-baseline-dir /path/to/baseline-worktree/build-coverage \
  --coverage-candidate-dir /path/to/candidate-worktree/build-coverage \
  --seconds 60 \
  --repeats 5

The aggregate source-coverage comparison unions all selected target pairs. Benchmark each candidate lane separately when judging whether a change should shift its CPU allocation; supporting lanes may retain coverage without needing to match the primary lane’s throughput.

When curl was supplied with codecoverage.sh -c, pass the source roots explicitly:

uv run compare_fuzzers \
  ... \
  --baseline-curl-source-dir /path/to/baseline-curl \
  --candidate-curl-source-dir /path/to/candidate-curl

Use --llvm-profdata and --llvm-cov for versioned executable names, and --coverage-wall-timeout to change the per-target replay limit. The helper validates all binaries and tools before fuzzing, requires byte-identical curl lib/ and src/ source trees, and rejects a coverage binary that produces no profile. This prevents line-number comparisons across different curl revisions from looking meaningful.

The main JSON gains a source_coverage object with per-target and aggregate summaries. Its comparison records baseline-retention percentages plus complete baseline_only and candidate_only function/region lists. Raw profiles, merged profiles, exports, and replay logs are preserved below <log-dir>/source-coverage/. The source metric deliberately replays a fixed input snapshot; discoveries from the timed fuzzing runs do not leak into it. Timer and event-loop scheduling can still move an occasional region between runs, so repeat the replay before treating a marginal one-region gap as real.

For a conventional combined HTML report, a single target can still be replayed after an instrumented build exists:

BUILD_DIR="$PWD/build-coverage" \
TARGETS=curl_fuzzer_proto_http \
scripts/run_coverage.sh

Use source coverage and libFuzzer features together when evaluating corpus minimization or harness changes: line coverage alone does not preserve value, counter, and indirect-call features used by the mutation scheduler.