Transparency

Methodology

Every benchmark on Backup Arena is designed to be reproducible. This page documents the exact hardware, software, site profiles, timing rules, and verification checks so you can audit or replicate any result yourself.

Plugin Selection Policy

Backup Arena benchmarks the premium/paid version of every plugin. This ensures each plugin is tested at its full capability — not limited by free-tier restrictions.

If a plugin has no paid version (the free version is the only version), we test that. The goal is to evaluate every plugin at its best.

Benchmark Configuration

Before each backup run, the Worker configures every plugin to maximize scope and normalize compression, creating a level playing field. We never change plugin architecture (single zip vs. multi-zip, streaming vs. batch) -- only settings.

• Every plugin is configured to back up the maximum possible scope (database, plugins, themes, uploads, wp-config, and wp-core where supported).
• Premium plugins with wp-core backup capability have it explicitly enabled via WP-CLI option commands before each run.
• Free plugins that cannot back up wp-core (e.g. All-in-One WP Migration, WPvivid, WP Staging) are noted on results pages with scope limitation badges.
• Compression is normalized to DEFLATE level 6 where configurable. Plugins that use proprietary formats or fixed compression are left at their defaults.
• Plugin architecture (single zip vs. multi-zip, streaming vs. batch, PHP ZipArchive vs. shell zip) is NOT changed -- each plugin runs its own engine.

Server Specifications

Benchmarks run on a dedicated Hetzner VPS that serves no other traffic. The machine is reserved exclusively for benchmark execution to eliminate noisy-neighbor effects.

Docker Environment

Each battle spins up four fresh containers: two WordPress + two MySQL instances, one pair per plugin. Containers run on an internal Docker network with no outbound internet access.

InnoDB Configuration

Each MySQL container uses a custom configuration file mounted at /etc/mysql/conf.d/bench.cnf:

[mysqld]
innodb_buffer_pool_size = 1G
max_allowed_packet = 256M
innodb_flush_log_at_trx_commit = 0

Site Profiles

Seven pre-built WordPress sites cover the full spectrum from a fresh install to enterprise-scale. Each profile is a deterministic snapshot loaded identically for every battle.

Snapshot Format & Checksums

Each site profile is stored as a snapshot archive containing four components:

• wp-content.tar.gz -- compressed uploads, themes, and plugin files
• database.sql.gz -- gzipped MySQL dump
• metadata.json -- profile name, file count, DB size, plugin list
• checksums.json -- SHA-256 hash for every file in the snapshot

Before each battle, the Worker verifies the snapshot's SHA-256 checksum against the published value. The checksum for every profile is displayed in each battle's environment card and can be independently verified by downloading the snapshot.

Timing Methodology

Each battle runs three iterations. Before each iteration, both container pairs are reset to the clean snapshot state. Reset time (~10-20 seconds) is NOT included in timing -- the clock starts when the backup/restore command executes.

Phases

Each plugin's output is parsed into standard phases using a configurable phase parser:

Plugins whose output cannot be broken into phases receive a single BACKUP / RESTORE bar.

Archive Formats & Compression

Different backup plugins use different archive formats, which affects both speed and output size:

Compression directly affects the fairness of speed comparisons. A plugin that skips compression will appear faster (less CPU work) but produces a larger archive. Conversely, a plugin that compresses well takes longer but saves storage and bandwidth.

To address this, Backup Arena shows both raw timing AND storage metrics for every battle: archive size with format badge, compression ratio (archive size vs. raw wp-content), and throughput in MB/s. The three-winner system (Speed, Storage, Best Overall) makes these trade-offs explicit -- a plugin that skips compression wins on speed but loses on storage.

Backup Scope Detection

Not all backup plugins back up the same data. Some include everything (database, plugins, themes, uploads, wp-config.php), while others skip certain categories. This makes speed comparisons unfair if one plugin simply backs up less data.

After each backup completes, Backup Arena inspects the backup output to detect which categories were included. It examines file listings, ZIP/TAR archive contents, and WPRESS file headers, looking for patterns like .sql files (database), plugins/, themes/, uploads/ directories, and wp-config.php.

Results display scope badges (DB, Plugins, Themes, Uploads, wp-config) for each plugin. When the two plugins have different scopes, an amber warning banner appears indicating that results may not be directly comparable. This gives readers the context they need to interpret timing differences fairly.

Winner Determination

Each battle determines three winners:

• Speed Winner -- the plugin with the lowest median total time (backup + restore). If the margin is less than 0.5 seconds, speed is declared a draw.
• Storage Winner -- the plugin that produces the smallest median archive size. If the difference is less than 1 MB, storage is declared a draw.
• Best Overall (Pareto) -- a plugin wins Best Overall when no competitor beats it on BOTH speed AND archive size. This is based on Pareto optimality-- the same approach used by compression benchmarks like Squash and lzbench. If one plugin is faster but the other produces a smaller archive, neither dominates and the result is "Different Strengths."

Speed draws do not affect Elo ratings. Best Overall Elo only changes when one plugin Pareto-dominates the other; "Different Strengths" results count as draws with no Elo change.

Restore Verification

Every restore is verified with four independent checks. If a plugin produces a corrupted restore, it is caught and shown publicly in the battle results.

Result: verified: true/false per plugin per iteration, displayed in every battle result.

Execution Modes

Backup Arena offers two execution modes. Users choose the mode when starting a battle, and the mode is displayed on every result page.

Parallel Mode (default)

Sequential Mode

Security

Benchmark containers are sandboxed to prevent any plugin from affecting the host system or other battles.

• Docker containers run on an --internal network with no outbound internet access
• Plugin commands are parameterized templates -- never executed as raw shell input
• Fresh containers are created for every battle and destroyed afterward
• backup_output_path is validated at three layers: frontend form, API endpoint, and Worker before every docker cp command
• Submitted plugin zips are inspected for path traversal, symlinks, PHP outside expected directories, and zip bombs (decompressed size must be < 10x compressed)
• Plugin submissions are never auto-approved -- each requires manual admin review in an isolated Docker container with no network access

Reproduce Locally

All site profile snapshots are available for download. You can load them into the same Docker setup and run the benchmark scripts yourself to independently verify any result.

# Pull the benchmark containers
docker compose -f docker-compose.bench.yml up bench-wp-a bench-db-a

# Load a snapshot and run a backup
docker exec bench-wp-a wp [plugin] backup ...

# Compare your timings against published results