Execution Payloads Benchmarks

Requirements

• uv
• Docker

Installation

From Github:

uv tool install --from git+https://github.com/NethermindEth/execution-payloads-benchmarks expb

Usage

1. Create a copy of the example configuration file.
2. Edit the configuration file.
3. Execute one or multiple scenarios.

Generating Payloads

Benchmarks replay the engine_newPayload / engine_forkchoiceUpdated requests that a consensus client sends to an execution client. Generate them for a block range from a Consensus (Beacon) API and an execution RPC:

expb generate-payloads \
  --rpc-url http://localhost:8545 \
  --beacon-url http://localhost:5052 \
  --network mainnet \
  --start-block 21000000 \
  --end-block 21001000 \
  --output-dir ./payloads

The requests are sourced from the beacon block, so the generated payloads, execution requests (EIP-7685), blob versioned hashes and parent beacon block root match exactly what the consensus client sends the execution client — across all forks through Osaka. Both endpoints must serve the requested range (an archive node is required for older ranges). This produces payloads.jsonl and fcus.jsonl, consumed by scenario execution and by send-payloads.

Scenarios Execution

Single Scenario

expb execute-scenario --scenario-name example --config-file expb.yaml

Multiple Scenarios

expb execute-scenarios --config-file expb.yaml --loop

Single Scenario with duration metrics per payload

expb execute-scenario --scenario-name example --config-file expb.yaml --per-payload-metrics

Single Scenario with per-payload metric table logs

expb execute-scenario --scenario-name example --config-file expb.yaml --print-logs --per-payload-metrics-logs

For other use cases or more details see CLI usage docs.

Outputs

After the execution of an scenario the outputs will be stored inside the specified paths.outputs configuration option. The outputs directory structure will looks like this:

• Scenario output directory: <outputs-directory>/expb-executor-<scenario-name>-<timestamp>.
  • Grafana K6 script generated for the test: k6-script.js.
  • Grafana K6 script configuration generated for the test: k6-config.json.
  • Grafana K6 results summary: k6-summary.json.
  • Grafana K6 process logs: k6.log.
  • Grafana Alloy generated configuration: config.alloy.
  • Execution client logs: <client_type>.log.
  • Execution client additional docker volumes directory for any additional volume thats requires it: volumes.
    • If no source directory/file is specified then a <volume-name> directory will be created.
  • Execution client additional commands outputs directory if required: commands
    • Additional commands outputs: cmd-<index>.log

Streamed Outputs

In addition to the previous outputs, metrics about clients and its performance are streamed based in the configured outputs. These includes:

• Grafana K6 results metrics: These are either stored in a k6-results.jsonl file with resultant metrics in the Grafana K6 format or sent to the configured Prometheus Remote Write API.
• Execution client metrics. If a Prometheus service is configured these metrics are sent to the Remote Write API.
• Execution client profiling data: If a Grafana Pyroscope service is configured the profiling data is sent to the configured Pyroscope instance. (WIP)