sparseml.benchmark package

Submodules

sparseml.benchmark.info module

Functionality related to running a benchmark for a given model.

The file is executable and will run a benchmark for the specified model.

[–batch-size BATCH_SIZE] [–iterations ITERATIONS] [–warmup-iterations WARMUP_ITERATIONS] [–framework FRAMEWORK] [–provider PROVIDER] [–device DEVICE] [–save-path SAVE_PATH] [–show-progress]

Run a benchmark for a specific model in an optionally specified framework.

optional arguments:
-h, --help

show this help message and exit

--model MODEL

The model used for inference. Accepts either a path to the directory where the model is saved or a zoo stub.

--data DATA

The path to the directory where the data is saved.

--batch-size BATCH_SIZE

The batch size to use for the benchmark. If not specified, will be set to 1.

--iterations ITERATIONS

The number of iteration steps to use for the benchmark. If not specified, will be set to 0 and go through entire dataset once.

--warmup-iterations WARMUP_ITERATIONS

The number of warmup iterations to use for the benchmark.

--framework FRAMEWORK

The framework to use for the benchmark. If not specified, will be automatically detected based on model provided.

--provider PROVIDER

The inference provider to use for the benchmark. If not specified, will be automatically detected.

--device DEVICE

The device to use for the benchmark. If not specified, will be automatically detected.

--save-path SAVE_PATH

A full file path to save the benchmark results to. If not supplied, will print out the benchmark results to the console.

--show-progress

If specified, will show the progress of the benchmark.

sparseml.benchmark –framework onnx –model zoo:cv/classification/mobilenet_v1-1.0/pytorch/sparseml/imagenet/base-none

sparseml.benchmark –framework onnx –batch-size 32 –iterations 100 –warmup-iterations 10 –model zoo:cv/classification/mobilenet_v1-1.0/pytorch/sparseml/imagenet/base-none

sparseml.benchmark –framework onnx –model ~/downloads/model.onnx

sparseml.benchmark –framework onnx –model ~/downloads/model.onnx –data ~/downloads/sample-inputs

class sparseml.benchmark.info.BenchmarkRunner[source]

Bases: abc.ABC

Abstract class for handling running benchmarks with different frameworks.

abstract property batch_size

the batch size

Type

return

property benchmark_config

The benchmark configuration.

Type

return

abstract property device

the device

Type

return

abstract property framework

the framework

Type

return

abstract property framework_args

the framework args

Type

return

abstract property framework_info

the framework info

Type

return

abstract property inference_provider

the inference provider

Type

return

abstract property iterations

the number of iterations

Type

return

load_data(data: Any, **kwargs) → Iterable[Any][source]

Uses the framework’s load_data method to load the data into an iterable for use in benchmarking.

Parameters

  • data – data to load

  • kwargs – additional arguments to pass to the framework’s load_data method

Returns

an iterable of the loaded data

abstract property model

the model as ran in the benchmark

Type

return

abstract property num_cores

the number of cores

Type

return

abstract property package_versions

the package versions

Type

return

run(data: Any, desc: str = '', load_data_kwargs: Dict[str, Any] = {}, show_progress: bool = False, *args, **kwargs) → sparseml.benchmark.serialization.BenchmarkResult[source]

Runs a benchmark on the given data. Results are serialized together using the BenchmarkResult class.

Parameters

  • data – data to use for benchmarking

  • desc – str to display if show_progress is True

  • show_progress – whether to show progress

  • load_data_kwargs – additional arguments to pass to the framework’s load_data method

  • args – additional arguments to pass to the framework

  • kwargs – additional arguments to pass to the framework

Returns

the results of the benchmark run

Return type

BenchmarkResult

abstract run_batch(batch: Any, *args, **kwargs) → sparseml.benchmark.serialization.BatchBenchmarkResult[source]

Runs a benchmark on a given batch.

Parameters

  • batch – the batch to benchmark

  • args – additional arguments to pass to the framework

  • kwargs – additional arguments to pass to the framework

run_iter(data: Any, desc: str = '', show_progress: bool = False, load_data_kwargs: Dict[str, Any] = {}, *args, **kwargs) → Iterator[sparseml.benchmark.serialization.BatchBenchmarkResult][source]

Iteratively runs a benchmark on the given data. Non warmup iterations results are returned serialized as BatchBenchmarkResult.

Parameters

  • data – data to use for benchmarking

  • desc – str to display if show_progress is True

  • show_progress – whether to show progress

  • load_data_kwargs – additional arguments to pass to the framework’s load_data method

  • args – additional arguments to pass to the framework

  • kwargs – additional arguments to pass to the framework

Returns

an iterator of the benchmark results for each batch

Return type

Iterator[BatchBenchmarkResult]

abstract property warmup_iterations

the number of warmup iterations

Type

return

sparseml.benchmark.info.load_and_run_benchmark(model: Any, data: Any, load: str, save_path: Optional[str] = None)[source]

Loads the benchmark configuration from a file or raw json and reruns the benchmark.

If load exists as a path, will read from the file and use that. Otherwise will try to parse the input as a raw json str.

Parameters

  • model – model to benchmark

  • data – data to benchmark

  • load – Either a file path to a json file or a raw json string.

  • save_path – path to save the new benchmark results

sparseml.benchmark.info.load_benchmark_info(load: str) → sparseml.benchmark.serialization.BenchmarkInfo[source]

Load the benchmark info from a file or raw json. If load exists as a path, will read from the file and use that. Otherwise will try to parse the input as a raw json str.

Parameters

load (str) – Either a file path to a json file or a raw json string.

Returns

The loaded benchmark info.

Return type

FrameworkInfo

sparseml.benchmark.info.save_benchmark_results(model: Any, data: Any, batch_size: int, iterations: int, warmup_iterations: int, framework: Optional[str], provider: Optional[str] = None, device: Optional[str] = None, save_path: Optional[str] = None, framework_args: Dict[str, Any] = {}, show_progress: bool = False)[source]

Saves the benchmark results ran for specific framework. If path is provided, will save to a json file at the path. If path is not provided, will print out the info.

If no framework is provided, will detect the framework based on the model.

Parameters

  • model – model to benchmark

  • data – data to benchmark

  • batch_size – batch size

  • iterations – number of iterations

  • warmup_iterations – number of warmup iterations

  • framework – the specific framework run the benchmark in

  • provider – the specific inference provider to use

  • device – the specific device to use

  • save_path – path to save the benchmark results

  • framework_args – additional framework specific arguments to pass to the runner

  • show_progress – True to show a tqdm bar when running, False otherwise

sparseml.benchmark.serialization module

Functionality related to serialization of the benchmarking run.

class sparseml.benchmark.serialization.BatchBenchmarkResult(*, batch_time: sparseml.benchmark.serialization.ConstrainedFloatValue, batches_per_second: sparseml.benchmark.serialization.ConstrainedFloatValue, items_per_second: sparseml.benchmark.serialization.ConstrainedFloatValue, ms_per_batch: sparseml.benchmark.serialization.ConstrainedFloatValue, ms_per_item: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_size: sparseml.benchmark.serialization.ConstrainedIntValue)[source]

Bases: pydantic.main.BaseModel

Class for storing the result of one batch of a benchmark run.

Extends pydantic BaseModel class for serialization to and from json in addition to proper type checking on construction.

batch_size: int
batch_time: float
batches_per_second: float
classmethod from_result(batch_time: float, batch_size: int) → sparseml.benchmark.serialization.BatchBenchmarkResult[source]

Creates a serialized BatchBenchmarkResult from a given batch time and batch size.

Parameters

  • batch_time – Time to process a batch of data in seconds

  • batch_size – Batch size of the result

Returns

A serialized BatchBenchmarkResult

items_per_second: float
ms_per_batch: float
ms_per_item: float
class sparseml.benchmark.serialization.BenchmarkConfig(*, batch_size: sparseml.benchmark.serialization.ConstrainedIntValue = 1, iterations: sparseml.benchmark.serialization.ConstrainedIntValue = 1, warmup_iterations: sparseml.benchmark.serialization.ConstrainedIntValue = 0, num_cores: sparseml.benchmark.serialization.ConstrainedIntValue = 1, device: str, framework_args: Dict[str, Any] = {}, inference_provider: sparseml.framework.info.FrameworkInferenceProviderInfo)[source]

Bases: pydantic.main.BaseModel

Class for storing the configuration of the benchmark run.

Extends pydantic BaseModel class for serialization to and from json in addition to proper type checking on construction.

batch_size: int
device: str
framework_args: Dict[str, Any]
inference_provider: sparseml.framework.info.FrameworkInferenceProviderInfo
iterations: int
num_cores: int
warmup_iterations: int
class sparseml.benchmark.serialization.BenchmarkInfo(*, framework: sparseml.base.Framework, package_versions: Dict[str, Optional[str]], benchmark: sparseml.benchmark.serialization.BenchmarkResult, config: sparseml.benchmark.serialization.BenchmarkConfig)[source]

Bases: pydantic.main.BaseModel

Class for storing the information of the benchmark run. Include configurations, results, and package versioning.

Extends pydantic BaseModel class for serialization to and from json in addition to proper type checking on construction.

benchmark: sparseml.benchmark.serialization.BenchmarkResult
config: sparseml.benchmark.serialization.BenchmarkConfig
framework: sparseml.base.Framework
package_versions: Dict[str, Optional[str]]
class sparseml.benchmark.serialization.BenchmarkResult(*, results: List[sparseml.benchmark.serialization.BatchBenchmarkResult] = [], batch_times_mean: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_times_median: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_times_std: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_times_median_90: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_times_median_95: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_times_median_99: sparseml.benchmark.serialization.ConstrainedFloatValue, items_per_second: sparseml.benchmark.serialization.ConstrainedFloatValue, batches_per_second: sparseml.benchmark.serialization.ConstrainedFloatValue, ms_per_batch: sparseml.benchmark.serialization.ConstrainedFloatValue, ms_per_item: sparseml.benchmark.serialization.ConstrainedFloatValue, num_items: sparseml.benchmark.serialization.ConstrainedIntValue, num_batches: sparseml.benchmark.serialization.ConstrainedIntValue)[source]

Bases: pydantic.main.BaseModel

Class for storing the results of a benchmark run. Includes any statistics of the benchmark batch times.

Extends pydantic BaseModel class for serialization to and from json in addition to proper type checking on construction.

batch_times_mean: float
batch_times_median: float
batch_times_median_90: float
batch_times_median_95: float
batch_times_median_99: float
batch_times_std: float
batches_per_second: float
classmethod from_results(results: List[Union[sparseml.benchmark.serialization.BatchBenchmarkResult, float]], batch_size: Optional[int] = None) → sparseml.benchmark.serialization.BenchmarkResult[source]

Creates a serialized BenchmarkResult from a given list of benchmark results. If the results are a list of batch times, the batch size is required.

Parameters

  • batch_time – Time to process a batch of data in seconds

  • batch_size – Batch size of the result

Returns

A serialized BatchBenchmarkResult

items_per_second: float
ms_per_batch: float
ms_per_item: float
num_batches: int
num_items: int
results: List[sparseml.benchmark.serialization.BatchBenchmarkResult]

Module contents

Functionality related to running benchmarks across ML frameworks.