sparseml.benchmark package¶
Submodules¶
sparseml.benchmark.info module¶
Functionality related to running a benchmark for a given model.
The file is executable and will run a benchmark for the specified model.
[–batch-size BATCH_SIZE] [–iterations ITERATIONS] [–warmup-iterations WARMUP_ITERATIONS] [–framework FRAMEWORK] [–provider PROVIDER] [–device DEVICE] [–save-path SAVE_PATH] [–show-progress]
Run a benchmark for a specific model in an optionally specified framework.
- optional arguments:
- -h, --help
show this help message and exit
- --model MODEL
The model used for inference. Accepts either a path to the directory where the model is saved or a zoo stub.
- --data DATA
The path to the directory where the data is saved.
- --batch-size BATCH_SIZE
The batch size to use for the benchmark. If not specified, will be set to 1.
- --iterations ITERATIONS
The number of iteration steps to use for the benchmark. If not specified, will be set to 0 and go through entire dataset once.
- --warmup-iterations WARMUP_ITERATIONS
The number of warmup iterations to use for the benchmark.
- --framework FRAMEWORK
The framework to use for the benchmark. If not specified, will be automatically detected based on model provided.
- --provider PROVIDER
The inference provider to use for the benchmark. If not specified, will be automatically detected.
- --device DEVICE
The device to use for the benchmark. If not specified, will be automatically detected.
- --save-path SAVE_PATH
A full file path to save the benchmark results to. If not supplied, will print out the benchmark results to the console.
- --show-progress
If specified, will show the progress of the benchmark.
sparseml.benchmark –framework onnx –model zoo:cv/classification/mobilenet_v1-1.0/pytorch/sparseml/imagenet/base-none
sparseml.benchmark –framework onnx –batch-size 32 –iterations 100 –warmup-iterations 10 –model zoo:cv/classification/mobilenet_v1-1.0/pytorch/sparseml/imagenet/base-none
sparseml.benchmark –framework onnx –model ~/downloads/model.onnx
sparseml.benchmark –framework onnx –model ~/downloads/model.onnx –data ~/downloads/sample-inputs
-
class
sparseml.benchmark.info.
BenchmarkRunner
[source]¶ Bases:
abc.ABC
Abstract class for handling running benchmarks with different frameworks.
-
abstract property
batch_size
¶ the batch size
- Type
return
-
property
benchmark_config
¶ The benchmark configuration.
- Type
return
-
abstract property
device
¶ the device
- Type
return
-
abstract property
framework
¶ the framework
- Type
return
-
abstract property
framework_args
¶ the framework args
- Type
return
-
abstract property
framework_info
¶ the framework info
- Type
return
-
abstract property
inference_provider
¶ the inference provider
- Type
return
-
abstract property
iterations
¶ the number of iterations
- Type
return
-
load_data
(data: Any, **kwargs) → Iterable[Any][source]¶ Uses the framework’s load_data method to load the data into an iterable for use in benchmarking.
- Parameters
data – data to load
kwargs – additional arguments to pass to the framework’s load_data method
- Returns
an iterable of the loaded data
-
abstract property
model
¶ the model as ran in the benchmark
- Type
return
-
abstract property
num_cores
¶ the number of cores
- Type
return
-
abstract property
package_versions
¶ the package versions
- Type
return
-
run
(data: Any, desc: str = '', load_data_kwargs: Dict[str, Any] = {}, show_progress: bool = False, *args, **kwargs) → sparseml.benchmark.serialization.BenchmarkResult[source]¶ Runs a benchmark on the given data. Results are serialized together using the BenchmarkResult class.
- Parameters
data – data to use for benchmarking
desc – str to display if show_progress is True
show_progress – whether to show progress
load_data_kwargs – additional arguments to pass to the framework’s load_data method
args – additional arguments to pass to the framework
kwargs – additional arguments to pass to the framework
- Returns
the results of the benchmark run
- Return type
-
abstract
run_batch
(batch: Any, *args, **kwargs) → sparseml.benchmark.serialization.BatchBenchmarkResult[source]¶ Runs a benchmark on a given batch.
- Parameters
batch – the batch to benchmark
args – additional arguments to pass to the framework
kwargs – additional arguments to pass to the framework
-
run_iter
(data: Any, desc: str = '', show_progress: bool = False, load_data_kwargs: Dict[str, Any] = {}, *args, **kwargs) → Iterator[sparseml.benchmark.serialization.BatchBenchmarkResult][source]¶ Iteratively runs a benchmark on the given data. Non warmup iterations results are returned serialized as BatchBenchmarkResult.
- Parameters
data – data to use for benchmarking
desc – str to display if show_progress is True
show_progress – whether to show progress
load_data_kwargs – additional arguments to pass to the framework’s load_data method
args – additional arguments to pass to the framework
kwargs – additional arguments to pass to the framework
- Returns
an iterator of the benchmark results for each batch
- Return type
Iterator[BatchBenchmarkResult]
-
abstract property
warmup_iterations
¶ the number of warmup iterations
- Type
return
-
abstract property
-
sparseml.benchmark.info.
load_and_run_benchmark
(model: Any, data: Any, load: str, save_path: Optional[str] = None)[source]¶ Loads the benchmark configuration from a file or raw json and reruns the benchmark.
If load exists as a path, will read from the file and use that. Otherwise will try to parse the input as a raw json str.
- Parameters
model – model to benchmark
data – data to benchmark
load – Either a file path to a json file or a raw json string.
save_path – path to save the new benchmark results
-
sparseml.benchmark.info.
load_benchmark_info
(load: str) → sparseml.benchmark.serialization.BenchmarkInfo[source]¶ Load the benchmark info from a file or raw json. If load exists as a path, will read from the file and use that. Otherwise will try to parse the input as a raw json str.
- Parameters
load (str) – Either a file path to a json file or a raw json string.
- Returns
The loaded benchmark info.
- Return type
-
sparseml.benchmark.info.
save_benchmark_results
(model: Any, data: Any, batch_size: int, iterations: int, warmup_iterations: int, framework: Optional[str], provider: Optional[str] = None, device: Optional[str] = None, save_path: Optional[str] = None, framework_args: Dict[str, Any] = {}, show_progress: bool = False)[source]¶ Saves the benchmark results ran for specific framework. If path is provided, will save to a json file at the path. If path is not provided, will print out the info.
If no framework is provided, will detect the framework based on the model.
- Parameters
model – model to benchmark
data – data to benchmark
batch_size – batch size
iterations – number of iterations
warmup_iterations – number of warmup iterations
framework – the specific framework run the benchmark in
provider – the specific inference provider to use
device – the specific device to use
save_path – path to save the benchmark results
framework_args – additional framework specific arguments to pass to the runner
show_progress – True to show a tqdm bar when running, False otherwise
sparseml.benchmark.serialization module¶
Functionality related to serialization of the benchmarking run.
-
class
sparseml.benchmark.serialization.
BatchBenchmarkResult
(*, batch_time: sparseml.benchmark.serialization.ConstrainedFloatValue, batches_per_second: sparseml.benchmark.serialization.ConstrainedFloatValue, items_per_second: sparseml.benchmark.serialization.ConstrainedFloatValue, ms_per_batch: sparseml.benchmark.serialization.ConstrainedFloatValue, ms_per_item: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_size: sparseml.benchmark.serialization.ConstrainedIntValue)[source]¶ Bases:
pydantic.main.BaseModel
Class for storing the result of one batch of a benchmark run.
Extends pydantic BaseModel class for serialization to and from json in addition to proper type checking on construction.
-
batch_size
: int¶
-
batch_time
: float¶
-
batches_per_second
: float¶
-
classmethod
from_result
(batch_time: float, batch_size: int) → sparseml.benchmark.serialization.BatchBenchmarkResult[source]¶ Creates a serialized BatchBenchmarkResult from a given batch time and batch size.
- Parameters
batch_time – Time to process a batch of data in seconds
batch_size – Batch size of the result
- Returns
A serialized BatchBenchmarkResult
-
items_per_second
: float¶
-
ms_per_batch
: float¶
-
ms_per_item
: float¶
-
-
class
sparseml.benchmark.serialization.
BenchmarkConfig
(*, batch_size: sparseml.benchmark.serialization.ConstrainedIntValue = 1, iterations: sparseml.benchmark.serialization.ConstrainedIntValue = 1, warmup_iterations: sparseml.benchmark.serialization.ConstrainedIntValue = 0, num_cores: sparseml.benchmark.serialization.ConstrainedIntValue = 1, device: str, framework_args: Dict[str, Any] = {}, inference_provider: sparseml.framework.info.FrameworkInferenceProviderInfo)[source]¶ Bases:
pydantic.main.BaseModel
Class for storing the configuration of the benchmark run.
Extends pydantic BaseModel class for serialization to and from json in addition to proper type checking on construction.
-
batch_size
: int¶
-
device
: str¶
-
framework_args
: Dict[str, Any]¶
-
inference_provider
: sparseml.framework.info.FrameworkInferenceProviderInfo¶
-
iterations
: int¶
-
num_cores
: int¶
-
warmup_iterations
: int¶
-
-
class
sparseml.benchmark.serialization.
BenchmarkInfo
(*, framework: sparseml.base.Framework, package_versions: Dict[str, Optional[str]], benchmark: sparseml.benchmark.serialization.BenchmarkResult, config: sparseml.benchmark.serialization.BenchmarkConfig)[source]¶ Bases:
pydantic.main.BaseModel
Class for storing the information of the benchmark run. Include configurations, results, and package versioning.
Extends pydantic BaseModel class for serialization to and from json in addition to proper type checking on construction.
-
framework
: sparseml.base.Framework¶
-
package_versions
: Dict[str, Optional[str]]¶
-
-
class
sparseml.benchmark.serialization.
BenchmarkResult
(*, results: List[sparseml.benchmark.serialization.BatchBenchmarkResult] = [], batch_times_mean: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_times_median: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_times_std: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_times_median_90: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_times_median_95: sparseml.benchmark.serialization.ConstrainedFloatValue, batch_times_median_99: sparseml.benchmark.serialization.ConstrainedFloatValue, items_per_second: sparseml.benchmark.serialization.ConstrainedFloatValue, batches_per_second: sparseml.benchmark.serialization.ConstrainedFloatValue, ms_per_batch: sparseml.benchmark.serialization.ConstrainedFloatValue, ms_per_item: sparseml.benchmark.serialization.ConstrainedFloatValue, num_items: sparseml.benchmark.serialization.ConstrainedIntValue, num_batches: sparseml.benchmark.serialization.ConstrainedIntValue)[source]¶ Bases:
pydantic.main.BaseModel
Class for storing the results of a benchmark run. Includes any statistics of the benchmark batch times.
Extends pydantic BaseModel class for serialization to and from json in addition to proper type checking on construction.
-
batch_times_mean
: float¶
-
batch_times_median
: float¶
-
batch_times_median_90
: float¶
-
batch_times_median_95
: float¶
-
batch_times_median_99
: float¶
-
batch_times_std
: float¶
-
batches_per_second
: float¶
-
classmethod
from_results
(results: List[Union[sparseml.benchmark.serialization.BatchBenchmarkResult, float]], batch_size: Optional[int] = None) → sparseml.benchmark.serialization.BenchmarkResult[source]¶ Creates a serialized BenchmarkResult from a given list of benchmark results. If the results are a list of batch times, the batch size is required.
- Parameters
batch_time – Time to process a batch of data in seconds
batch_size – Batch size of the result
- Returns
A serialized BatchBenchmarkResult
-
items_per_second
: float¶
-
ms_per_batch
: float¶
-
ms_per_item
: float¶
-
num_batches
: int¶
-
num_items
: int¶
-
results
: List[sparseml.benchmark.serialization.BatchBenchmarkResult]¶
-
Module contents¶
Functionality related to running benchmarks across ML frameworks.