sparsify.models package

Submodules

sparsify.models.base module

Base DB model classes for the server

class sparsify.models.base.BaseCreatedModifiedModel(*args, **kwargs)[source]

Bases: sparsify.models.base.BaseModel

Base peewee model that includes created and modified timestamp functionality

DoesNotExist: alias of sparsify.models.base.BaseCreatedModifiedModelDoesNotExist

created = <DateTimeField: BaseCreatedModifiedModel.created>

id = <AutoField: BaseCreatedModifiedModel.id>

modified = <DateTimeField: BaseCreatedModifiedModel.modified>

save(*args, **kwargs)[source]

class sparsify.models.base.BaseModel(*args, **kwargs)[source]

Bases: peewee.Model

Base peewee model all DB models must extend from

DoesNotExist: alias of sparsify.models.base.BaseModelDoesNotExist

id = <AutoField: BaseModel.id>

refresh()[source]: Refresh the data for the model instance from the DB

class sparsify.models.base.CSVField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)[source]

Bases: peewee.TextField

CSV field for handling lists of strings in a peewee database

db_value(value)[source]

python_value(value)[source]

class sparsify.models.base.CSVFloatField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)[source]

Bases: sparsify.models.base.CSVField

CSV field for handling lists of floats in a peewee database

python_value(value)[source]

class sparsify.models.base.CSVIntField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)[source]

Bases: sparsify.models.base.CSVField

CSV field for handling lists of integers in a peewee database

python_value(value)[source]

class sparsify.models.base.FileStorage[source]

Bases: object

Class for handling local file storage and the path that is located at. Used for storing large files that would not be good in the DB such as model and data files.

init(root_path: str)[source]

Initialize the file storage class for a given path

Parameters: root_path – the root path on the local file system for where to store files

property root_path

the root path on the local file system for where to store files

Type: return

class sparsify.models.base.ListObjField(json_dumps=None, json_loads=None, **kwargs)[source]

Bases: playhouse.sqlite_ext.JSONField

Field for handling lists of objects in a peewee database

db_value(value)[source]

python_value(value)[source]

sparsify.models.jobs module

DB model classes for jobs

class sparsify.models.jobs.Job(*args, **kwargs)[source]

Bases: sparsify.models.base.BaseModel

DB model for a project’s job.

DoesNotExist: alias of sparsify.models.jobs.JobDoesNotExist

baseprojectprofile_set

created = <DateTimeField: Job.created>

error = <TextField: Job.error>

job_id = <CharField: Job.job_id>

modified = <DateTimeField: Job.modified>

progress = <JSONField: Job.progress>

project_id = <CharField: Job.project_id>

projectbenchmark_set

projectdata_set

projectlossprofile_set

projectmodel_set

projectperfprofile_set

save(*args, **kwargs)[source]: Override for peewee save function to update the modified date

status = <JobStatusField: Job.status>

type_ = <CharField: Job.type_>

worker_ack = <BooleanField: Job.worker_ack>

worker_args = <JSONField: Job.worker_args>

class sparsify.models.jobs.JobStatus(value)[source]

Bases: enum.Enum

Enumerator class for tracking the status of jobs

canceled = 'canceled'

canceling = 'canceling'

completed = 'completed'

error = 'error'

pending = 'pending'

started = 'started'

class sparsify.models.jobs.JobStatusField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)[source]

Bases: peewee.Field

peewee DB field for saving and loading JobStatus from the database

db_value(value: sparsify.models.jobs.JobStatus)[source]

field_type = 'VARCHAR'

python_value(value: str)[source]

sparsify.models.projects module

DB model classes for a project and it’s nested files

class sparsify.models.projects.BaseProjectModel(*args, **kwargs)[source]

Bases: sparsify.models.base.BaseModel

DoesNotExist: alias of sparsify.models.projects.BaseProjectModelDoesNotExist

abstract delete_filesystem()[source]: Delete the state from the local file system

id = <AutoField: BaseProjectModel.id>

abstract setup_filesystem()[source]: Setup the local file system so that it can be used with the data

abstract validate_filesystem()[source]: Validate that the local file system and expected files are correct and exist

class sparsify.models.projects.Project(*args, **kwargs)[source]

Bases: sparsify.models.projects.BaseProjectModel

DB model for a project’s data file. A project may have multiple data files stored in the DB.

DoesNotExist: alias of sparsify.models.projects.ProjectDoesNotExist

benchmarks

created = <DateTimeField: Project.created>

data

delete_filesystem()[source]: Delete the folder from the local file system containing all of the files for the project

description = <TextField: Project.description>

property dir_path

the local directory path for where project’s files are stored

Type: return

property dir_size: Size of the folder on the local file system containing all of the files for the project

models

modified = <DateTimeField: Project.modified>

name = <TextField: Project.name>

optims

profiles_loss

profiles_perf

project_id = <CharField: Project.project_id>

save(*args, **kwargs)[source]: Override for peewee save function to update the modified date

setup_filesystem()[source]: Setup the local file system so that it can be used with the data

training_epochs = <IntegerField: Project.training_epochs>

training_lr_final = <FloatField: Project.training_lr_final>

training_lr_init = <FloatField: Project.training_lr_init>

training_optimizer = <TextField: Project.training_optimizer>

validate_filesystem()[source]: Validate that the local file system and expected files are correct and exist

sparsify.models.projects_benchmark module

DB model classes for project’s benchmark

class sparsify.models.projects_benchmark.ProjectBenchmark(*args, **kwargs)[source]

Bases: sparsify.models.base.BaseCreatedModifiedModel

DB model for a project’s benchmark

DoesNotExist: alias of sparsify.models.projects_benchmark.ProjectBenchmarkDoesNotExist

batch_sizes = <JSONField: ProjectBenchmark.batch_sizes>

benchmark_id = <CharField: ProjectBenchmark.benchmark_id>

core_counts = <JSONField: ProjectBenchmark.core_counts>

created = <DateTimeField: ProjectBenchmark.created>

inference_models = <ListObjField: ProjectBenchmark.inference_models>

instruction_sets = <JSONField: ProjectBenchmark.instruction_sets>

iterations_per_check = <IntegerField: ProjectBenchmark.iterations_per_check>

job = <ForeignKeyField: ProjectBenchmark.job>

job_id = <ForeignKeyField: ProjectBenchmark.job>

modified = <DateTimeField: ProjectBenchmark.modified>

name = <TextField: ProjectBenchmark.name>

project = <ForeignKeyField: ProjectBenchmark.project>

project_id = <ForeignKeyField: ProjectBenchmark.project>

result = <JSONField: ProjectBenchmark.result>

source = <TextField: ProjectBenchmark.source>

warmup_iterations_per_check = <IntegerField: ProjectBenchmark.warmup_iterations_per_check>

sparsify.models.projects_data module

DB model classes for a project’s sample data files

class sparsify.models.projects_data.ProjectData(*args, **kwargs)[source]

Bases: sparsify.models.projects.BaseProjectModel

DB model for a project’s data file. A project may have multiple data files stored in the DB.

DoesNotExist: alias of sparsify.models.projects_data.ProjectDataDoesNotExist

created = <DateTimeField: ProjectData.created>

data_id = <CharField: ProjectData.data_id>

delete_filesystem()[source]: Delete the data file from the local file system

property dir_path

the local directory path for where the data is stored

Type: return

file = <TextField: ProjectData.file>

property file_path

the local file path to the data file

Type: return

job = <ForeignKeyField: ProjectData.job>

job_id = <ForeignKeyField: ProjectData.job>

project = <ForeignKeyField: ProjectData.project>

project_id = <ForeignKeyField: ProjectData.project>

setup_filesystem()[source]: Setup the local file system so that it can be used with the data

source = <TextField: ProjectData.source>

validate_filesystem()[source]: Validate that the local file system and expected files are correct and exist

sparsify.models.projects_model module

DB model classes for a project’s model file

class sparsify.models.projects_model.ProjectModel(*args, **kwargs)[source]

Bases: sparsify.models.projects.BaseProjectModel

DB model for a project’s model file. A project must have only one model file stored in the DB.

DoesNotExist: alias of sparsify.models.projects_model.ProjectModelDoesNotExist

analysis = <JSONField: ProjectModel.analysis>

created = <DateTimeField: ProjectModel.created>

delete_filesystem()[source]: Delete the model file from the local file system

property dir_path

the local directory path for where the model file is stored

Type: return

file = <TextField: ProjectModel.file>

property file_path

the local file path to the data file

Type: return

job = <ForeignKeyField: ProjectModel.job>

job_id = <ForeignKeyField: ProjectModel.job>

model_id = <CharField: ProjectModel.model_id>

project = <ForeignKeyField: ProjectModel.project>

project_id = <ForeignKeyField: ProjectModel.project>

setup_filesystem()[source]: Setup the local file system so that it can be used with the data

source = <TextField: ProjectModel.source>

validate_filesystem()[source]: Validate that the local file system and expected files are correct and exist

sparsify.models.projects_optimizations module

DB model classes for project’s optimizations and modifiers

class sparsify.models.projects_optimizations.ProjectOptimization(*args, **kwargs)[source]

Bases: sparsify.models.base.BaseCreatedModifiedModel

DB model for a project’s optimization (stores modifier settings). A project may have multiple optimizations stored in the DB.

DoesNotExist: alias of sparsify.models.projects_optimizations.ProjectOptimizationDoesNotExist

created = <DateTimeField: ProjectOptimization.created>

end_epoch = <FloatField: ProjectOptimization.end_epoch>

lr_schedule_modifiers

modified = <DateTimeField: ProjectOptimization.modified>

name = <TextField: ProjectOptimization.name>

notes = <TextField: ProjectOptimization.notes>

optim_id = <CharField: ProjectOptimization.optim_id>

profile_loss = <ForeignKeyField: ProjectOptimization.profile_loss>

profile_loss_id = <ForeignKeyField: ProjectOptimization.profile_loss>

profile_perf = <ForeignKeyField: ProjectOptimization.profile_perf>

profile_perf_id = <ForeignKeyField: ProjectOptimization.profile_perf>

project = <ForeignKeyField: ProjectOptimization.project>

project_id = <ForeignKeyField: ProjectOptimization.project>

pruning_modifiers

quantization_modifiers

start_epoch = <FloatField: ProjectOptimization.start_epoch>

trainable_modifiers

class sparsify.models.projects_optimizations.ProjectOptimizationModifierLRSchedule(*args, **kwargs)[source]

Bases: sparsify.models.base.BaseCreatedModifiedModel

DB model for a project’s learning rate schedule modifier.

DoesNotExist: alias of sparsify.models.projects_optimizations.ProjectOptimizationModifierLRScheduleDoesNotExist

created = <DateTimeField: ProjectOptimizationModifierLRSchedule.created>

end_epoch = <FloatField: ProjectOptimizationModifierLRSchedule.end_epoch>

final_lr = <FloatField: ProjectOptimizationModifierLRSchedule.final_lr>

init_lr = <FloatField: ProjectOptimizationModifierLRSchedule.init_lr>

lr_mods = <ListObjField: ProjectOptimizationModifierLRSchedule.lr_mods>

modified = <DateTimeField: ProjectOptimizationModifierLRSchedule.modified>

modifier_id = <CharField: ProjectOptimizationModifierLRSchedule.modifier_id>

optim = <ForeignKeyField: ProjectOptimizationModifierLRSchedule.optim>

optim_id = <ForeignKeyField: ProjectOptimizationModifierLRSchedule.optim>

start_epoch = <FloatField: ProjectOptimizationModifierLRSchedule.start_epoch>

class sparsify.models.projects_optimizations.ProjectOptimizationModifierPruning(*args, **kwargs)[source]

Bases: sparsify.models.base.BaseCreatedModifiedModel

DB model for a project’s optimization pruning modifier.

DoesNotExist: alias of sparsify.models.projects_optimizations.ProjectOptimizationModifierPruningDoesNotExist

balance_perf_loss = <FloatField: ProjectOptimizationModifierPruning.balance_perf_loss>

compression = <FloatField: ProjectOptimizationModifierPruning.compression>

created = <DateTimeField: ProjectOptimizationModifierPruning.created>

end_epoch = <FloatField: ProjectOptimizationModifierPruning.end_epoch>

est_loss_sensitivity = <FloatField: ProjectOptimizationModifierPruning.est_loss_sensitivity>

est_perf_sensitivity = <FloatField: ProjectOptimizationModifierPruning.est_perf_sensitivity>

est_recovery = <FloatField: ProjectOptimizationModifierPruning.est_recovery>

est_time = <FloatField: ProjectOptimizationModifierPruning.est_time>

est_time_baseline = <FloatField: ProjectOptimizationModifierPruning.est_time_baseline>

est_time_gain = <FloatField: ProjectOptimizationModifierPruning.est_time_gain>

filter_min_perf_gain = <FloatField: ProjectOptimizationModifierPruning.filter_min_perf_gain>

filter_min_recovery = <FloatField: ProjectOptimizationModifierPruning.filter_min_recovery>

filter_min_sparsity = <FloatField: ProjectOptimizationModifierPruning.filter_min_sparsity>

flops = <FloatField: ProjectOptimizationModifierPruning.flops>

flops_baseline = <FloatField: ProjectOptimizationModifierPruning.flops_baseline>

flops_gain = <FloatField: ProjectOptimizationModifierPruning.flops_gain>

mask_type = <TextField: ProjectOptimizationModifierPruning.mask_type>

modified = <DateTimeField: ProjectOptimizationModifierPruning.modified>

modifier_id = <CharField: ProjectOptimizationModifierPruning.modifier_id>

nodes = <ListObjField: ProjectOptimizationModifierPruning.nodes>

optim = <ForeignKeyField: ProjectOptimizationModifierPruning.optim>

optim_id = <ForeignKeyField: ProjectOptimizationModifierPruning.optim>

params = <FloatField: ProjectOptimizationModifierPruning.params>

params_baseline = <FloatField: ProjectOptimizationModifierPruning.params_baseline>

sparsity = <FloatField: ProjectOptimizationModifierPruning.sparsity>

start_epoch = <FloatField: ProjectOptimizationModifierPruning.start_epoch>

update_frequency = <FloatField: ProjectOptimizationModifierPruning.update_frequency>

class sparsify.models.projects_optimizations.ProjectOptimizationModifierQuantization(*args, **kwargs)[source]

Bases: sparsify.models.base.BaseCreatedModifiedModel

DB model for a project’s optimization quantization modifier.

DoesNotExist: alias of sparsify.models.projects_optimizations.ProjectOptimizationModifierQuantizationDoesNotExist

balance_perf_loss = <FloatField: ProjectOptimizationModifierQuantization.balance_perf_loss>

compression = <FloatField: ProjectOptimizationModifierQuantization.compression>

created = <DateTimeField: ProjectOptimizationModifierQuantization.created>

end_epoch = <FloatField: ProjectOptimizationModifierQuantization.end_epoch>

est_loss_sensitivity = <FloatField: ProjectOptimizationModifierQuantization.est_loss_sensitivity>

est_perf_sensitivity = <FloatField: ProjectOptimizationModifierQuantization.est_perf_sensitivity>

est_recovery = <FloatField: ProjectOptimizationModifierQuantization.est_recovery>

est_time = <FloatField: ProjectOptimizationModifierQuantization.est_time>

est_time_baseline = <FloatField: ProjectOptimizationModifierQuantization.est_time_baseline>

est_time_gain = <FloatField: ProjectOptimizationModifierQuantization.est_time_gain>

filter_min_perf_gain = <FloatField: ProjectOptimizationModifierQuantization.filter_min_perf_gain>

filter_min_recovery = <FloatField: ProjectOptimizationModifierQuantization.filter_min_recovery>

flops = <FloatField: ProjectOptimizationModifierQuantization.flops>

flops_baseline = <FloatField: ProjectOptimizationModifierQuantization.flops_baseline>

flops_gain = <FloatField: ProjectOptimizationModifierQuantization.flops_gain>

level = <TextField: ProjectOptimizationModifierQuantization.level>

modified = <DateTimeField: ProjectOptimizationModifierQuantization.modified>

modifier_id = <CharField: ProjectOptimizationModifierQuantization.modifier_id>

nodes = <ListObjField: ProjectOptimizationModifierQuantization.nodes>

optim = <ForeignKeyField: ProjectOptimizationModifierQuantization.optim>

optim_id = <ForeignKeyField: ProjectOptimizationModifierQuantization.optim>

params = <FloatField: ProjectOptimizationModifierQuantization.params>

params_baseline = <FloatField: ProjectOptimizationModifierQuantization.params_baseline>

start_epoch = <FloatField: ProjectOptimizationModifierQuantization.start_epoch>

class sparsify.models.projects_optimizations.ProjectOptimizationModifierTrainable(*args, **kwargs)[source]

Bases: sparsify.models.base.BaseCreatedModifiedModel

DB model for a project’s optimization trainable modifier.

DoesNotExist: alias of sparsify.models.projects_optimizations.ProjectOptimizationModifierTrainableDoesNotExist

created = <DateTimeField: ProjectOptimizationModifierTrainable.created>

end_epoch = <FloatField: ProjectOptimizationModifierTrainable.end_epoch>

modified = <DateTimeField: ProjectOptimizationModifierTrainable.modified>

modifier_id = <CharField: ProjectOptimizationModifierTrainable.modifier_id>

nodes = <ListObjField: ProjectOptimizationModifierTrainable.nodes>

optim = <ForeignKeyField: ProjectOptimizationModifierTrainable.optim>

optim_id = <ForeignKeyField: ProjectOptimizationModifierTrainable.optim>

start_epoch = <FloatField: ProjectOptimizationModifierTrainable.start_epoch>

sparsify.models.projects_profiles module

DB model classes for project’s profiles such as performance and loss

class sparsify.models.projects_profiles.BaseProjectProfile(*args, **kwargs)[source]

Bases: sparsify.models.base.BaseModel

Base DB model for project’s profiles such as loss and perf

DoesNotExist: alias of sparsify.models.projects_profiles.BaseProjectProfileDoesNotExist

analysis = <JSONField: BaseProjectProfile.analysis>

created = <DateTimeField: BaseProjectProfile.created>

job = <ForeignKeyField: BaseProjectProfile.job>

job_id = <ForeignKeyField: BaseProjectProfile.job>

name = <TextField: BaseProjectProfile.name>

profile_id = <CharField: BaseProjectProfile.profile_id>

source = <TextField: BaseProjectProfile.source>

class sparsify.models.projects_profiles.ProjectLossProfile(*args, **kwargs)[source]

Bases: sparsify.models.projects_profiles.BaseProjectProfile

DB model for a project’s loss profile. A project may have multiple loss profiles stored in the DB.

DoesNotExist: alias of sparsify.models.projects_profiles.ProjectLossProfileDoesNotExist

analysis = <JSONField: ProjectLossProfile.analysis>

created = <DateTimeField: ProjectLossProfile.created>

job = <ForeignKeyField: ProjectLossProfile.job>

job_id = <ForeignKeyField: ProjectLossProfile.job>

name = <TextField: ProjectLossProfile.name>

profile_id = <CharField: ProjectLossProfile.profile_id>

project = <ForeignKeyField: ProjectLossProfile.project>

project_id = <ForeignKeyField: ProjectLossProfile.project>

projectoptimization_set

pruning_estimation_type = <TextField: ProjectLossProfile.pruning_estimation_type>

pruning_estimations = <BooleanField: ProjectLossProfile.pruning_estimations>

pruning_structure = <TextField: ProjectLossProfile.pruning_structure>

quantized_estimation_type = <TextField: ProjectLossProfile.quantized_estimation_type>

quantized_estimations = <BooleanField: ProjectLossProfile.quantized_estimations>

source = <TextField: ProjectLossProfile.source>

class sparsify.models.projects_profiles.ProjectPerfProfile(*args, **kwargs)[source]

Bases: sparsify.models.projects_profiles.BaseProjectProfile

DB model for a project’s performance profile. A project may have multiple perf profiles stored in the DB

DoesNotExist: alias of sparsify.models.projects_profiles.ProjectPerfProfileDoesNotExist

analysis = <JSONField: ProjectPerfProfile.analysis>

batch_size = <IntegerField: ProjectPerfProfile.batch_size>

core_count = <IntegerField: ProjectPerfProfile.core_count>

created = <DateTimeField: ProjectPerfProfile.created>

instruction_sets = <CSVField: ProjectPerfProfile.instruction_sets>

iterations_per_check = <IntegerField: ProjectPerfProfile.iterations_per_check>

job = <ForeignKeyField: ProjectPerfProfile.job>

job_id = <ForeignKeyField: ProjectPerfProfile.job>

name = <TextField: ProjectPerfProfile.name>

profile_id = <CharField: ProjectPerfProfile.profile_id>

project = <ForeignKeyField: ProjectPerfProfile.project>

project_id = <ForeignKeyField: ProjectPerfProfile.project>

projectoptimization_set

pruning_estimations = <BooleanField: ProjectPerfProfile.pruning_estimations>

quantized_estimations = <BooleanField: ProjectPerfProfile.quantized_estimations>

source = <TextField: ProjectPerfProfile.source>

warmup_iterations_per_check = <IntegerField: ProjectPerfProfile.warmup_iterations_per_check>

sparsify.models.utils module

sparsify.models.utils.database_setup(working_dir: str, app: Optional[flask.app.Flask] = None)[source]

Module contents

DB models and objects for working with sparsify server