storage

sqlite_schema

SQLite Schema for Intermediate Simulation Data Storage.

This schema provides efficient storage for simulation updates with:

• Numpy array storage with dtype preservation

• CSR sparse array support via indptr

• Time-series update tracking

• Entity-attribute data model

class AttributeData(**kwargs)

Bases: Base

Stores entity attribute data with support for both uniform and CSR sparse arrays.

attribute_name

data

data_id

entity_group

get_data() → dict

Get attribute data in movici format.

Returns:

Dictionary with ‘data’ key and optionally ‘row_ptr’ for sparse arrays

id

indptr

indptr_id

initial_datasets

property is_sparse: bool

Check if this attribute uses CSR sparse representation.

Returns:

True if sparse, False otherwise

max_val

min_val

updates

class DatasetFormat_(*values)

Bases: str, Enum

Format types for dataset storage.

Matches the format field used in the platform:

• ENTITY_BASED: Entity-oriented JSON data, destructured into numpy arrays

• UNSTRUCTURED: Unstructured JSON data, stored as blob but JSON-loadable

• BINARY: Binary data, stored as blob and passed transparently

BINARY = 'binary'

ENTITY_BASED = 'entity_based'

UNSTRUCTURED = 'unstructured'

class InitialDataset(**kwargs)

Bases: Base

Stores initial dataset snapshots for self-contained database archives.

This allows the database to be a complete simulation record without requiring separate init_data directory.

Supports three storage formats:

• ENTITY_BASED: Destructured into entity groups and attributes (references AttributeData)

• UNSTRUCTURED: JSON blob, loadable but not queryable by attribute

• BINARY: Raw binary blob, passed transparently to consumers

attributes

data

dataset_name

format

id

class InitialDatasetAttribute(**kwargs)

Bases: Base

Junction table linking initial datasets (entity_based format) to their attribute data.

attribute_data

attribute_data_id

initial_dataset

initial_dataset_id

class Metadata(**kwargs)

Bases: Base

id

version

class NumpyArray(**kwargs)

Bases: Base

Store numpy arrays efficiently as binary data with metadata.

Supports both regular and sparse (CSR) array storage.

data

dtype

classmethod from_array(arr: ndarray) → NumpyArray

Create NumpyArray record from numpy array.

Parameters:

arr – NumPy array to store

Returns:

NumpyArray instance

id

shape

to_array() → ndarray

Reconstruct numpy array from stored data.

Returns:

Reconstructed NumPy array

class SimulationDatabase(db_path: str | Path)

Bases: object

High-level interface for storing and retrieving simulation data.

Thread-safe for concurrent writes from multiple workers.

close()

Close database connections.

ensure_metadata()

get_all_initial_datasets() → Dict[str, dict | bytes]

Retrieve all initial datasets from database.

Returns:

Dictionary mapping dataset names to their data

get_dataset_updates(dataset_name: str) → List[dict]

Retrieve all updates for a dataset in chronological order.

Parameters:

dataset_name – Name of the dataset

Returns:

List of updates in movici format

get_datasets() → List[str]

Get list of all dataset names in database.

Returns:

List of dataset names

get_initial_dataset(dataset_name: str) → dict | bytes | None

Retrieve initial dataset from database.

Parameters:

dataset_name – Name of the dataset

Returns:

Dataset data (dict for JSON formats, bytes for binary), or None if not found

get_metadata()

get_session()

get_timestamps(dataset_name: str) → List[int]

Get all timestamps for a dataset.

Parameters:

dataset_name – Name of the dataset

Returns:

List of timestamps in ascending order

get_update_count(dataset_name: str | None = None) → int

Get total number of updates (optionally filtered by dataset).

Parameters:

dataset_name – Optional dataset name to filter by

Returns:

Number of updates

has_initial_datasets() → bool

Check if database contains any initial datasets.

Returns:

True if initial datasets are stored, False otherwise

initialize()

store_initial_dataset(dataset_name: str, dataset_data: dict | bytes, format: DatasetFormat_ = DatasetFormat_.UNSTRUCTURED) → int

Store initial dataset snapshot in database.

This allows the database to be self-contained without requiring separate init_data directory.

Parameters:

• dataset_name – Name of the dataset

• dataset_data – Dataset data - dict for JSON formats, bytes for binary

• format – Storage format (entity_based, unstructured, or binary)

Returns:

Initial dataset ID

store_update(timestamp: int, iteration: int, dataset_name: str, entity_data: dict, origin: str | None = None) → int

Store a simulation update.

Parameters:

• timestamp – Simulation timestamp

• iteration – Iteration number at this timestamp

• dataset_name – Name of the dataset

• entity_data –

  Update data in movici format:

  {"entity_group_name": {"attribute_name": {"data": [...], "row_ptr": [...]}}}
  

• origin – Optional model identifier

Returns:

Update ID

class Update(**kwargs)

Bases: Base

Represents a simulation update at a specific timestamp and iteration.

attributes

dataset_name

id

iteration

origin

timestamp

class UpdateAttribute(**kwargs)

Bases: Base

Junction table linking updates to their attribute data.

attribute_data

attribute_data_id

update

update_id