movici_simulation_core¶

Subpackages¶

attributes¶

This module contains AttributeSpec objects for common, generic, attribute types that can be used in datasets

csr¶

assert_numeric_array(arr)¶

compare_array(a, b)¶: compare function when both a and b are numpy arrays (and not float arrays)

compare_scalar(a, b)¶: compare function for comparing an array against a scalar :param a: a numpy array :param b: a scalar

csr_binop(data, row_ptr, operand, operator)¶: Perform binary operation operator rowwise on a csr array, the operand must be a 1d array of length equal to the number of rows in the csr array

float_compare(rtol=1e-05, atol=1e-08, equal_nan=True)¶: factory function for creating a float compare function

generate_update(data, row_ptr, mask, changed, undefined)¶

get_new_csr_array(row_lengths, dtype, secondary_shape)¶

get_row(data, row_ptr, index)¶

reduce_rows(data, row_ptr, func)¶

reduce_rows_with_substitute(data, row_ptr, func, substitute)¶

remove_undefined_csr(data: ndarray, row_ptr: ndarray, indices: ndarray, undefined, num_undefined, new_data_shape, compare) → Tuple[ndarray, ndarray, ndarray]¶

row_wise_max(data, row_ptr, empty_row=None)¶

row_wise_min(data, row_ptr, empty_row=None)¶

row_wise_sum(data, row_ptr)¶

rows_contain(data, row_ptr, val, compare)¶

rows_equal(data, row_ptr, row, compare)¶

rows_intersect(data, row_ptr, vals, compare)¶

set_row(data, row_ptr, index, new_row)¶: Set a new row on at the specific index of the csr_array. WARNING: the length of the new row must be allocated in the data array, otherwise this function may override other rows

slice_csr_array(data, row_ptr, indices)¶

update_csr_array(data, row_ptr, upd_data, upd_row_ptr, upd_indices, compare, changes=None)¶: Update a csr array (data and row_ptr) in place with an update csr array (upd_data and upd_row_ptr at the locations upd_indices. data and upd_data must be of the same dtype and may only differ in shape in the first dimension. Can optionally track changes by changes output argument as an boolean array of zeros that has the length equal to the number of rows in of the data csr array ( len( row_ptr)-1). When tracking changes rtol, atol and equal_nan mean the same as in np.isclose

exceptions¶

exception FSMDone¶: Bases: FSMException

exception FSMError¶: Bases: FSMException

exception FSMException¶: Bases: Exception

exception FSMStarted¶: Bases: FSMException

exception InvalidCommand¶: Bases: OrchestratorException

exception InvalidMessage¶: Bases: SimulationException

exception NotReady¶: Bases: SimulationException

exception OrchestratorException¶: Bases: SimulationException

exception SimulationException¶: Bases: Exception

exception StartupFailure¶: Bases: SimulationException

exception StreamDone¶: Bases: SimulationException

messages¶

class AcknowledgeMessage¶

Bases: Message

Response to an NewTimeMessage or QuitMessage

class BaseUpdateMessage¶

Bases: object

address: str | None¶

property has_data¶

key: str | None¶

origin: str | None¶

class ClearDataMessage(prefix: 'str')¶

Bases: Message

prefix: str¶

class DataMessage(data: 'bytes')¶

Bases: Message

data: bytes¶

classmethod from_bytes(raw_message: Sequence[bytes]) → Message¶

size: int¶

to_bytes() → Sequence[bytes]¶

class ErrorMessage(error: 't.Optional[str]' = None)¶

Bases: Message

error: str | None = None¶

class GetDataMessage(key: 'str', mask: 't.Optional[dict]' = None)¶

Bases: Message

key: str¶

mask: dict | None = None¶

class Message¶

Bases: object

classmethod from_bytes(raw_message: Sequence[bytes]) → Message¶

classmethod from_dict(dict_: dict)¶

to_bytes() → Sequence[bytes]¶

class NewTimeMessage(timestamp: 'int')¶

Bases: Message

timestamp: int¶

class PathMessage(path: 't.Optional[Path]')¶

Bases: Message

classmethod from_bytes(raw_message: Sequence[bytes]) → Message¶

path: Path | None¶

to_bytes() → Sequence[bytes]¶

class PutDataMessage(key: 'str', data: 'bytes')¶

Bases: Message

data: bytes¶

classmethod from_bytes(raw_message: Sequence[bytes]) → Message¶

key: str¶

size: int¶

to_bytes() → Sequence[bytes]¶

class QuitMessage(due_to_failure: bool = False)¶

Bases: Message

A message to indicate that we should stop the simulation. Allows models to gracefully terminate.

Parameters:: due_to_failure – flag to indicate that a component has failed and that it is not a regular shutdown

due_to_failure: bool = False¶

class RegistrationMessage(pub: 't.Optional[dict]', sub: 't.Optional[dict]')¶

Bases: Message

pub: dict | None¶

sub: dict | None¶

class ResultMessage(key: str | None = None, address: str | None = None, next_time: int | None = None, origin: str | None = None)¶

Bases: Message, BaseUpdateMessage

Response to an UpdateMessage

address: str | None = None¶

key: str | None = None¶

next_time: int | None = None¶

origin: str | None = None¶

class UpdateMessage(timestamp: 'int', key: 't.Optional[str]' = None, address: 't.Optional[str]' = None, origin: 't.Optional[str]' = None)¶

Bases: Message, BaseUpdateMessage

address: str | None = None¶

key: str | None = None¶

origin: str | None = None¶

timestamp: int¶

class UpdateSeriesMessage(updates: 't.List[UpdateMessage]')¶

Bases: Message

classmethod from_bytes(raw_message: Sequence[bytes]) → Message¶

classmethod from_dict(dict_: dict)¶

property timestamp¶

to_bytes() → Sequence[bytes]¶

updates: List[UpdateMessage]¶

dump_message(message: Message) → Sequence[bytes]¶

load_message(msg_type: bytes, *payload: bytes) → Message¶

settings¶

class Settings(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: EnvPrefixTarget | None = None, _env_file: DotenvType | None = PosixPath('.'), _env_file_encoding: str | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: Mapping[str, str | list[str]] | None = None, _secrets_dir: PathType | None = None, _build_sources: tuple[tuple[PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, *, data_dir: ~typing.Annotated[~pathlib.Path, ~pydantic.types.PathType(path_type=dir)] = PosixPath('.'), loglevel: str = 'INFO', logformat: str = '[{asctime}] [{levelname:8s}] {name:17s}: {message}', name: str = '', storage: ~typing.Literal['api', 'file', 'sqlite'] = 'file', storage_dir: ~pathlib.Path | None = None, temp_dir: ~typing.Annotated[~pathlib.Path, ~pydantic.types.PathType(path_type=dir)] = PosixPath('/tmp'), reference: float = 0, time_scale: float = 1, start_time: int = 0, duration: int = 0, datasets: ~typing.List[dict] = <factory>, model_names: ~typing.List[str] = <factory>, models: ~typing.List[dict] = <factory>, service_types: ~typing.List[str] = <factory>, scenario_config: dict | None = None, service_discovery: ~typing.Dict[str, str] = <factory>, distributed: bool = True)¶

Bases: BaseSettings

apply_scenario_config(config: dict)¶

classmethod check_deprecated_storage(value: str)¶

data_dir: DirectoryPath¶

datasets: t.List[dict]¶

distributed: bool¶

duration: int¶

log_format: str¶

log_level: str¶

model_config = {'arbitrary_types_allowed': True, 'case_sensitive': False, 'cli_avoid_json': False, 'cli_enforce_required': False, 'cli_exit_on_error': True, 'cli_flag_prefix_char': '-', 'cli_hide_none_type': False, 'cli_ignore_unknown_args': False, 'cli_implicit_flags': False, 'cli_kebab_case': False, 'cli_parse_args': None, 'cli_parse_none_str': None, 'cli_prefix': '', 'cli_prog_name': None, 'cli_shortcuts': None, 'cli_use_class_docs_for_groups': False, 'enable_decoding': True, 'env_file': None, 'env_file_encoding': None, 'env_ignore_empty': False, 'env_nested_delimiter': None, 'env_nested_max_split': None, 'env_parse_enums': None, 'env_parse_none_str': None, 'env_prefix': 'movici_', 'env_prefix_target': 'variable', 'extra': 'forbid', 'json_file': None, 'json_file_encoding': None, 'nested_model_default_partial_update': False, 'protected_namespaces': ('model_validate', 'model_dump', 'settings_customise_sources'), 'secrets_dir': None, 'toml_file': None, 'validate_default': True, 'yaml_config_section': None, 'yaml_file': None, 'yaml_file_encoding': None}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_names: t.List[str]¶

models: t.List[dict]¶

name: str¶

reference: float¶

scenario_config: t.Optional[dict]¶

service_discovery: t.Dict[str, str]¶

service_types: t.List[str]¶

start_time: int¶

storage: t.Literal['api', 'file', 'sqlite']¶

storage_dir: t.Optional[Path]¶

temp_dir: DirectoryPath¶

time_scale: float¶

property timeline_info¶

types¶

class CSRAttributeData¶

Bases: UniformAttributeData

data: ndarray¶

ind_ptr: ndarray¶

indptr: ndarray¶

row_ptr: ndarray¶

class DataMask¶

Bases: TypedDict

pub: dict | None¶

sub: dict | None¶

class ExternalSerializationStrategy¶

Bases: object

dumps(data: dict, filetype: FileType, non_data_dict_keys: Sequence[str] | None = None) → bytes¶

loads(raw_data: bytes, type: FileType, non_data_dict_keys: Sequence[str] | None = None) → dict¶

supported_file_type_or_raise(filetype: FileType)¶

supported_file_types() → Sequence[FileType]¶

with_schema(schema: AttributeSchema) → ExternalSerializationStrategy¶

class FileType(*values)¶

Bases: Enum

CSV = ('.csv',)¶

JSON = ('.json',)¶

MSGPACK = ('.msgpack',)¶

NETCDF = ('.nc',)¶

OTHER = ('.dat',)¶

property default_extension¶

classmethod from_extension(ext)¶

class InternalSerializationStrategy(*args, **kwargs)¶

Bases: Protocol[T]

dumps(data: dict) → T¶

loads(raw_data: T) → dict¶

class UniformAttributeData¶

Bases: TypedDict

data: ndarray¶

validate¶

class AttributeSchemaLookup(dataset_names: Sequence[str] | None = None, schema: AttributeSchema | None = None)¶

Bases: MoviciTypeLookup

attribute(attribute_type)¶

class FromDictLookup(datasets: List[dict] | None = None, entity_types: list | None = None, attribute_types: list | None = None, validate_dataset_types: bool = True)¶: Bases: MoviciTypeLookup

class ModelConfigSchema(schema: 'dict | Path', convert_from_previous: 't.Callable[[dict], dict] | None' = None)¶

Bases: object

convert_from_previous: Callable[[dict], dict] | None = None¶

schema: dict | Path¶

class MoviciDataRefInfo(path: tuple[str | int, ...], value: Any, movici_type: Literal['dataset', 'entityGroup', 'attribute'] | None = None)¶

Bases: object

A class containing information regarding a reference in a model config to a Movici object (such as a dataset, entity group or attribute). It can be to set links between a model config and the associated Movici object stored in a database. Instances of MoviciDataRefInfo are returned by validate_and_process() and do generally not need to be created manually.

Parameters:

path – the path of the reference in a JSON structure as a tuple. str values indicate keys in object, while int values represent positions in an array
value – the value of the reference
movici_type – one of "dataset", "entityGroup" or "attribute"

classmethod from_path_string(path: str, value: Any, movici_type: Literal['dataset', 'entityGroup', 'attribute'] | None = None)¶

property json_path¶

movici_type: Literal['dataset', 'entityGroup', 'attribute'] | None = None¶

path: tuple[str | int, ...]¶

set_value(obj)¶

unset_value(obj)¶

value: Any¶

class MoviciTypeLookup(datasets: Mapping[str, str | None] | None = None, entity_types: Collection[str] | None = None, attribute_types: Collection[str] | None = None)¶

Bases: object

class for looking up wether a specific dataset, entity_group, attribute exists or whether a dataset is of a specific type. Used alongside validate_and_process. This class can be subclassed to provide custom logic for determining whether these objects exist

attribute(attribute_type)¶

dataset(dataset_name)¶

dataset_type(dataset_name, required_type)¶

entity_group(entity_type)¶

exception MoviciTypeReport(movici_type: Literal['dataset', 'entityGroup', 'attribute'], instance: str)¶

Bases: ValidationError

Indicates the existence of a movici.type field in the instance. By deriving from exceptions.ValidationError, we hook into the existing jsonschema code that sets the location of the fields. In our own code we process and drop these “errors” so that they are not raised as actual errors

asinfo()¶

movici_type: Literal['dataset', 'entityGroup', 'attribute']¶

anyOf(validator, anyOf, instance, schema)¶

ensure_schema(schema_identifier: dict | str | Path, add_name_and_type=True)¶

extract_reports(errors)¶

get_validation_errors(config, schema)¶

has_dataset_type(instance: str, dataset_type: str, lookup)¶

movici_dataset_type(lookup)¶

movici_type(lookup)¶

movici_validator(schema, lookup: MoviciTypeLookup | None = None)¶

oneOf(validator, oneOf, instance, schema)¶

validate_and_migrate_config(config: dict, versions: Sequence[ModelConfigSchema], add_name_and_type=True)¶

validate_and_process(instance: Any, schema: dict, lookup: MoviciTypeLookup | None = None, return_errors=False) → tuple[list[MoviciDataRefInfo], list[ValidationError]]¶: Extension of jsonschema.validators.validate that strips out and processes MoviciTypeReports

validate_movici_type(instance, movici_type, lookup)¶