.rst .pdf

repository open issue suggest edit

plumpy.persistence module

class plumpy.persistence.Bundle(savable: plumpy.persistence.Savable, save_context: Optional[LoadSaveContext] = None, dereference: bool = False)

Bases: dict

unbundle(load_context: Optional[LoadSaveContext] = None) → plumpy.persistence.Savable

This method loads the class of the object and calls its recreate_from method passing the positional and keyword arguments.

Parameters

load_context – The optional load context

Returns

An instance of the Savable

class plumpy.persistence.InMemoryPersister(loader: Optional[plumpy.loaders.ObjectLoader] = None)

Bases: plumpy.persistence.Persister

Mainly to be used in testing/debugging

_abc_impl = <_abc_data object>

delete_checkpoint(pid: Hashable, tag: Optional[str] = None) → None

Delete a persisted process checkpoint. No error will be raised if the checkpoint does not exist

Parameters

• pid – the process id of the plumpy.Process

• tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

delete_process_checkpoints(pid: Hashable) → None

Delete all persisted checkpoints related to the given process id

Parameters

pid – the process id of the plumpy.Process

get_checkpoints() → List[plumpy.persistence.PersistedCheckpoint]

Return a list of all the current persisted process checkpoints with each element containing the process id and optional checkpoint tag

Returns

list of PersistedCheckpoint

get_process_checkpoints(pid: Hashable) → List[plumpy.persistence.PersistedCheckpoint]

Return a list of all the current persisted process checkpoints for the specified process with each element containing the process id and optional checkpoint tag

Parameters

pid – the process pid

Returns

list of PersistedCheckpoint tuples

load_checkpoint(pid: Hashable, tag: Optional[str] = None) → plumpy.persistence.Bundle

Load a process from a persisted checkpoint by its process id

Parameters

• pid – the process id of the plumpy.Process

• tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

Returns

a bundle with the process state

Raises

plumpy.PersistenceError Raised if there was a problem loading the checkpoint

save_checkpoint(process: Process, tag: Optional[str] = None) → None

Persist a Process instance

Parameters

• process – plumpy.Process

• tag – optional checkpoint identifier to allow distinguishing multiple checkpoints for the same process

Raises

plumpy.PersistenceError Raised if there was a problem saving the checkpoint

class plumpy.persistence.LoadSaveContext(loader: Optional[plumpy.loaders.ObjectLoader] = None, **kwargs: Any)

Bases: object

copyextend(**kwargs: Any) → plumpy.persistence.LoadSaveContext

Add additional information to the context by making a copy with the new values

class plumpy.persistence.PersistedCheckpoint(pid, tag)

Bases: tuple

_asdict()

Return a new dict which maps field names to their values.

_field_defaults = {}

_fields = ('pid', 'tag')

_fields_defaults = {}

classmethod _make(iterable)

Make a new PersistedCheckpoint object from a sequence or iterable

_replace(**kwds)

Return a new PersistedCheckpoint object replacing specified fields with new values

pid

Alias for field number 0

tag

Alias for field number 1

class plumpy.persistence.Persister

Bases: object

_abc_impl = <_abc_data object>

abstract delete_checkpoint(pid: Hashable, tag: Optional[str] = None) → None

Delete a persisted process checkpoint. No error will be raised if the checkpoint does not exist

Parameters

• pid – the process id of the plumpy.Process

• tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

abstract delete_process_checkpoints(pid: Hashable) → None

Delete all persisted checkpoints related to the given process id

Parameters

pid – the process id of the plumpy.Process

abstract get_checkpoints() → List[plumpy.persistence.PersistedCheckpoint]

Return a list of all the current persisted process checkpoints with each element containing the process id and optional checkpoint tag

Returns

list of PersistedCheckpoint

abstract get_process_checkpoints(pid: Hashable) → List[plumpy.persistence.PersistedCheckpoint]

Return a list of all the current persisted process checkpoints for the specified process with each element containing the process id and optional checkpoint tag

Parameters

pid – the process pid

Returns

list of PersistedCheckpoint tuples

abstract load_checkpoint(pid: Hashable, tag: Optional[str] = None) → plumpy.persistence.Bundle

Load a process from a persisted checkpoint by its process id

Parameters

• pid – the process id of the plumpy.Process

• tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

Returns

a bundle with the process state

Raises

plumpy.PersistenceError Raised if there was a problem loading the checkpoint

abstract save_checkpoint(process: Process, tag: Optional[str] = None) → None

Persist a Process instance

Parameters

• process – plumpy.Process

• tag – optional checkpoint identifier to allow distinguishing multiple checkpoints for the same process

Raises

plumpy.PersistenceError Raised if there was a problem saving the checkpoint

class plumpy.persistence.PicklePersister(pickle_directory: str)

Bases: plumpy.persistence.Persister

Implementation of the abstract Persister class that stores Process states in pickles on a filesystem.

_abc_impl = <_abc_data object>

_pickle_filepath(pid: Hashable, tag: Optional[str] = None) → str

Returns the full filepath of the pickle for the given process id and optional checkpoint tag

delete_checkpoint(pid: Hashable, tag: Optional[str] = None) → None

Delete a persisted process checkpoint. No error will be raised if the checkpoint does not exist

Parameters

• pid – the process id of the plumpy.Process

• tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

delete_process_checkpoints(pid: Hashable) → None

Delete all persisted checkpoints related to the given process id

Parameters

pid – the process id of the plumpy.Process

static ensure_pickle_directory(dirpath: str) → None

Will attempt to create the directory at dirpath and raise if it fails, except if the exception arose because the directory already existed

get_checkpoints() → List[plumpy.persistence.PersistedCheckpoint]

Return a list of all the current persisted process checkpoints with each element containing the process id and optional checkpoint tag

Returns

list of PersistedCheckpoint

get_process_checkpoints(pid: Hashable) → List[plumpy.persistence.PersistedCheckpoint]

Return a list of all the current persisted process checkpoints for the specified process with each element containing the process id and optional checkpoint tag

Parameters

pid – the process pid

Returns

list of PersistedCheckpoint

load_checkpoint(pid: Hashable, tag: Optional[str] = None) → plumpy.persistence.Bundle

Load a process from a persisted checkpoint by its process id

Parameters

• pid – the process id of the plumpy.Process

• tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

Returns

a bundle with the process state

static load_pickle(filepath: str) → plumpy.persistence.PersistedPickle

Load a pickle from disk

Parameters

filepath – absolute filepath to the pickle

Returns

the loaded pickle

static pickle_filename(pid: Hashable, tag: Optional[str] = None) → str

Returns the relative filepath of the pickle for the given process id and optional checkpoint tag

save_checkpoint(process: Process, tag: Optional[str] = None) → None

Persist a process to a pickle on disk

Parameters

• process – plumpy.Process

• tag – optional checkpoint identifier to allow distinguishing multiple checkpoints for the same process

class plumpy.persistence.Savable

Bases: object

CLASS_NAME: str = 'class_name'

_auto_persist: Optional[Set[str]] = None

_ensure_persist_configured() → None

static _get_class_name(saved_state: MutableMapping[str, Any]) → str

static _get_create_meta(out_state: MutableMapping[str, Any]) → Dict[str, Any]

static _get_meta_type(saved_state: MutableMapping[str, Any], name: str) → Any

_get_value(saved_state: MutableMapping[str, Any], name: str, load_context: Optional[plumpy.persistence.LoadSaveContext]) → Union[method, plumpy.persistence.Savable]

_persist_configured = False

static _set_class_name(out_state: MutableMapping[str, Any], name: str) → None

static _set_meta_type(out_state: MutableMapping[str, Any], name: str, type_spec: Any) → None

classmethod auto_persist(*members: str) → None

static get_custom_meta(saved_state: MutableMapping[str, Any], name: str) → Any

static load(saved_state: MutableMapping[str, Any], load_context: Optional[plumpy.persistence.LoadSaveContext] = None) → plumpy.persistence.Savable

Load a Savable from a saved instance state. The load context is a way of passing runtime data to the object being loaded.

Parameters

• saved_state – The saved state

• load_context – Additional runtime state that can be passed into when loading. The type and content (if any) is completely user defined

Returns

The loaded Savable instance

load_instance_state(*args: Any, **kwargs: Any) → None

load_members(members: Iterable[str], saved_state: MutableMapping[str, Any], load_context: Optional[plumpy.persistence.LoadSaveContext] = None) → None

classmethod persist() → None

classmethod recreate_from(saved_state: MutableMapping[str, Any], load_context: Optional[plumpy.persistence.LoadSaveContext] = None) → plumpy.persistence.Savable

Recreate a Savable from a saved state using an optional load context.

Parameters

• saved_state – The saved state

• load_context – An optional load context

Returns

The recreated instance

save(save_context: Optional[plumpy.persistence.LoadSaveContext] = None) → MutableMapping[str, Any]

save_instance_state(*args: Any, **kwargs: Any) → None

save_members(members: Iterable[str], out_state: MutableMapping[str, Any]) → None

static set_custom_meta(out_state: MutableMapping[str, Any], name: str, value: Any) → None

class plumpy.persistence.SavableFuture(*, loop=None)

Bases: _asyncio.Future, plumpy.persistence.Savable

A savable future.

_auto_persist: Optional[Set[str]] = {'_result', '_state'}

load_instance_state(saved_state: MutableMapping[str, Any], load_context: plumpy.persistence.LoadSaveContext) → None

classmethod recreate_from(saved_state: MutableMapping[str, Any], load_context: Optional[plumpy.persistence.LoadSaveContext] = None) → plumpy.persistence.Savable

Recreate a Savable from a saved state using an optional load context.

Parameters

• saved_state – The saved state

• load_context – An optional load context

Returns

The recreated instance

save_instance_state(out_state: MutableMapping[str, Any], save_context: plumpy.persistence.LoadSaveContext) → None

plumpy.persistence.auto_persist(*members: str) → Callable[[Type[plumpy.persistence.Savable]], Type[plumpy.persistence.Savable]]

plumpy.mixins module plumpy.ports module