plumpy.persistence module

class plumpy.persistence.Bundle(savable: plumpy.persistence.Savable, save_context: Optional[LoadSaveContext] = None, dereference: bool = False)[source]

Bases: dict

unbundle(load_context: Optional[LoadSaveContext] = None)plumpy.persistence.Savable[source]

This method loads the class of the object and calls its recreate_from method passing the positional and keyword arguments.

Parameters

load_context – The optional load context

Returns

An instance of the Savable

class plumpy.persistence.InMemoryPersister(loader: Optional[plumpy.loaders.ObjectLoader] = None)[source]

Bases: plumpy.persistence.Persister

Mainly to be used in testing/debugging

_abc_impl = <_abc_data object>
delete_checkpoint(pid: Hashable, tag: Optional[str] = None)None[source]

Delete a persisted process checkpoint. No error will be raised if the checkpoint does not exist

Parameters
  • pid – the process id of the plumpy.Process

  • tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

delete_process_checkpoints(pid: Hashable)None[source]

Delete all persisted checkpoints related to the given process id

Parameters

pid – the process id of the plumpy.Process

get_checkpoints() → List[plumpy.persistence.PersistedCheckpoint][source]

Return a list of all the current persisted process checkpoints with each element containing the process id and optional checkpoint tag

Returns

list of PersistedCheckpoint

get_process_checkpoints(pid: Hashable) → List[plumpy.persistence.PersistedCheckpoint][source]

Return a list of all the current persisted process checkpoints for the specified process with each element containing the process id and optional checkpoint tag

Parameters

pid – the process pid

Returns

list of PersistedCheckpoint tuples

load_checkpoint(pid: Hashable, tag: Optional[str] = None)plumpy.persistence.Bundle[source]

Load a process from a persisted checkpoint by its process id

Parameters
  • pid – the process id of the plumpy.Process

  • tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

Returns

a bundle with the process state

Raises

plumpy.PersistenceError Raised if there was a problem loading the checkpoint

save_checkpoint(process: Process, tag: Optional[str] = None)None[source]

Persist a Process instance

Parameters
  • processplumpy.Process

  • tag – optional checkpoint identifier to allow distinguishing multiple checkpoints for the same process

Raises

plumpy.PersistenceError Raised if there was a problem saving the checkpoint

class plumpy.persistence.LoadSaveContext(loader: Optional[plumpy.loaders.ObjectLoader] = None, **kwargs: Any)[source]

Bases: object

copyextend(**kwargs: Any)plumpy.persistence.LoadSaveContext[source]

Add additional information to the context by making a copy with the new values

class plumpy.persistence.PersistedCheckpoint(pid, tag)

Bases: tuple

_asdict()

Return a new dict which maps field names to their values.

_field_defaults = {}
_fields = ('pid', 'tag')
_fields_defaults = {}
classmethod _make(iterable)

Make a new PersistedCheckpoint object from a sequence or iterable

_replace(**kwds)

Return a new PersistedCheckpoint object replacing specified fields with new values

pid

Alias for field number 0

tag

Alias for field number 1

class plumpy.persistence.Persister[source]

Bases: object

_abc_impl = <_abc_data object>
abstract delete_checkpoint(pid: Hashable, tag: Optional[str] = None)None[source]

Delete a persisted process checkpoint. No error will be raised if the checkpoint does not exist

Parameters
  • pid – the process id of the plumpy.Process

  • tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

abstract delete_process_checkpoints(pid: Hashable)None[source]

Delete all persisted checkpoints related to the given process id

Parameters

pid – the process id of the plumpy.Process

abstract get_checkpoints() → List[plumpy.persistence.PersistedCheckpoint][source]

Return a list of all the current persisted process checkpoints with each element containing the process id and optional checkpoint tag

Returns

list of PersistedCheckpoint

abstract get_process_checkpoints(pid: Hashable) → List[plumpy.persistence.PersistedCheckpoint][source]

Return a list of all the current persisted process checkpoints for the specified process with each element containing the process id and optional checkpoint tag

Parameters

pid – the process pid

Returns

list of PersistedCheckpoint tuples

abstract load_checkpoint(pid: Hashable, tag: Optional[str] = None)plumpy.persistence.Bundle[source]

Load a process from a persisted checkpoint by its process id

Parameters
  • pid – the process id of the plumpy.Process

  • tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

Returns

a bundle with the process state

Raises

plumpy.PersistenceError Raised if there was a problem loading the checkpoint

abstract save_checkpoint(process: Process, tag: Optional[str] = None)None[source]

Persist a Process instance

Parameters
  • processplumpy.Process

  • tag – optional checkpoint identifier to allow distinguishing multiple checkpoints for the same process

Raises

plumpy.PersistenceError Raised if there was a problem saving the checkpoint

class plumpy.persistence.PicklePersister(pickle_directory: str)[source]

Bases: plumpy.persistence.Persister

Implementation of the abstract Persister class that stores Process states in pickles on a filesystem.

_abc_impl = <_abc_data object>
_pickle_filepath(pid: Hashable, tag: Optional[str] = None)str[source]

Returns the full filepath of the pickle for the given process id and optional checkpoint tag

delete_checkpoint(pid: Hashable, tag: Optional[str] = None)None[source]

Delete a persisted process checkpoint. No error will be raised if the checkpoint does not exist

Parameters
  • pid – the process id of the plumpy.Process

  • tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

delete_process_checkpoints(pid: Hashable)None[source]

Delete all persisted checkpoints related to the given process id

Parameters

pid – the process id of the plumpy.Process

static ensure_pickle_directory(dirpath: str)None[source]

Will attempt to create the directory at dirpath and raise if it fails, except if the exception arose because the directory already existed

get_checkpoints() → List[plumpy.persistence.PersistedCheckpoint][source]

Return a list of all the current persisted process checkpoints with each element containing the process id and optional checkpoint tag

Returns

list of PersistedCheckpoint

get_process_checkpoints(pid: Hashable) → List[plumpy.persistence.PersistedCheckpoint][source]

Return a list of all the current persisted process checkpoints for the specified process with each element containing the process id and optional checkpoint tag

Parameters

pid – the process pid

Returns

list of PersistedCheckpoint

load_checkpoint(pid: Hashable, tag: Optional[str] = None)plumpy.persistence.Bundle[source]

Load a process from a persisted checkpoint by its process id

Parameters
  • pid – the process id of the plumpy.Process

  • tag – optional checkpoint identifier to allow retrieving a specific sub checkpoint for the corresponding process

Returns

a bundle with the process state

static load_pickle(filepath: str) → plumpy.persistence.PersistedPickle[source]

Load a pickle from disk

Parameters

filepath – absolute filepath to the pickle

Returns

the loaded pickle

static pickle_filename(pid: Hashable, tag: Optional[str] = None)str[source]

Returns the relative filepath of the pickle for the given process id and optional checkpoint tag

save_checkpoint(process: Process, tag: Optional[str] = None)None[source]

Persist a process to a pickle on disk

Parameters
  • processplumpy.Process

  • tag – optional checkpoint identifier to allow distinguishing multiple checkpoints for the same process

class plumpy.persistence.Savable[source]

Bases: object

CLASS_NAME: str = 'class_name'
_auto_persist: Optional[Set[str]] = None
_ensure_persist_configured()None[source]
static _get_class_name(saved_state: MutableMapping[str, Any])str[source]
static _get_create_meta(out_state: MutableMapping[str, Any]) → Dict[str, Any][source]
static _get_meta_type(saved_state: MutableMapping[str, Any], name: str) → Any[source]
_get_value(saved_state: MutableMapping[str, Any], name: str, load_context: Optional[plumpy.persistence.LoadSaveContext]) → Union[method, plumpy.persistence.Savable][source]
_persist_configured = False
static _set_class_name(out_state: MutableMapping[str, Any], name: str)None[source]
static _set_meta_type(out_state: MutableMapping[str, Any], name: str, type_spec: Any)None[source]
classmethod auto_persist(*members: str)None[source]
static get_custom_meta(saved_state: MutableMapping[str, Any], name: str) → Any[source]
static load(saved_state: MutableMapping[str, Any], load_context: Optional[plumpy.persistence.LoadSaveContext] = None)plumpy.persistence.Savable[source]

Load a Savable from a saved instance state. The load context is a way of passing runtime data to the object being loaded.

Parameters
  • saved_state – The saved state

  • load_context – Additional runtime state that can be passed into when loading. The type and content (if any) is completely user defined

Returns

The loaded Savable instance

load_instance_state(*args: Any, **kwargs: Any)None
load_members(members: Iterable[str], saved_state: MutableMapping[str, Any], load_context: Optional[plumpy.persistence.LoadSaveContext] = None)None[source]
classmethod persist()None[source]
classmethod recreate_from(saved_state: MutableMapping[str, Any], load_context: Optional[plumpy.persistence.LoadSaveContext] = None)plumpy.persistence.Savable[source]

Recreate a Savable from a saved state using an optional load context.

Parameters
  • saved_state – The saved state

  • load_context – An optional load context

Returns

The recreated instance

save(save_context: Optional[plumpy.persistence.LoadSaveContext] = None) → MutableMapping[str, Any][source]
save_instance_state(*args: Any, **kwargs: Any)None
save_members(members: Iterable[str], out_state: MutableMapping[str, Any])None[source]
static set_custom_meta(out_state: MutableMapping[str, Any], name: str, value: Any)None[source]
class plumpy.persistence.SavableFuture(*, loop=None)[source]

Bases: _asyncio.Future, plumpy.persistence.Savable

A savable future.

_auto_persist: Optional[Set[str]] = {'_result', '_state'}
load_instance_state(saved_state: MutableMapping[str, Any], load_context: plumpy.persistence.LoadSaveContext)None[source]
classmethod recreate_from(saved_state: MutableMapping[str, Any], load_context: Optional[plumpy.persistence.LoadSaveContext] = None)plumpy.persistence.Savable[source]

Recreate a Savable from a saved state using an optional load context.

Parameters
  • saved_state – The saved state

  • load_context – An optional load context

Returns

The recreated instance

save_instance_state(out_state: MutableMapping[str, Any], save_context: plumpy.persistence.LoadSaveContext)None[source]
plumpy.persistence.auto_persist(*members: str) → Callable[[Type[plumpy.persistence.Savable]], Type[plumpy.persistence.Savable]][source]