Multiprocessing

import pyblaze.multiprocessing as xmp

The multiprocessing module provides utilities to easily distribute work across multiple processes. Its sole purpose is to speed up computation on a single machine when it is too costly to switch to a faster language than Python.

Vectorization

class pyblaze.multiprocessing.vectorize.Vectorizer(worker_func, worker_init=None, callback_func=None, num_workers=- 1, **kwargs)

The Vectorizer class ought to be used in cases where a result tensor of size N is filled with values computed in some complex way. The computation of these N computations can then be parallelized over multiple processes.

__init__(worker_func, worker_init=None, callback_func=None, num_workers=- 1, **kwargs)

Initializes a new vectorizer.

Parameters

• worker_func (callable) – The function which receives as input an item of the input to process and outputs a value which ought to be returned.

• worker_init (callable, default: None) – The function receives as input the rank of the worker (i.e. every time this function is called, it is called with a different integer as parameter). Its return values are passed as last parameters to worker_func upon every invocation.

• callback_func (callable, default: None) – A function to call after every item has been processed. Must not need to be a free function as it is called on the main thread.

• num_workers (int, default: -1) – The number of processes to use. If set to -1, it defaults to the number of available cores. If set to 0, everything is executed on the main thread.

• kwargs (keyword arguments) – Additional arguments passed to the worker initialization function.

process(items, *args)

Uses the vectorizer’s worker function in order to process all items in parallel.

Parameters

• items (list of object or iterable of object) – The items to be processed by the workers. If given as iterable only (i.e. it does not support index access), the performance might suffer slightly due to an increased number of synchronizations.

• args (variadic arguments) – Additional arguments passed directly to the worker function.

Returns

The output generated by the worker function for each of the input items.

Return type

list of object