ec_plists

plists is a drop-in replacement for module lists, making most list operations parallel. It can operate on each element in parallel, for IO-bound operations, on sublists in parallel, for taking advantage of multi-core machines with CPU-bound operations, and across erlang nodes, for parallizing inside a cluster. It handles errors and node failures. It can be configured, tuned, and tweaked to get optimal performance while minimizing overhead.

Almost all the functions are identical to equivalent functions in lists, returning exactly the same result, and having both a form with an identical syntax that operates on each element in parallel and a form which takes an optional "malt", a specification for how to parallize the operation.

fold is the one exception, parallel fold is different from linear fold. This module also include a simple mapreduce implementation, and the function runmany. All the other functions are implemented with runmany, which is as a generalization of parallel list operations.

Malts =====

A malt specifies how to break a list into sublists, and can optionally specify a timeout, which nodes to run on, and how many processes to start per node.

Malt = MaltComponent | [MaltComponent] MaltComponent = SubListSize::integer() | {processes, integer()} | {processes, schedulers} | {timeout, Milliseconds::integer()} | {nodes, [NodeSpec]}

NodeSpec = Node::atom() | {Node::atom(), NumProcesses::integer()} | {Node::atom(), schedulers}

An integer can be given to specify the exact size for sublists. 1 is a good choice for IO-bound operations and when the operation on each list element is expensive. Larger numbers minimize overhead and are faster for cheap operations.

If the integer is omitted, and you have specified a {processes, X}`, the list is split into X sublists. This is only useful when the time to process each element is close to identical and you know exactly how many lines of execution are available to you. If neither of the above applies, the sublist size defaults to 1. You can use `{processes, X}` to have the list processed by `X` processes on the local machine. A good choice for `X` is the number of lines of execution (cores) the machine provides. This can be done automatically with {processes, schedulers}, which sets the number of processes to the number of schedulers in the erlang virtual machine (probably equal to the number of cores). `{timeout, Milliseconds}` specifies a timeout. This is a timeout for the entire operation, both operating on the sublists and combining the results. exit(timeout) is evaluated if the timeout is exceeded. `{nodes, NodeList}` specifies that the operation should be done across nodes. Every element of NodeList is of the form `{NodeName, NumProcesses}` or NodeName, which means the same as `{NodeName, 1}`. plists runs NumProcesses processes on NodeName concurrently. A good choice for NumProcesses is the number of lines of execution (cores) a node provides plus one. This ensures the node is completely busy even when fetching a new sublist. This can be done automatically with `{NodeName, schedulers}`, in which case plists uses a cached value if it has one, and otherwise finds the number of schedulers in the remote node and adds one. This will ensure at least one busy process per core (assuming the node has a scheduler for each core). plists is able to recover if a node goes down. If all nodes go down, exit(allnodescrashed) is evaluated. Any of the above may be used as a malt, or may be combined into a list. `{nodes, NodeList}` and {processes, X} may not be combined. Examples ======== %%start a process for each element (1-element sublists)< 1 %% start a process for each ten elements (10-element sublists) 10 %% split the list into two sublists and process in two processes {processes, 2} %% split the list into X sublists and process in X processes, %% where X is the number of cores in the machine {processes, schedulers} %% split the list into 10-element sublists and process in two processes [10, {processes, 2}] %% timeout after one second. Assumes that a process should be started %% for each element.<br/> {timeout, 1000} %% Runs 3 processes at a time on apple@desktop, and 2 on orange@laptop %% This is the best way to utilize all the CPU-power of a dual-core<br/> %% desktop and a single-core laptop. Assumes that the list should be<br/> %% split into 1-element sublists.<br/> {nodes, [{apple@desktop, 3}, {orange@laptop, 2}]} %% Like above, but makes plists figure out how many processes to use. {nodes, [{apple@desktop, schedulers}, {orange@laptop, schedulers}]} %% Gives apple and orange three seconds to process the list as<br/> %% 100-element sublists.<br/> [100, {timeout, 3000}, {nodes, [{apple@desktop, 3}, {orange@laptop, 2}]}] Aside: Why Malt? ================ I needed a word for this concept, so maybe my subconsciousness gave me one by making me misspell multiply. Maybe it is an acronym for Malt is A List Tearing Specification. Maybe it is a beer metaphor, suggesting that code only runs in parallel if bribed with spirits. Its jargon, learn it or you can't be part of the in-group.

Messages and Errors ===================

plists assures that no extraneous messages are left in or will later enter the message queue. This is guaranteed even in the event of an error.

Errors in spawned processes are caught and propagated to the calling process. If you invoke

plists:map(fun (X) -> 1/X end, [1, 2, 3, 0]).

you get a badarith error, exactly like when you use lists:map.

plists uses monitors to watch the processes it spawns. It is not a good idea to invoke plists when you are already monitoring processes. If one of them does a non-normal exit, plists receives the 'DOWN' message believing it to be from one of its own processes. The error propagation system goes into effect, which results in the error occuring in the calling process.