Multicore Applications

See IoTPy/multiprocessing/multicore.py for code

and IoTPy/tests/multicore_test.py for tests and examples.

The examples in the directory listed above have extensive documentation. You may find the example code sufficient to use the concepts described below.

A multicore application is a set of communicating shared-memory processes. We first describe a single shared-memory process and then show how to connect multiple processes together.

SHARED MEMORY PROCESS

A shared memory process is a single Python process. A shared memory process is defined by:

  1. in_stream_names: A list of names or identifiers of its input streams

  2. out_stream_names: A list of names or identifiers of its output streams

  3. connect_sources: A list of connections to sources

  4. connect_actuators: A list of connections to actuators

  5. compute_func: A function that reads the process’ input streams and extends its output streams.

A shared memory process is created by calling the function shared_memory_process.

shared_memory_process(
   compute_func,
   in_stream_names, out_stream_names,
   connect_sources, connect_actuators)

In the section on multiple threads we associate the j-th sensor with the j-th input stream of the computational function, compute_func. Connecting streams by name rather than number can be convenient when dealing with a large number of streams. The j-th name in in_stream_names is the name or identifier associated with the j-th stream in in_streams. Likewise, the j-th name in out_stream_names is the name or identifier associated with the j-th stream in out_streams. The lengths of in_streams and out_streams should equal the lengths of in_stream_names and out_stream_names, respectively. The names of the input and output streams must be unique.

For example, the computational function for a seismic application has 3 input streams fed by accelerometer readings from the east, north and vertical directions. We could give the streams names or numbers as in:

in_stream_names = ['east', 'north', 'vertical']

In this case in_streams[0] is called ‘east’, in_streams[1] is called ‘north’ and in_streams[2] is called ‘vertical.’ If you prefer you can use numbers instead of strings for stream names as in:

in_stream_names = range(3)
A SHARED MEMORY PROCESS

A SHARED MEMORY PROCESS

Connecting sources and actuators

connect_sources is a list of pairs where each pair is: (in_stream_name, s) where in_stream_name is an element of in_stream_names and where s is a source. The data generated by s is placed on the input stream uniquely identified by the name in_stream_name.

connect_actuators is a list of pairs where each pair is: (out_stream_name, actuator) where out_stream_name is an element of out_stream_names and where actuator is an actuator. The data in the stream called out_stream_name is placed in the queue read by actuator.

Example: in_stream_names is [‘temperature’, ‘pressure’], and we have two sources called t_source and p_source. We want to connect t_source to the input stream called ‘temperature’ and connect p_source to the input stream called ‘pressure’. We do so with the following connect statement:

connect_sources = [
   ('temperature', t_source), ('pressure', 'p_source)]

An example of the creation of a shared memory process, proc is shown below. For more examples see IoTPy/IoTPy/tests/multiprocessing/multicore.py

proc=shared_memory_process(
   compute_func=f,
   in_stream_names=['in'],
   out_stream_names=['out'],
   connect_sources=[('in', random_number_source)],
   connect_actuators=[['out', print_from_queue]])
Slide1.jpg

MULTIPROCESS programs: Multicore applications

A multiprocess program consists of a set of shared memory processes that communicate with each other using shared memory. An input stream of a shared memory process can be connected to an output stream of a shared memory process within the same multiprocess program. If an input stream t of a shared memory process p is connected to an output stream u of a shared memory process q, then the elements of u are copied into t.

An input stream of a shared memory process can also get data from a source, and an output stream can send data to an actuator.

Connections between shared memory processes

You create a multicore application in two steps: (1) create shared-memory processes, including their sources and actuators, by calling shared_memory_process(); (2) connect the output streams of the processes to input streams of other processes. A connection is specified by the identity of the sending process, the name of the output stream, the identity of the receiving process, and the name of the input stream. An example of a connection is (p, ‘x’, r, ‘v’) where p and r are stream processes, and ‘x’ is the name of an output stream of stream process p, and ‘v’ is the name of an input stream of steam process r.

The following example shows the connections for a multiprocess program with three shared memory processes, p, q, r, as pictured in the next diagram and three connections. The same name can be used for streams in different shared memory processes; for example 'x', is the name of an output stream of process p and also of an input stream of process q.

Slide4.jpg

Execute a multiprocess application for the above example:

#Create stream processes
p = shared_memory_process(....)
q = shared_memory_process(....)
r = shared_memory_process(....)
# Start and run the multicore application.
mp = Multiprocess(
    processes=[p, q, r],
    connections=[(p, 'y', q, 'x'), 
                 (p, 'x, r, 'y')
                 (q, 'y', r, 'u')]
    )

Running a multiprocess program

You can execute the above program with the statement:

mp.run()

Look at Multiprocess Examples for simple examples.

next: distributed applications

previous: testing AN agent