Acoustics and signal processing Example

This section describes different examples using streams of acoustic data. The first example takes a sound as input and creates an output which adds reverberation to the input sound. This example is an illustration of the use of IoTPy; it is not a discussion of acoustics. The model of acoustics and echoes that we use here is simplistic. These examples were developed by S. Deepak Narayanan and Atishay Jain, IIT (Indian Institute of Technology) Gandhinagar, with Julian Bunn and Mani Chandy at Caltech.

echoes and Reverberation of sound

The diagram shows the agents (boxes) and streams (arrows) of a simple reverberation generator which assumes that sound is generated at the center of a cubic room and a microphone at the center of the room records what is heard. When a sound wave hits the walls of the room it is reflected. Let T be the time (in number of sample points) for sound to traverse from one wall to the opposite wall. An impulse generated at time 0 will cause an echo at time T, 2T, 3T,… as each echo causes a further echo.

The reflected wave is an attenuation of the incoming wave. Let the attenuation factor from all the walls jointly be A. Then the microphone will pick up the original impulse of intensity 1 at time 0 followed by an impulse of magnitude A**n (i.e. A raised to the n-th power) at time n*T for n = 1, 2, 3, … For example, if the attenuation factor is 0.5 and the delay is 4 sample points then the microphone will record [1, 0, 0, 0, 0.5, 0, 0, 0, 0.25, 0, 0, 0, 0.125, ….].

A sound wave may be scattered when it is reflected from a surface, and one gross approximation to scattering is to use a vector of attenuations and represent each echo by a dot product of the incoming wave and the attenuation vector.

Heard sound is the sum of the original sound and the echo. The echo is an attenuated delay of the heard sound.

Heard sound is the sum of the original sound and the echo. The echo is an attenuated delay of the heard sound.

The diagram shows a process that creates the heard sound which includes reverberation added to the original sound. The original sound stream is the only input stream to the process and the heard sound stream is the only output stream from the process. The heard sound stream may be processed further in other processors or it may cause an actuator (e.g. speaker) to generate the sound or it may be stored in a file. The original sound stream may be generated by an electronic instrument or microphone or may be generated from a stored sound (e.g. .wav) file.

If the sound was generated in a rectangular room, rather than a cube, then we would have multiple echoes from different walls, and we would represent that by multiple echo agents. Consider a room which was much longer than it was broad. The echo from the far-away walls would re-echo at a time period determined by the length of the long-side of the room while the echo from the near walls would re-echo at a period determined by its breadth. The heard sound at the center of the room would consists of the sum of the original sound and both echoes. The diagram is shown below.

Slide2.jpg

In our first example, the sound stream is generated from a file; parameters such as the rate of data generation can be set so as to simulate a microphone or other source. The source generator executes within its own thread. Also, in this initial example the heard sound is stored in a file.

The echo stream is internal to the process: it is produced and consumed entirely by agents within the process.

The process has an agent which creates the heard sound by summing the original sound and the echo. The process has another agent which generates the echo from the heard stream. In addition the process has agents that store streams in files; we will plot data from the files. The agents that store streams in files are not shown in the diagram.

The computational function of the process is shown in the code below. The code for the source thread which generates the original sound stream is shown later.

    # Define the computational function.
    def compute_func(in_streams, out_streams):
        # Name external streams for convenience
        original_sound = in_streams[0]
        heard_sound = out_streams[0]

        # Define internal streams
        echo = Stream(name='echo', initial_value=[0]*delay)
        
        # Create agents
        # Agent that creates heard sound from original sound and echo
        zip_map(func=sum,
                in_streams=[original_sound, echo],
                out_stream=heard_sound)
        # Agent that creates the echo from the heard sound.
        window_dot_product(
            in_stream=heard_sound, out_stream=echo,
            multiplicand_vector=attenuation_vector)
        # Agents that store sounds in files
        stream_to_file(in_stream=heard_sound, filename='heard.txt')
        stream_to_file(in_stream=echo, filename='echo.txt')
        stream_to_file(in_stream=original_sound, filename='original_sound.txt')

The computational function has two parameters: a list of input streams and a list of output streams. This example has only a single input stream and a single output stream. We identify these streams as original_sound and heard_sound —- see the first two lines of the function. Giving variable names for these streams is merely for convenience; we could have used in_streams[0] and out_streams[0] instead. In the next line we create the only internal stream: echo. The initial value of the echo stream is a list of delay zeros. This is because an echo is heard at the center of the room only after delay samples.

The next line creates an agent which generates the heard sound by summing the original sound and the echo. The following line creates an agent which generates the echo from the heard sound and the attenuation vector. The following lines store streams in files.

The response to an impulse in the center of a cube is shown in the diagram below where the top plot shows the original sound — the impulse — the middle plot shows the echo and the the lowest plot shows the heard sound. (The echoes in the response may look like they are getting longer in the plot, but that’s an artifact of the plot; in fact each echo is of the same duration.)

ReverberationSignals1.png

In the compute function we use zip_map which is an example of a merge agent with more than one input and a single output, and which is found in IoTPY/IoTPy/agent_types/merge.py. We also use window_dot_product which is an example of an op (operator) with a single input and single output and which is found in IoTPy/examples/signal_processing_examples. The stream_to_file agent is an example of a sink agent: single input and no output and which is found in IoTPY/IoTPy/agent_types/sink.py.

The Source Thread

If the source data is in a list then we use source_list_to_stream to create a thread that copies data from the source into the stream.

    def generate_original_sound(original_sound):
        return source_list_to_stream(
            in_list=original_sound_list,
            out_stream=original_sound,
            time_interval=0)

The time interval is the rate at which data from the list is copied to the stream; you can simulate different types of source devices, such as microphones, by setting the time interval appropriately.

Processes

In this example we run the application as a single process. Since this process does not communicate with remote processes we make this process a shared_memory_process rather than a distributed_process.

    # Create processes
    proc = shared_memory_process(
        compute_func=compute_func,
        in_stream_names=['original_sound'],
        out_stream_names=['heard_sound'],
        connect_sources=[('original_sound', generate_original_sound)],
        connect_actuators=[],
        name='proc')

This process has a single input stream named ‘original_sound’ and a single output stream named ‘heard_sound’ and its source thread generate_original_sound puts data into the input stream called ‘original_sound’. The process in this example has no actuators and so connect_actuators is the empty list. (Note: You don’t have to include the connect_actuators = [] line because that’s the default.)

Creating and Running the Multiprocess Application

Later, if you wished, you could process the heard sound in another processor; however, for now let’s restrict the application to a single process. Since the application has only one process, this process has no connections to other processes. We now create a multiprocess application consisting of a single process, proc, and no connections.

    mp = Multiprocess(
        processes=[proc],
        connections=[])
    mp.run()

More Functions in Signal Processing Examples

Generate Waves: Generates different types of waves such as square waves and sine waves.

Window Dot Product: Agent that outputs a stream consisting of the inner product of a moving window of an input stream and a specified vector. This is useful in many applications including reverberation.