Alex's blog: Visualization of live data streams with the gnuplot and bash (Part 2)

In my first article I have demonstrated the gnuplotwindow.sh script and other utilities to feed the gnuplot with live data streams. Shown scripts are easy to use and if combined together they build powerful chains of data filters. The only problem with the script is that it can only display a continuous data stream updating the plot with each new incoming data sample. To display even more complex data like scatter or 3D plots a different approach of data feeding is needed. It would be necessary to update the plot blockwise and input data should also be fed in blocks. This new concept didn't match well with the gnuplotwindow.sh script so I have written a new script because mixing both concepts in one script would make it unnecessary complex and unmaintainable.

gnuplotblock.sh script

The gnuplotblock.sh is based on the gnuplotwindow.sh :

#!/bin/bash

terminal="qt"      # terminal type (x11,wxt,qt)
range=${1:-:;:;:}  # min:max values of displayed ranges.
                   # ":" for +/- infinity. Default ":;:;:"
shift              # the rest are the titles

# titles definitions examples:
# - "Spectrum;1;blue"
# - "Scatter plot;points pointtype 5 pointsize 10;red;xy"
# - "3D plot;2;#903489;3d;32;32"

declare -A styles_def
styles_def=( [0]="filledcurves x1" [1]="boxes" [2]="lines" [3]="points" )
# remove the color adjustment line below to get
# default gnuplot colors for the first six plots
#colors_def=("red" "blue" "green" "yellow" "cyan" "magenta")
#colors=( "${colors_def[@]}" )
colors=(  )

# parsing input plots descriptions
i=0
IFS=$';'
while [ -n "$1" ]; do
  tmparr=( $1 )
  titles[$i]=${tmparr[0]}
  [ -n "${tmparr[1]}" ] || tmparr[1]=0
  styles[$i]=${styles_def[${tmparr[1]}]-${tmparr[1]}}
  [ -n "${styles[$i]}" ] || styles=${styles_def[0]}
  colors[$i]=${tmparr[2]}
  dtype[$i]=${tmparr[3]}
  dtype_arg[$i]=${tmparr[4]}
  dtype_arg2[$i]=${tmparr[5]}
  [ "${dtype[$i]}" = "xy" ] && ((i++))
  ((i++))
  shift
done

tmparr=( $range )
xrange=${tmparr[0]}
yrange=${tmparr[1]}
zrange=${tmparr[2]}
IFS=$'\n'
blocks=0          # blocks counter
(
 echo "set term $terminal noraise"
 echo "set style fill transparent solid 0.5"

 echo "set xrange [$xrange]"
 echo "set yrange [$yrange]"
 echo "set zrange [$zrange]"
 echo "set ticslevel 0"
 echo "set hidden3d"
# uncomment to remove axis, border and ticks
# echo "set tics scale 0;set border 0;set format x '';set format y '';set format z ''"
 [[ "${dtype[0]}" != "xyz" ]] && [[ "${dtype[0]}" != "map" ]] && {
   echo "set dgrid3d ${dtype_arg[0]},${dtype_arg[0]} gauss 0.25"
 }

 [[ "${dtype[0]}" = "map" ]] && {
   echo "set view map"
 }

 while read newLine; do
  if [[ -n "$newLine" ]]; then
    a+=("$newLine") # add to the end
  else
    nf=$(echo "${a[0]}"|awk '{print NF}')
    if [[ "${dtype[0]}" = "map" ]] || [[ "${dtype[0]}" = "xyz" ]] || [[ "${dtype[0]}" =~ ^3d.* ]]; then
      # only one splot command is used for all blocks for 3db type plots
      if [[ "${dtype[0]}" != "3db" ]] || [[ $((blocks%dtype_arg2[0])) -eq 0 ]]; then
        if [[ "${dtype[0]}" = "map" ]]; then
          echo -n "plot "
        else
          echo -n "splot "
        fi
        echo -n "'-' u 1:2:3 t '${titles[0]}'"
        echo -n " w ${styles[0]-${styles_def[0]}}"
        [[ -n "${colors[0]}" ]] && echo -n " lc rgb '${colors[0]}'"
        echo -n ","
      fi
    else
      echo -n "plot "
      for ((j=0;j<nf;++j)); do
        c1=1; c2=$((j+2));
        [[ "${dtype[j]}" = "xy" ]] && {
          c1=$((j+2)); c2=$((j+3));
        }
        echo -n " '-' u $c1:$c2 t '${titles[$j]}' "
        echo -n "w ${styles[$j]-${styles_def[0]}} "
        [[ -n "${colors[$j]}" ]] && echo -n "lc rgb '${colors[$j]}'"
        echo -n ","
        [[ $c1 = 1 ]] || ((j++))
      done
    fi
    echo
    if [[ "${dtype[0]}" = "3d" ]] || [[ "${dtype[0]}" = "map" ]]; then
      [[ ${#a[@]} -gt $((dtype_arg[0]*dtype_arg2[0])) ]] && {
        a=( "${a[@]:dtype_arg[0]}" )
      }
      for ((j=1;j<=dtype_arg2[0];++j)); do
        [[ -n "${a[(dtype_arg2[0]-j)*${dtype_arg[0]}]}" ]] || continue;
        for ((i=0;i<dtype_arg[0];++i)); do
          echo  "$i $j ${a[(dtype_arg2[0]-j)*${dtype_arg[0]}+i]}"
        done
      done
      echo e # gnuplot's end of dataset marker
    elif [[ "${dtype[0]}" = "3db" ]]; then
      for ((i=0;i<dtype_arg[0];++i)); do
        echo  "$i $((blocks%dtype_arg2[0])) ${a[i]}"
      done
      unset a
      [[ $((blocks%dtype_arg2[0])) -eq $((dtype_arg2[0]-1)) ]] && {
        echo e # gnuplot's end of dataset marker
      }
    elif [[ "${dtype[0]}" = "xyz" ]]; then
      for i in "${a[@]}"; do echo  "$i"; done
      unset a
      echo e # gnuplot's end of dataset marker
    else
      for ((j=0;j<nf;++j)); do
        tc=0 # temp counter
        for i in "${a[@]}"; do
          echo "$tc $i"
          ((tc++))
        done
        echo e # gnuplot's end of dataset marker
      done
      unset a
    fi
    ((blocks++))
  fi
done) | gnuplot 2>/dev/null

Script has the following input parameters:

min and max for x,y and z axis values formatted as "xmin:xmax;ymin:ymax,zmin:zmax". min and max or both could be omitted to let gnuplot automatically scale the plot. Default value is ":;:;:" – autoscaling for all axis.

the rest are the descriptions for each plot in the following form: "Title;Style;Color;Plot_type". Here:

Title – is the legend displayed on the graph
Style – can be an index into an array of predefined line styles. Currently the following styles are supported:

Index Line Style Default

0 Filled Curves yes

1 Boxes no

2 Lines no

3 Points no

alternatively instead of the index one can specify gnuplot supported style notion like - 'lines lw 3'
Color – color specified either as a color name supported by the gnuplot (e.g. red) or as a hex code (e.g. #ff007d). If color isn't explicitly given for a plot then the default gnuplot color will be used.

Index	Line Style	Default
0	Filled Curves	yes
1	Boxes	no
2	Lines	no
3	Points	no

Plot_type – the following plot types are supported currently:

Type	Default	Description
-	yes	If no plot type is specified then input data is interpreted as a simple 'y' values stream and x axis will be presented by a range from 0 to the input block size
xy	no	Input data is presented by "X Y" pair of values separated by a white space (i.e. spaces or tabs).
3d	no	Simple 'y' values stream blocks are presented in 3D showing how the signal is changing with time. 3D plot is updated with each new incoming signal block.
map	no	Similar to the '3d' type but shows a heat map instead of 3D
3db	no	Same as '3d' type but the plot is updated only when all blocks are received. This type can be thought as a matrix 'X_Z' and input values are the Y axis.
xyz	no	Input data is presented by "X Y Z" values. The whole data set is presented by one block.

Note: As far as semicolon is used as a separator it is not allowed to have it in the Title or in the Style description.

'3d', '3db' and 'map' plot types needs additional input parameters that should be specified after the Plot_type separated by ';'. These plot types needs to know how many blocks should be shown (i.e. how deep should be the signal history) and how many samples are in a block.

Example: 'My signal title;line lw 2;red;3d;32;64' – 32 samples per block, 64 blocks

Blocks in an input stream are separated by an empty line.

gnuplotblock.sh usage examples and other useful scripts

As in my first article it's better to explain how the script works using real examples with full commands chains and the resulting pictures.

Filtered white noise

In this first example a continues data stream from the /dev/urandom is filtered by a low pass filter and then it is presented as blocks with 100 samples in each. This is an example of the 'no type' plots:

$ dd if=/dev/urandom 2>/dev/null | hexdump -v -e '/1 "%u\n"'| bin/dsp/fir.sh| \
  (a=();while sleep 0.02;do read l;a=("$l" "${a[@]}"); \
   a=( "${a[@]:0:100}" );printf "%s\n" "${a[@]}";echo;done) | \
  bin/gp/gnuplotblock.sh "0:100;0:255" "Noise;2;blue" "Filtered Noise;l lw 2;red"

Figure 1. Filtered white noise

The same can be done using the older gnuplotwindow.sh script:

$ dd if=/dev/urandom 2>/dev/null | hexdump -v -e '/1 "%u\n"' | bin/dsp/fir.sh | \
  (while sleep 0.02;do read l;echo $l;done)| \
  bin/gp/gnuplotwindow.sh 100 "0:255" "Noise;2" "Filtered Noise;2"

By the way if you also want to listen to the filtered white noise you may execute the following command:

# Change '$2' to '$1' to hear not filtered white noise.

$ dd if=/dev/urandom 2>/dev/null | hexdump -v -e '/1 "%u\n"' | \
  bin/dsp/fir.sh | awk -b '{printf("%c",$2);fflush()}' | aplay

Gnuplot biker

There is a saying - "One can endlessly stare at moving water, leaping fire and the gnuplot biker riding the sine wave". This is an example of the 'xy' plots:

$ (while true; do echo line; sleep 0.03; done) | \
  awk 'BEGIN{PI=atan2(0,-1);t=0;f0=0.1;}
       {print sin(2*PI*t*f0)+2;t+=0.1;fflush()}'| \
  bin/gp/biker.sh 64 "10;4" | \
  bin/gp/gnuplotblock.sh "0:63;0:8" "Sine wave;0;#008000" \
   "gnuplot biker;l lw 3;red;xy"

Figure 2. Gnuplot biker

bin/gp/biker.sh script expects a continuous data stream as its input and adjusts biker position accordingly. Then it adds biker coordinates as the second column to the input stream and feeds the gnuplotblock.sh with the updated data.

And here the Gnuplot biker struggles through random terrains:

$ dd if=/dev/urandom | hexdump -v -e '/1 "%u\n"' | \
  (while read line;do echo "$line"; sleep 0.03; done) | \
  bin/dsp/fir.sh | bin/gp/removecolumns.sh 1 | \
  awk '{print $1/8;fflush()}'| \
  bin/gp/biker.sh 80 '20;40'| \
  bin/gp/gnuplotblock.sh "0:79;0:80" "Filtered Noise;0;#008000" "biker;l lw 3;red;xy"

Figure 3. Gnuplot biker riding random terrains

Maze

Prepare to be amazed! It is also possible to use the script interactively! As an example the bin/gp/maze.sh script generates a random maze with defined size and one can move the red dot through it using WASD keys:

$ bin/gp/maze.sh 12 12 | \
  bin/gp/gnuplotblock.sh "-1.5:23.5;-1.5:23.5" \
         ";points pointtype 5 pointsize 3;blue;xy" \
         ";points pointtype 5 pointsize 3;red;xy"

Figure 4. Walking through a maze

Real time clocks

Another example of the 'xy' plots is digital clocks:

$ bin/gp/clock.sh|bin/gp/gnuplotblock.sh ":;5:11" ";points pointtype 5 pointsize 5;blue;xy"
$ bin/gp/clock2.sh|bin/gp/gnuplotblock.sh ":;0:12" ";points pointtype 7 pointsize 2;blue;xy"

Figure 5. A gnuplot clock

Figure 6. Another gnuplot clock

The bin/gp/clock.sh uses the same font I used in my bin/pingpong.py implementation and the bin/gp/clock2.sh uses a 7x10 font that I found somewhere on the net.

But should it be possible to create an analog clock using xy plot type and the same principle as in the Gnuplot biker example? I leave it to the reader as an exercise.

GOL

I just couldn't let it happen that somebody else would be the first one to implement the Convay's Game of Life using Gnuplot! So the script is here and it is also interactive! One could specify 'keyboard' as 4'th parameter to step through iterations by pressing any key. The 3'd parameter is a fill factor in percent. If it is set to 's1' then the 'ship' configuration is generated:

$ bin/gp/gol.sh 50 50 62|bin/gp/gnuplotblock.sh "0:49;0:49" ";points pointtype 7 pointsize 1;blue;xy"

Figure 7. Convay's Game of Life

Chirp signal and its spectrum

This example is a complex one. It demonstrates how to multiplex one input data stream to show different signal properties at the same time. Here I generate a chirp signal which is multiplexed by the tee command and is shown by the old gnuplotwindow.sh script and in parallel its frequency spectrum is calculated and shown in real time:

$ (while true; do echo line; sleep 0.01; done) | \
   awk 'BEGIN{PI=atan2(0,-1);t=0;f0=1;f1=32;T=256;k=(f1-f0)/T}
        {print sin(2*PI*(t*f0+k/2*t*t));t+=0.1;fflush()}'| \
   tee >(bin/dsp/fft.sh 64 "w1;0.5;0.5"|bin/dsp/fftabs.sh 64 | \
         bin/gp/gnuplotblock.sh '-0.5:31.5;0:0.3' 'Chirp signal spectrum;1')| \
   bin/gp/gnuplotwindow.sh 128 "-1:1" "Chirp signal;2"

Figure 8. Chirp signal and its spectrum

Note: click on the image to see an animated version (Warning: 7.4MB)

The FFT algorithm for frequency spectrum calculation is implemented in AWK - bin/dsp/fft.sh. Another simple script bin/dsp/fftabs.sh does FFT normalization.

Chirp signal spectrum and its history

Now I want to see how the signal spectrum is changing with time. Here the chirp signal spectrum history is shown using the '3d' plot type:

$ (while true; do echo line; sleep 0.01; done) | \
  awk 'BEGIN{PI=atan2(0,-1);t=0;f0=1;f1=32;T=128;k=(f1-f0)/T;}
       {print sin(2*PI*(t*f0+k/2*t*t));t+=0.1;fflush()}'| \
  bin/dsp/fft.sh 64 "w0" | bin/dsp/fftabs.sh 64 | \
  bin/gp/gnuplotblock.sh '0:31;0:31' \
  'Chirp signal spectrum with history;l lw 1 palette;;3d;32;32'

Figure 9. Chirp signal spectrum and its history

So each newly calculated spectrum block is added and then the plot is updated. You may have noticed that the frequency spectrum doesn't look the same like in the previous example. It's because in this example the rectangular window ("w0" parameter) is used to calculate signal frequency spectrum.

Sinc function

But what if I want to show a 3D signal? In this case I need to wait until all blocks of the signal are fed and then update the plot. This is what the '3db' plot type is for. As an example the 3D Sinc function is shown:

$ (while true; do echo line; sleep 0.25; done)| \
  awk 'BEGIN{c=0.01;xmin=-16;xmax=16;ymin=-16;ymax=16}
      {for(j=ymin;j<=ymax;++j) {
         for(i=xmin;i<=xmax;++i) {
           r=sqrt(i*i+j*j);
           if (r!=0) {t=sin(sin(c)*r)/(sin(c)*r)}
           else {t=1.0}
           print t
         }
         print ""
       }
       fflush();c+=0.1;}'| \
  bin/gp/gnuplotblock.sh '0:32;0:32;-0.25:1' \
    'sinc(sqrt(x*x+y*y));l lw 1 palette;;3db;33;33'

Figure 10. Sinc function in 3D

Hilbert space filling curve 3D

3d andn 3db plot types input can be thought as a matrix values where the matrix indices are calculated automatically by the script. But what if I want to show a 3D scatter plot where points are presented by all three coordinates? In this case data should be presented by three coordinates per line and an empty line separates data blocks. Such data can be shown using the 'xyz' plot type. To demonstrate this type of plots the Hilbert space filling curve script (bin/gp/zpk.sh) is used. The script is interactive one. One can change the iteration using 'WS' keys. Pressing any other key shows current iteration in the terminal window:

$ bin/gp/zpk.sh | bin/gp/gnuplotblock.sh '' 'ZPK;l lw 1 palette;;xyz'

Figure 11. Hilbert space filling curve 3D

Note: click on the image to see an animated version (Warning: 1.2MB)
Note_2: above the 5'th iteration the gnuplot becomes almost not interactive at least on my computer.

RTL-SDR

In this last example I'll show how to use the gnuplotblock.sh with the rtl_power tool. Here an FM radio station frequency spectrum is shown. The data is fed by the rtl_power program once per second:

$ rtl_power -i 1 -f 102.225M:102.375M:256 -g 20.7 -p 30 -w hamming |
  awk -F, '{for(i=7;i<NF;++i){j=i-7;print $3+$5*j" "$i;};print "";fflush()}' |
  bin/gp/gnuplotblock.sh ":;-60:0" "rtl\_power 102.225M:102.375M:256;2;blue;xy"

Figure 12. An FM radio station spectrum

And here the same data is shown with the 'map' plot type that produces the heat map:

$ rtl_power -i 1 -f 102.225M:102.375M:1k -g 20.7 -p 30 -w hamming |
  awk -F, '{for(i=7;i<NF;++i){j=i-7;print $3+$5*j" "$i;};print "";fflush()}' |
  bin/gp/removecolumns.sh 1 |
  bin/gp/gnuplotblock.sh "-1:256:-1:30;:" "rtl\_power 102.225M:102.375M:1k;image;;map;256;30"

Figure 13. Heat map of last 30 measurements

Known issues

If data is fed too fast then the Gnuplot can produce some artifacts. For example instead of a 3D plot it could show this:

Figure 14. Data is fed too fast

This is it. All scripts described in the article are also available from my Github repository

Alex's blog

Thursday, September 8, 2016

Visualization of live data streams with the gnuplot and bash (Part 2)