This project was conceptualized to be run on a Raspberry Pi while the heaviest neural networks, palm and landmark offloaded to a Coral usb accelerator. Unfortunately, MediaPipe’s team could not yet provide a quantized version of the Landmarks NN because Coral doesn’t yet support deconvolution, as stated in this issue: https://github.com/google/mediapipe/issues/426#issuecomment-581536031
So, at this moment I don’t expect great performance running HandCommander on the RPi cpu but still worth trying.
In order to get the full potential out of the new RPi 4 (I’m working on a 4gb version) we need to switch to a 64bit OS. Coincidentally, even though MediaPipe doesn’t support RPi but the Coral dev board, their instructions for cross compiling and the provided scripts are meant for arm64 so this also justifies changing the Kernel/OS.
The first step is to backup Node-RED flows, those are:
– The “Mqtt Sony TV”
[{"id":"15918c8d.3a42a3","type":"tab","label":"Mqtt Sony TV","disabled":false,"info":""},{"id":"42c28eac.d602c","type":"exec","z":"15918c8d.3a42a3","command":"irsend -#3 SEND_ONCE sony_46 ","addpay":true,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"","x":440,"y":120,"wires":[[],[],["9b30fe6.df454"]]},{"id":"79678e76.ba6ed","type":"inject","z":"15918c8d.3a42a3","name":"","topic":"","payload":" KEY_VOLUMEUP","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":150,"y":280,"wires":[["3f687891.a79368"]]},{"id":"9b30fe6.df454","type":"debug","z":"15918c8d.3a42a3","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":670,"y":40,"wires":[]},{"id":"7fa1d0ed.91cb8","type":"mqtt in","z":"15918c8d.3a42a3","name":"","topic":"handCommander/tv/ir_command","qos":"2","datatype":"auto","broker":"5e1c71ef.270ab","x":150,"y":40,"wires":[["61ef8f2e.cf623"]]},{"id":"3f687891.a79368","type":"mqtt out","z":"15918c8d.3a42a3","name":"","topic":"handCommander/tv/ir_command","qos":"","retain":"","broker":"5e1c71ef.270ab","x":500,"y":320,"wires":[]},{"id":"e2483257.3f315","type":"inject","z":"15918c8d.3a42a3","name":"","topic":"","payload":" KEY_VOLUMEDOWN","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":160,"y":320,"wires":[["3f687891.a79368"]]},{"id":"7d610a3e.1596a4","type":"inject","z":"15918c8d.3a42a3","name":"","topic":"","payload":" KEY_MUTE","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":130,"y":360,"wires":[["3f687891.a79368"]]},{"id":"ec12f179.6fa82","type":"inject","z":"15918c8d.3a42a3","name":"","topic":"","payload":" KEY_POWER","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":130,"y":400,"wires":[["3f687891.a79368"]]},{"id":"61ef8f2e.cf623","type":"delay","z":"15918c8d.3a42a3","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"0.25","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":235,"y":120,"wires":[["42c28eac.d602c"]],"l":false},{"id":"e922d019.f5b41","type":"comment","z":"15918c8d.3a42a3","name":"Testing","info":"","x":365,"y":254,"wires":[]},{"id":"8bc8054f.142af8","type":"comment","z":"15918c8d.3a42a3","name":"Mqtt to Signal","info":"","x":410,"y":40,"wires":[]},{"id":"5e1c71ef.270ab","type":"mqtt-broker","z":"","name":"Mosquitto on localhost","broker":"localhost","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}] – “Mqtt VLC”
[{"id":"529907f6.3d4e68","type":"tab","label":"Mqtt VLC","disabled":false,"info":""},{"id":"4fd8896e.b3ac58","type":"inject","z":"529907f6.3d4e68","name":"","topic":"","payload":"play","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":150,"y":260,"wires":[[]]},{"id":"16addc6c.a3aea4","type":"exec","z":"529907f6.3d4e68","command":"/home/pi/telnetCommands.sh 192.168.1.58 8212 733 ","addpay":true,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"","x":420,"y":140,"wires":[[],[],["dcf2d74e.11c6a8"]]},{"id":"dcf2d74e.11c6a8","type":"debug","z":"529907f6.3d4e68","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":670,"y":60,"wires":[]},{"id":"63b4a660.89bfb8","type":"inject","z":"529907f6.3d4e68","name":"","topic":"","payload":"play","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":150,"y":260,"wires":[["75bc670.c3a1998"]]},{"id":"8f576b9c.97f0d8","type":"inject","z":"529907f6.3d4e68","name":"","topic":"","payload":"stop","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":150,"y":300,"wires":[["75bc670.c3a1998"]]},{"id":"5eb6cff2.a53e6","type":"inject","z":"529907f6.3d4e68","name":"","topic":"","payload":"prev","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":150,"y":340,"wires":[["75bc670.c3a1998"]]},{"id":"bd003741.827538","type":"inject","z":"529907f6.3d4e68","name":"","topic":"","payload":"next","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":150,"y":380,"wires":[["75bc670.c3a1998"]]},{"id":"8810e7e3.631a98","type":"mqtt in","z":"529907f6.3d4e68","name":"","topic":"handCommander/VLC","qos":"2","datatype":"auto","broker":"5e1c71ef.270ab","x":120,"y":60,"wires":[["16addc6c.a3aea4"]]},{"id":"75bc670.c3a1998","type":"mqtt out","z":"529907f6.3d4e68","name":"","topic":"handCommander/VLC","qos":"","retain":"","broker":"5e1c71ef.270ab","x":380,"y":320,"wires":[]},{"id":"ad32ef13.900b4","type":"comment","z":"529907f6.3d4e68","name":"Mqtt to Signal","info":"","x":410,"y":60,"wires":[]},{"id":"d95665cb.884bb8","type":"comment","z":"529907f6.3d4e68","name":"Testing","info":"","x":350,"y":260,"wires":[]},{"id":"5e1c71ef.270ab","type":"mqtt-broker","z":"","name":"Mosquitto on localhost","broker":"localhost","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}] Remember to import those settings in the new NodeRed installation
Next is to backup your LIRC configuration as well as any other software you may have installed.
The safest way is to save an image of the entire SD card in your computer, the Read function of win32DiskImager will do just fine.
Raspbian doesn’t yet have an official 64 bit os, there are some good alternatives like Gentoo but in this case I went for Ubuntu https://ubuntu.com/download/raspberry-pi
You can download and flash an image from that site or use the new Raspberry Pi Imager https://www.raspberrypi.org/blog/raspberry-pi-imager-imaging-utility/
After the initial boot, just instal the prerequisites as described in the first post in this series https://www.deuxexsilicon.com/2020/03/16/1st-motivation-and-first-steps/
Once everything is setup, its time to cross compile HandCommander, beware that it takes several steps. This instructions are based on MediaPipe’s steps to cross compile for Coral Dev Board https://github.com/google/mediapipe/blob/master/mediapipe/examples/coral/README.md, since they have a similar architecture, only minor modifications are required:
Download my latest fork of MediaPipe as well as HandCommander
on Host
$ cd ~
$ git clone -b lisbravo_01 https://github.com/lisbravo/mediapipe.git
$ cd mediapipe
$ git clone https://github.com/lisbravo/myMediapipe.git
$ mkdir bazel-bin
$ sh mediapipe/examples/coral/setup.sh On Pi
$ cd ~
$ git clone -b lisbravo_01 https://github.com/lisbravo/mediapipe.git
$ cd mediapipe
$ git clone https://github.com/lisbravo/myMediapipe.git
$ mkdir bazel-bin Back on host
$ docker build -t coral .
$ docker run -it --name coral coral:latest Inside docker
Update library paths in /mediapipe/third_party/opencv_linux.BUILD
(replace ‘x86_64-linux-gnu’ with ‘aarch64-linux-gnu’)
“lib/aarch64-linux-gnu/libopencv_core.so”,
“lib/aarch64-linux-gnu/libopencv_calib3d.so”,
“lib/aarch64-linux-gnu/libopencv_features2d.so”,
“lib/aarch64-linux-gnu/libopencv_highgui.so”,
“lib/aarch64-linux-gnu/libopencv_imgcodecs.so”,
“lib/aarch64-linux-gnu/libopencv_imgproc.so”,
“lib/aarch64-linux-gnu/libopencv_video.so”,
“lib/aarch64-linux-gnu/libopencv_videoio.so”,
$ sed -i 's/x86_64-linux-gnu/aarch64-linux-gnu/g' /mediapipe/third_party/opencv_linux.BUILD – Comment the EdgeTpu section of WORKSPACE
Line 357:
# EdgeTPU
#new_local_repository(
# name = “edgetpu”,
# path = “/edgetpu/libedgetpu”,
# build_file = “/edgetpu/libedgetpu/BUILD”
#)
#new_local_repository(
# name = “libedgetpu”,
# path = “/usr/lib/aarch64-linux-gnu”,
# build_file = “/edgetpu/libedgetpu/BUILD”
#)
Or just use sed:
$ sed -i '357,367{s/^/#/}' WORKSPACE
$ sed -i 's/x86_64-linux-gnu/aarch64-linux-gnu/g' WORKSPACE
$ sed -i 's/\/usr\/include/\/usr\/aarch64-linux-gnu\/include/g' WORKSPACE – Attempt to build hello world (to download external deps)
$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hello_world:hello_world – Edit ./bazel-mediapipe/external/com_github_glog_glog/src/signalhandler.cc
$ sed -i 's/(void\*)context->PC_FROM_UCONTEXT/NULL/g' ./bazel-mediapipe/external/com_github_glog_glog/src/signalhandler.cc – Try to cross compile the hello world
$ bazel clean
$ bazel build -c opt --crosstool_top=@crosstool//:toolchains --compiler=gcc --cpu=aarch64 --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hello_world:hello_world – Set the correct mqtt broker ip (your RPi ip) in the “broker_ip” setting at line 216
$ vi myMediapipe/graphs/dynamicGestures/dynamic_gestures_cpu.pbtxt Cross compile HandCommander:
– Option 1: Compile it for using a rtp streaming video source
$ bazel build -c opt --crosstool_top=@crosstool//:toolchains --compiler=gcc --cpu=aarch64 --define MEDIAPIPE_DISABLE_GPU=1 myMediapipe/projects/dynamicGestures:dynamic_gestures_cpu_tflite – Option 2: Compile it to use a local Webcam
$ bazel build -c opt --crosstool_top=@crosstool//:toolchains --compiler=gcc --cpu=aarch64 --define MEDIAPIPE_DISABLE_GPU=1 myMediapipe/projects/dynamicGestures:dynamic_gestures_cpu_tflite_cam – Copy HandCommander to Raspberry
$ scp -r bazel-bin/myMediapipe/projects/dynamicGestures/ ubuntu@<replace with yours RPi IP>:/home/ubuntu/mediapipe/bazel-bin On Pi, run HandCommander
(with monitor or through VNC, RDP works but way too slow)
$ cd mediapipe – Option 1: Using a rtp streaming video source
$ bazel-bin/dynamicGestures/dynamic_gestures_cpu_tflite \
--calculator_graph_config_file=myMediapipe/graphs/dynamicGestures/mainGraph_desktop.pbtxt --input_side_packets=input_video_path=rtp://0.0.0.0:5000 – Option 2: Using a Webcam
$ bazel-bin/dynamicGestures/dynamic_gestures_cpu_tflite_cam \
--calculator_graph_config_file=myMediapipe/graphs/dynamicGestures/mainGraph_desktop_cam.pbtxt
If everything is in order you should see an OpenCV window with HandCommander recognizing your gestures as well as the debug messages in the terminal:
It’s interesting to check how much resources HandCommander uses, here a baseline, only XFCE is running:
With HandCommander running:
It consumes around 30-40% of the total RPi cpu, shared across all cores, and 50MB of RAM, not bad considering its making inference on 3 neural nets, 2 of them highly complex.
Also, even though I don’t currently have a fancy heatsink installed, just the cheap aluminum little squares, the max temp stays below 63 celsius:
Well, if you reached this far a sound “Congratulations!” is in order 😀
