Managing a machine vision project in PYNQ
It's time for more PYNQ tips and tricks. This time I'll showcase a bit on software distribution, testing and some funny OpenCV face detection
If you created a notebook and you want to share it and make it easy to install then there are few steps you have to follow.
First of all create a repository, github is most recommended here. The goal is to put the notebook in the repository and make it install via simple command like:
Installation is handled by a setup.py file that contains the install instructions. I made a very simple repository that installs a simple notebook - pynq-example-notebook. You can clone/copy it and modify for your needs.
If you check the setup.py it looks like so:
We want that our notebook shows up on the notebooks list and to do that we have to copy it explicitly. This is handled by install_notebook function that is called during installation. Thanks to PYNQ_JUPYTER_NOTEBOOKS environment variable we know where the notebooks are stored so we can create a new folder there, copy files and done.
The setup function can also be used to define Python requirements. If you need some system requirements then you can call apt via
More advanced projects may need to compile stuff or differentiate code based on target board. This also can be done. You can check setup.py files of other projects, like Xilinx/PYNQ-HelloWorld to get an example how to do these things.
When writing code it's good to have some tests ready to quickly validate given piece of logic. For Jupyter notebooks we can use ipytest package that brings pytest tests to notebooks.
To get it running we have to install ipytest. For the default PYNQ-Z2 board I've noticed some dependencies were to old and not compatible, so you have to update them. At the time of writing this article these versions were working for me:
Then import it in desired notebook:
Then you can create a block with test functions and make it execute like so:
And lets look at an example function I've wrote to get bigger face box than what detection returned:
This function takes face detection (x, y, width, height) and the source image object (Pillow). Returns expanded face coordinates by given amount of pixels. Testing it live on some captured frame would take to much time to check all cases and even more problematic when it would be a part of much complex flow. So lets write some tests:
Tests can also be used to showcase how a part of code works based on provided test data instead of like a frame captured from HDMI - which may or may not be available when someone is browsing the notebook, or has no matches on the frame from the camera. Here is an example:
This a function that crops the face from the image, draws deep pink overlay and adds a text written with Comic Sans:
In normal code flow it would need an image, like a frame from a camera, face detection run on it and so on. But a notebook can be provided with additional files that can be used to test/showcase such smaller pieces:
Here we manually provide the image and coordinates of the face. We also use IPython.display.display to force-display the images:
And having test image we can do a quick test against external libraries we use, like face detection:
You should know where it is going... we are going to make a meme generator as a machine vision project. Our PYNQ-Z2 board will read frames from HDMI-IN, detect faces and if present draw memes, put them on a single frame and display that on HDMI-OUT. The full notebook is available on riklaunim/pynq-meme-generator.
We start with the general setup:
And then the logic:
We do a loop over 10 frames - for each frame we try to detect faces and if so we make memes out of them (crop, draw the outline and text), then place all memes on a 1080p image that will be then sent as the output. If no faces are detected we just display the original frame so it's easier to see what's going on and why no faces were found.
There are Python libraries like face-recognition that can compare/recognize faces. It can run on PYNQ-Z2 although installation will take more than hour while the process itself won't be to fast either as it's not accelerated. If you would want to do it quickly then either detected and cropped faces would have to be sent to the main server capable of doing this computation, or maybe trying to use a USB-based NPU to get some edge acceleration of some similar implementation.
face-recognition can be used like so:
The result is a one element list with True/False value depending if the faces are assumed of same person. If face can't be detected then no encoding will be returned.
You can get the full notebook from riklaunim/pynq-meme-generator github repository.