TGU Logo
Testing Digital Solutions

Testing a ESP32 LoRa device with Python and Jenkins

This article describes setting up a test suite in Jenkins using pytest which will determine if an esp32 LoRa development board transmits data. The milestones of the test suite are ...

To achieve these milestones Jenkins has to ...

Index

The Tool Chain

The Hardware

Setting up Jenkins

Jenkins requires a Git plug-in for us to do what we want to do. A screen shot of the one we have installed is can be seen here. We also require a Git repository for our tests and MicroPython code which can be found here on git hub. To use this repository at the command line we do the following ... 

git clone https://github.com/tgu-ltd/jenkins_esp32_lora_testing.git
Now we want to setup a pipenv environment so we don't go overwriting system files and also make our repository portable to be used by Jenkins. 
pipenv --python 3.6 install
Now we can install some python packages ... 
pipenv run pip install pytest rshell esptool
If everything went ok you'll see something like ... 
Successfully installed esptool-2.6 pyserial-3.4 pytest-4.4.0 rshell-0.0.21 ...
To record a list packages that pipenv has installed we can run ... 
pipenv run pip freeze > requirements.txt
We need the requirements file for Jenkins to pick up and install packages into its own user space when it comes to execute the job. The reason for this will become evident later. If the Jenkins set up went OK we will be able to see something similar to this image. Clicking the create jobs link seen in the image will take us to a screen asking for details of the job. In this case it will be using a Freestyle project as show in this image Once we've clicked Ok to create the project you will see something very similar to this image

Configuring a Jenkins Job

This image details everything about the configuration. The sections to pay attention to are "Source Code Management" and "Build" -> "Execute shell". The source code management is set up to detect changes from the repository we created with git and github. The Execute shell commands does the following

pip install --user pipenv
Install pipenv
PATH=$PATH:~/.local/bin
Set the local path for pipenv application
pipenv --python 3.6 run pip install -r requirements.txt
Install the required packages
pipenv run pytest --junitxml=./report.xml -c tests/pytest.ini
Run the tests

Creating Tests with pytest

In the "projects/python/jenkins_esp32_lora_testing" directory the following directory and files have been created. Refer to the repo for these files.

./tests/pytest.ini
pytest configuration
./tests/test_firmware.py
The test suite
What we have done here is created a test suite to download and flash the latest firmware onto a esp32. We can now run / build this test suite in Jenkins.

Running Tests in Jenkins

Now that we have some tests lets get Jenkins to execute these tests to see what results we get back. To do this we can either set up Jenkins to poll the SCM, I.E check git for any changes, or we can click the "Build Now" option which is what we will be doing. The "Build Now" option can be found in the project's default page which can be seen in this this image. Once we've clicked the build now option a build will appear in the build history which is covered in more detail in the next Examining the Test Results section. If you have copied or cloned tests from the repository at this stage and they break you may want to follow the instructions in the Bigger and Better Test Reporting section and return back to this point.

Examining the Test Results

Now that the tests have been run we will have some build history which is shown in this image. To see what actually happened in the test we can click on the console output. This will show us what Jenkins executed and the output of that execution. Because the output is lengthy it has been been split up into three images. The first image shows the python packages being installed into a virtual environment The second image shows the tests being executed and this is where it all goes a bit Pete Tong. At the bottom of the third image the amount of tests that failed and passed can be seen. The important bits of the image to look at are ... 

raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 13] could not open port /dev/ttyUSB0: [Errno 13] Permission denied: '/dev/ttyUSB0'
...
...
...
		cmd = subprocess.run(flash, stdout=subprocess.PIPE)
		if cmd.returncode == 0:
			flashed = True
>       assert(flashed is True)
E       assert False is True
tests/test_firmware.py:64: AssertionError
- generated xml file: /var/lib/jenkins/home/workspace/ESP32_LoRa_Testing/report.xml -
====================== 1 failed, 2 passed in 1.40 seconds ======================
Build step 'Execute shell' marked build as failure
Recording test results
Finished: FAILURE

The first couple of lines show that the test failed on a rasied exception. In this case Jenkins did not have the correct permissions to access the USB device. We will fix this in a jiffy. The last few lines detail ...

Now that we've seen the tests failing lets have a look at what Jenkins is showing us back at the project page which can be seen here . The project overview page show us the number of times the tests have been run and a graph showing the trend of those tests. Lets go and fix that broken test.

Fixing the Test

To correct this test failure all we have to do is grant Jenkins access to the ttyUSB devices by issuing the following command as root. 

usermod -a -G dialout jenkins
We can now re-run the tests and admire our successes which you can be seen in this image . Before we continue to write more tests there are a few more features in Jenkins that make analysing tests a little easier. For example in this screen shot you can see that Jenkins has now reported the failing test as fixed. You can navigate to this information by first clicking the build number then the "Test Result" link.

Expanding the Firmware Tests

Now that the we think the firmware is being loaded onto the esp32 we want to write a test to make sure that it is. To do this we will using rshell to copy and execute files on the esp32. The following directories and files have been created in the github repository so we can do just that.

We also created our last firmware test in file ./tests/test_firmware.py named test_firmware_loaded and the result of which can be seen here. Note the extra test added to the results graph. The purpose of the archive directory is store information about the build which Jenkins can be configured to archive. To do this click the "Add post-build action" and then "Archive the artifacts". This screen shot shows what it should look like. After re-running the tests again we will now have an archive of the version info collected from the esp32 device. This image shows Jenkins archiving the firmware_text.txt which contains the following text 

(sysname='esp32', nodename='esp32', release='1.10.0', version='v1.10-315-ge70c438c7 on 2019-05-02', machine='ESP32 module with ESP32')
At this point it is also a good idea in the Jenkins build configuration under the "Source Code Management" section to add the option to "Clean before checkout". This will ensure that no artifacts will be left over from the previous build.

Pipenv Hiccups

When running the test suite again the following problem was encountered ... 

⠋ Creating virtual environment...
⠙ Creating virtual environment...
⠹ Creating virtual environment...Using base prefix '/usr/lib/python-exec/python3.6/../../..'
New python executable in /var/lib/jenkins/.local/share/virtualenvs/ESP32_LoRa_Testing-AEcau7YN/bin/python3
Running virtualenv with interpreter /usr/bin/python3

✘ Failed creating virtual environment 
[pipenv.exceptions.VirtualenvCreationException]:   File "/var/lib/jenkins/.local/lib64/python3.5/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
...
  File "/usr/lib64/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
OSError: [Errno 40] Too many levels of symbolic links: '/var/lib/jenkins/.local/share/virtualenvs/ESP32_LoRa_Testing-AEcau7YN/bin/python3'
To fix this problem the following commands were issued ... 
rm -rf /var/lib/jenkins/home/workspace/ESP32_LoRa_Testing
rm -rf /var/lib/jenkins/home/workspace/ESP32_LoRa_Testing@tmp
rm -rf /var/lib/jenkins/.local/share/virtualenvs/ESP32_LoRa_Testing-AEcau7Y
The reason these errors occurred was because a build was stopped half way through its execution. This sort of problem should not happen and requires further investigation.

First Milestone Achieved

We have reached our first milestone by loading the esp32 device with the latest firmware and tested that it was loaded. We can now move onto importing and writing some MicroPython code to get the LoRa enabled device to chirp. Before we go and do that lets setup some other tools for better reporting.

Bigger and Better Test Reporting

To fine tune the tests and reporting we are going to ...

Adding git submodules

For us to use the LoRa functionality on the esp32 device we need to add the uPyLora repository to our own repository as a submodule and configure Jenkins to pull it in. To do this at the command line in our repository directory execute ... 

git submodule add https://github.com/lemariva/uPyLora esp32/lora
and in Jenkins configure the build to add additional git sub-modules behaviour as shown in this. screen shot . Also note the "Clean before checkout option" just below the "Additional Behaviours" section. If we now go and execute the tests again the console output in the Jenkins build should show something similar to this image.

The Chirpy LoRa Code

We almost have everything needed to make the esp32 device chirp we just have to write some Python code and copy all the required files over to the esp32. The below two files combined with the uPyLora repository do just that ...

Before we venture off into the world of SDR's and rtl-fm we can manually make sure that the esp32 is chirping by using the gqrx tool to visualize what is happening on a specific frequency. This image shows the esp32 device chirping. It is worth noting here how much bandwidth the transmitted signal is taking up. This will need further investigation and explanation when we come to use the rtl-fm.

Testing the LoRa Code

We have manually tested that the esp32 LoRa node is chirping we can now write a new set of tests to check that the esp32 is outputting the expected file which we write out whilst chirping. We will also be checking that our test ordering is working.

To see how the ordering works the ./tests/test_firmware.py contains a test named test_firmware_directory_exists and just above this test is a decorator named @pytest.mark.run(order=1). This tells pytest to execute this test first. The subsequent tests in this file have decorators pointing to the test above them. The first test in ./tests/test_ulora.py has a decorator pointing to the last test in the ./tests/test_firmware.py file. This makes pytest upload and check the firmware before loading and executing our main MicroPython code.

We also now have html reports. This report snippet details a failed build and this report snippet details a successful one. There is allot of information to be gained from these reports such as build ID, build number and git branch which will hopefully come to use later in this article.

Second Milestone Achieved

We have reached our second milestone by loading the esp32 device with our own and external MicroPython code and tested it. We have also improved the reporting and added a bit more structure to the tests with ordering. The final milestone is to use a rtl-fm with a SDR to independently verify that the esp32 device is chirping. This does bring an extra complication in the form of threading but we will cross that bridge when we get to it. Before we dive into threads lets recap on what Jenkins is telling us and what can we gain from this.

Our current test trend looks like this. The last three failures in the graph where related to retrieving an outputted file from the esp32 device as this html report details. This is where Jenkins comes into its own by storing everything we need to know about failed and successful builds. We obviously want to know what makes a build fail so we can correct that problem whether it be with the tests or with another part of the stack. Conversely we also want to know when a build is successful as we may want to promote or release this build to other builds or projects.

Tagging Successful Builds

Because Jenkins does not point to our github repository but a local file system it took some tinkering and building to make it commit tags. We achieved this by adjusting the file permissions on the repository and added the Jenkins user to the users group in our OS. This is not the best way to use Jenkins but it suits our needs for now. So what has Jenkins done? And why go to all that trouble? We went to all this trouble because git now contains tags of our successful builds ... 

$ git tag
jenkins-ESP32_LoRa_Testing-54
jenkins-ESP32_LoRa_Testing-55
This is important because if we wanted to promote or release this build / product we now have a record of all the required python package versions and firmware. This image shows an overview of build 55 and this is the console output and this is firmware output and this is the esp32 device chirp output file. All this information can be gathered with a couple of mouse clicks or better yet be automatically gathered and bundled with a product.

Listening for the Chirps

Our last two tests named test_sdr_listen_for_silence and test_sdr_listen_for_lora_chirp can be found in the ./tests/test_chirps.py file. The first test, test_sdr_listen_for_silence, tests the outputted file written by rtl_fm contains zero bytes because no chirps are being sent. The second test tests that the outputted file written by rtl_fm contains more than 800000 bytes with chirps being sent. It is worth mentioning that we side stepped the threading issue by putting the rtl_fm into a Linux background process and then started the esp32 device chirping. This is not the best way to thread the rtl_fm process because the kill command that stops that process may kill more than just the process we are interested in. This image shows out final test trend with the last two tests added and the outputted rtl_fm files.

Going Forward

Here a list of things that can be done to improve and progress the tests ...

Summary

Hopefully this article outlines how a embedded device can be continually tested with Jenkins and python and provides some insight into the steps required to do this.

Amendment

The pytest-html ordering package was removed from this document and the code base due to a moderate security announcement which can be found here

TwitterX Youtube