At Atomic, we build a lot of systems that incorporate device firmware talking to software. We also commonly interface with other firmware or software teams.
I believe every project that involves firmware talking to software should include a high-level suite of tests that are written against the firmware’s interface to the software (e.g. Bluetooth, Ethernet, USB, etc). The tests should exercise the complete set of functionality that is exposed to the software. Writing these tests is not always straight forward because they commonly require human interaction with the physical device, but they are invaluable to a product team.
Unfortunately, these types of test suites are not always prioritized by product managers. They should be, and here is why.
1) It proves that the firmware works.
The main purpose of the test suite is to prove that the firmware and hardware function properly. Ideally, the test suite should be decoupled from the software product and created by the firmware team. These tests lead to a lot less finger pointing between the software and firmware teams.
2) It’s easier to debug the complete system.
When issues arise in a complex system, being able to isolate and reproduce the problem is the first step to resolving them. Having a set of high-level firmware tests allows the team to quick determine if the issue is caused by the firmware or software. For the same reason, it’s also extremely valuable for the software to have a hardware/firmware simulator, but I’ll save that discussion for another post.
3) It allows less-technical team members to test the firmware.
Very few people are skilled enough to read a protocol analyzer or shove a series of bytes into a firmware program. But lots of people are smart enough, with a little training, to set up and run a firmware test suite. The ability to get firmware testing time out of non-firmware engineers is huge leverage for any team.
In order to make your firmware test suite more realistic and not a complete nightmare to author, you should leverage the fact that you’ll have a smart person running the test suite. When you need to interact with a device (e.g. push a button, plug in a cord, etc), simply prompt the human to do that step and then continue the automated suite. In some cases, you may even require them to physically observe a change on the hardware and report their observations before continuing.
4) It enables regression testing.
During the product lifecycle, you’ll most likely rev your hardware or firmware. When that happens, the firmware test suite will be your best friend for the validation process.
5) It gives you a leg up on production testing.
Testing firmware during the manufacturing process is an engineering best practice. I’ve witnessed portions of firmware test suites being reused for end-of-production-line hardware device testing. Having the firmware test suite in place before you begin the manufacturing process gives you a head start on that effort.
6) It helps with field testing.
If you have a sophisticated hardware product, chances are you’ll need to provide some onsite or remote support. Having a firmware test suite will allow you to quickly identify issues with the hardware in the field.
If you have a war story about how a firmware test suite saved your life during a project, or a story about how the lack of such a test suite caused you pain, please share it in the comments below.
Twenty years ago, with two friends of mine, we had started a small company that was developing business real-time systems for professional fleets of vehicles (e.g. taxi dispatch, etc.). We had subcontracted design and development of the on-board device to a well-known radio communication equipment manufacturer. When I received it, one of the first things I did was exactly what you describe in your article: to develop a test suite aimed at validating firmware functions. And after a few tests, two major bugs were revealed. One was an hardware bug (the device designer had decided to multiplex two external serial links on the same UART, while it could happen that data was received on the two links at the same time…), the other one a software bug (the protocol stack handling serial communication over the two multiplexed serial links was not reentrant).
Having clearly identified those two bugs, we were able to implement a software workaround for the HW bug, and to correct the software bug. This was not the perfect solution, but allowed us to deliver our system on time. For our next customer, we designed our own on-board device :-)
So, I fully agree with your article. And if I try to say it more generally: when planning to integrate a component you don’t have experience with, plan spending time validating it before integration (who said “unit testing”? :-) ), developing any software and hardware required to perform this validation.
Thanks for sharing your story, Pascal.
Hi,
What kind of tool can you use to automate your firmware test suite ? Is there any off the shelf tool that can be used ?
Thanks,
Celine
Hi Celine, I’ve used a simple unit testing harnesses in the past (e.g. rspec, nunit, etc.). I haven’t used any off the shelf tools. I unsure if any good ones exist. Lots of the testing that I’ve done is at the protocol/message passing layer. In many situations the protocol is specific to the project.
i’m planing to test Blue tooth device.. On holding bluetooth device.. it sends some signal which we are trying to record. So how test these w.r.t to Blue Signla or Through put or other things.. like signal loss etc.