Testing

From SailfishOS Documentation

Testing

When testing a change, it is important to keep the testing environment as stable as possible, to ensure that the contributor is testing the same code each time. As an example, synchronisation is greatly affected by connectivity issues, so when testing changes to synchronisation plugins, it is important to have a stable internet connection (e.g., strong wireless LAN connection) - unless, of course, it is precisely the connection-loss-handling which is being tested.


Debugging Output & Journal

Most application output (including debug logging as well as warnings or critical error information) is recorded in the systemd journal. It can be accessed via, e.g., devel-su journalctl -a or devel-su journalctl -af to follow output. The output can be piped through other tools (like grep) to filter out unwanted information.


Note that by default, the systemd journal throttles output from particularly noisy processes, which can be frustrating when trying to debug an application. The configuration can be changed by modifying /etc/systemd/journald.conf and setting the RateLimitBurst and RateLimitInterval settings as appropriate. For example, setting RateLimitBurst=5000 and RateLimitInterval=5s will cause systemd to throttle journal output from any process which emits more than five thousand lines of output in any five second interval. After modifying any systemd journal configuration settings, you should reboot the device.


In some cases, a software component will need to have extra logging explicitly enabled for it. Usually, that requires specifying a particular environment variable (e.g., QT_LOGGING_RULES="*.debug=true" can commonly be set to enable Qt debug logging categories, while other components will have specific runtime switches. The sync framework checks MSYNCD_LOGGING_LEVEL which can be set from 1 to 9, so MSYNCD_LOGGING_LEVEL=9 produces the most verbose output; the calendar framework checks if KCALDEBUG=1 is set). In other cases, a configuration file (usually located under /home/nemo/.config) will contain switches which can be set to enable more or less debugging. Unfortunately, due to the heterogeneous nature of the packages in Sailfish OS, there is no unified way to enable verbose debug logging, and so the contributor will in some cases have to read the source code to figure out what they need to do to make debug logging work. It may even require setting some #define in source code, and rebuilding the package.

Unit Tests

Any behaviour change made by a contributor should be matched with a new unit test, to ensure that a future code change doesn't break the functionality (i.e., a regression). In most cases, the unit tests are run automatically as part of the continuous integration gating process for the package, and so the contributor should ensure that their newly added unit test is added to the "tests.xml" file (if it exists in the repository) which tells "testrunnerlite" (the CI unit-test runner) to run the test as part of integration testing. The contributor should also run all of the unit tests manually, to ensure that their code change hasn't accidentally caused a regression in behaviour.


If a particular package doesn't include unit tests, a change may still be accepted by the maintainer even if the contribution doesn't include a unit test for that change. However, this should be considered an exception rather than the rule; where possible, unit tests should always be added. If a particular component lacks unit tests for existing functionality, this is a prime candidate for contribution from quality-focused community members! The more automatable the quality-assurance process is, the faster we can release new versions of Sailfish OS, and with higher confidence that no regressions were introduced.