Collaborative Development

From SailfishOS Documentation
Revision as of 10:49, 26 November 2016 by Sage (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Sailfish OS is mainly open source and almost all development on the platform itself happens in the open on public systems either in the Mer Project or in Sailfish OS repositories. There are many ways to contribute to Sailfish OS, by reporting bugs, fixing them, implementing new features, adding new unit tests and improving documentation (on this Wiki and for programming APIs). This document will focus on providing information related to software contributions, by listing a few common development tasks, and describing in detail how to work through some scenarios. After reading this document, the willing contributor should have the tools and knowledge required to contribute bug fixes, feature implementation, unit tests or documentation to Sailfish OS.


The community is welcoming and supportive but it's polite to get to know a little about how things are done to avoid wasting other people's time. You should be familiar with Sailfish OS Platform Development, commonly used development commands, have a Platform SDK setup and be able to build packages using a suitable target. The SailfishOS Source page talks about finding and downloading the source for any packages you need to work on.


Filing Bug Reports

One of the most common tasks in development and testing is that of filing bug reports. Any complex software codebase will have bugs or missing features which need to be recorded in a rigorous manner in order to ensure that they are correctly prioritised and then have their status tracked through the various stages of development, testing, verification and releasing. Information related to the bug or task must be searchable and indexable, and kept in one place. Sailfish OS utilises the Mer Project's Bugzilla instance https://bugs.merproject.org/ to track issues in its middleware stack for these purposes.


When filing a bug report, it is vital that the report contains all of the information required to progress a solution for the issue.


This includes:

  • on which type of device the bug was observed
  • what behaviour was expected
  • what behaviour was observed (i.e., fault description)
  • concrete steps to reproduce the issue
  • the software component / package which contains the bug (if known)


Depending on the issue, the area maintainer will categorise it as either a bug or a missing-feature task, and give the task a severity and priority. High priority tasks should be roadmapped based upon expected resourcing and the project critical path, among other factors.


Finding A Task

A community member or developer who wishes to contribute code (or documentation, or testcases, etc) to the project first needs to find a bug or task which they wish to contribute a solution for. This may include pre-existing bugs, or they may wish to report an issue (as described in the above section on filing bug reports) and work on it themselves. In most cases, they can simply search the bug tracker for open issues in the area they are interested in contributing to, and picking a bug from there. The bug report should contain enough information so that they can immediately begin working on a solution, although in some cases the reporter would not have known which specific component or software package contained the bug.


If the contributor decides to start working on a given issue, they should make a comment to that effect in the bug report, for the following reasons:

  • this facilitates discussion with other people who may have information to share about the issue
  • it prevents duplication of effort, and allows collaboration, in the case that multiple contributors may want to help solve the issue
  • it provides the wider community visibility into efforts to improve the software


Finding The Code

When a contributor is deciding whether to work on a task, they will usually want to find out where the code is first, so that they can see whether their skills will be helpful in solving the issue. This is particularly important in cases where a framework may have separate components which utilise different programming languages or core libraries in their implementation.


In some cases, the bug report will include the information needed (e.g., the package name). In other cases, the contributor must determine the package name and find the source for it themselves.


Sailfish OS uses git for source control and versioning. Once the contributor has found the appropriate source code repository (most likely from git.merproject.org), they should create a personal fork of that repository. Any changes they make will occur in that fork, and later on their changes can be merged from their personal fork into the upstream source code repository.


Building The Code And Testing Changes

Once a contributor has found the source code, they should build the package and deploy it to their device to familiarise themselves with the process prior to attempting to change the code. Once they are comfortable with the build and deployment processes, the contributor can then begin modifying the code and deploying their changes.


Initially, the changes might be aimed at increasing the amount of information which is logged, to provide more information about why a bug is occurring. Later on, the changes might be more functional, modifying the implementation of the component so that it no longer exhibits the incorrect behaviour. Every change must be tested carefully and unit tests should be added so that the quality of the package improves.


Contributing The Change

Fixing a bug locally is the first important step to resolving a bug in Sailfish OS, however that fix must then be reviewed, accepted, merged, built and released. This section explains each of the steps in that process, and describes some potential reasons why a particular contribution may not be accepted.


Commit policies

There are a number of policies which support the QA process in Sailfish OS. For example, the first line of commit should contain a changelog category, a brief description of the change, and a Mer bug number, adhering to the following format: "[component] Change description. Contributes to MER#XYZ". Mer and Sailfish OS use automation systems to construct changelogs, to verify that policies are met and that builds succeed when changes are made.


Read more about Mer and SailfishOS coding policies.


Creating A Merge Request

Once the contributor has finished fixing an issue, and they have tested it on their own device, they should commit their changes and push it to their personal fork of the upstream repository, adhering to the commit policies described above. This allows changelogs to be generated automatically, and for the commit history to contain meaningful back-links to the bug reports which led to the change being made.


Then, they should submit their change request to the upstream repository via a merge request. At this point, they should add as much information to the original bug report as possible (including a link to the merge request) so that interested people can review the change. The maintainer of the component will usually be automatically informed of the merge request, however it may be useful to ping the developer on IRC also, as synchronous discussion can be helpful when reviewing a change.


Review Feedback

Reviewers may find various issues with a contribution. Common issues include code-style violations, violating the component architecture, missing documentation, missing unit tests, performance regression or edge-case logic errors. Sometimes reviewers also misread or otherwise make mistakes during review. We are all human and make mistakes! We require that every contributor and reviewer be polite and considerate when reviewing contributions and discussing those changes, and ask contributors to understand that review comments are not intended to be a sign of lack of appreciation for the work they have done, but rather are intended to further improve the contribution by helping the original contributor in their goal of improving Sailfish OS.


Sometimes disagreement may occur between various people about what the correct behaviour should be. In those cases, the area maintainer will make a decision, after consulting the various stakeholders including community members and other contributors. In some rare cases, the maintainer may choose not to accept a contribution, either on technical (e.g., code quality) or subjective (e.g., UI design considerations) grounds. We ask that all parties respect the decisions of the maintainers if that occurs.


When developing user-facing features, remember to follow the user interface definition of done checklist and Qt and QML coding conventions.


Updating The Contribution

Usually the maintainer will want to merge the contribution after some small fixes (e.g., a missing code comment is added) are made. In that case, the contributor should make the change and amend the original commit where possible (to avoid filling the commit history with small commits), and then update their personal fork with the amended commit. The merge request should automatically be updated, and at this point, a developer or other community member with write access to the upstream repository will be able to merge the change.


Releasing The Change

At this point, the code or documentation change will have been applied to the upstream Sailfish OS repository via an accepted merge request. It will not, however, be included in a release, until a package is generated from the repository. For that to occur, the maintainer will tag the repository with the new package version tag, and a webhook will automatically inform the CI system that it needs to pull the new code, build the package, run the automated tests on it, and promote it to "testing" level. At that point, the original bug report should be updated with the package version which should contain the fix, future future reference.


From there, the entire release is tested manually and automatically, and eventually the new package version (or a later one) will be included in a Sailfish OS release. The time it takes for a new Sailfish OS release to be created depends on a variety of factors, and so no solid ETA (from when a contribution is merged until when it appears in a release) can be given. When the new release is shipped to customers, that release will contain your contribution.


Building in OBS

After the maintainer has tagged the repository, the CI system will be informed (via webhook) of the change. This will trigger a package build in OBS. OBS will attempt to build the package, with a build environment constructed specifically for that package after examining the build requirements which are listed in its RPM .spec file.


If the build fails, either due to a missing dependency specification in the .spec file, or due to a code error, the package will not be tested, and the developer is expected to respond to the failure notification and fix the issue.


If the build succeeds, a set of automated tests will be run (if specified for the package), allowing a first-stage quality gate to be applied. If those automated tests succeed, the package will be promoted to "devel" level in the repository to which that package belongs.


Promotion in OBS

When a package is promoted, it becomes accessible to any device which has that repository added to their update configuration (modifiable via the ssu command). There are different promotion layers which correspond to different release types:

  • The "devel" promotion layer corresponds to "development" devices. These are generally used by developers only. Developers and testers will use packages from these repositories, and when they agree that the content is stable enough, the packages will be promoted to "testing" level.
  • The "testing" promotion layer corresponds to "quality assurance" devices. These are generally used by quality assurance engineers only. These people will thoroughly test the package using an array of automated and manual test cases, and only when they are satisfied that the package is of high quality and causes no regressions will they allow promotion of the package from "testing" to "release" level.
  • The "release" promotion layer corresponds to "release" devices. These devices are used by consumers. Packages at this level are considered stable and able to form part of a release image during the next release cycle.


QA images in IMG

Before a release image is made, the release level packages are combined into a series of release-candidate images for quality-assurance purposes. These images are tested thoroughly in order to catch any regressions or other major issues which managed to slip through the previous quality gates. Images are built by the Image Creator.


Contribution Example

For this example, we will consider a hypothetical bug which affects the Sailfish Browser. The community member "Alice" discovers that for some content-aggregator sites, the wrong User Agent string seems to be used when fetching content. She decides to investigate and contribute a fix. These are the steps she takes:


1) She files a bug on bugs.merproject.org about the issue, describing which device she sees the issue, which sites seem to trigger the issue, and which version of Sailfish OS she sees the issue on.


2) She comments on the bug to say that she will investigate the issue.


3) She searches for the upstream code repository, and finds https://github.com/sailfishos/sailfish-browser and notes that it depends on embedlite-components-qt5. After browsing that source code, she realises that the user agent override behaviour is defined in code from the embedlite-components-qt5 package.


4) She searches for that package on build.merproject.org and finds https://build.merproject.org/package/show/mer-core:devel/embedlite-components-qt5 whose _service file shows the upstream repository URL https://git.merproject.org/mer-core/embedlite-components.git


5) She creates a personal fork of that repository using the web interface at https://git.merproject.org/mer-core/embedlite-components/


6) She clones that personal fork to her local host machine via `git clone ssh://git@git.merproject.org:2222/alice/embedlite-components.git alice-embedlite-components`


7) She attempts to build the package, and updates her SDK target so that she can build it.

   # cd into her local clone
   ~ $ cd alice-embedlite-components
   # enter Platform SDK chroot
   ~/alice-embedlite-components $ /srv/mer/sdks/sdk/mer-sdk-chroot
   # attempt to build the package for her SailfishOS-armv7hl target
   MerSDK ~/alice-embedlite-components $ mb2 -t SailfishOS-armv7hl build
   No provider of 'xulrunner-qt5-devel >= 31.7.0.14' found.
   Building target platforms: armv7hl-meego-linux
   Building for target armv7hl-meego-linux
   error: Failed build dependencies:
   	xulrunner-qt5-devel >= 31.7.0.14 is needed by embedlite-components-qt5-1.0.0-1.armv7hl
   	pkgconfig(nspr) is needed by embedlite-components-qt5-1.0.0-1.armv7hl
   # she knows that she needs to update her SDK target.
   # to find out which ssu repository provides those packages, she now ssh's into her device,
   # and runs `zypper info` (can also use `pkcon get-details` or search on OBS).
   ~/alice-embedlite-components $ ssh nemo@device
   [nemo@SailfishOS ~]$ zypper info xulrunner-qt5-devel
   Loading repository data...
   Reading installed packages...
   Information for package xulrunner-qt5-devel:
   --------------------------------------------
   Repository: apps-mw
   Name: xulrunner-qt5-devel
   Version: 31.8.0.8-10.44.19.jolla
   Arch: armv7hl
   Vendor: meego
   Installed: No
   Status: not installed
   Installed Size: 31.5 MiB
   Summary: Headers for xulrunner
   Description: 
     Development files for xulrunner.
   # she now knows that it comes from the "apps-mw" repository.
   # she goes back to her Platform SDK chroot terminal, and
   # enters ScratchBox2 to update the ssu repositories.
   MerSDK ~/alice-embedlite-components $ sb2 -t SailfishOS-armv7hl -m sdk-install -R
   # might be required to register her SDK depending on her SDK setup.
   [SB2 sdk-install SailfishOS-armv7hl] user@host alice-embedlite-components # ssu r
   # list the available repos
   [SB2 sdk-install SailfishOS-armv7hl] user@host alice-embedlite-components # ssu lr
   ## snipped ##
   Disabled repositories (global, might be overridden by user config):
    - apps-mw ... https://download.jollamobile.com/pj:/apps:/mw/latest_armv7hl/
   ## snipped ##
   # she enables the repository.
   [SB2 sdk-install SailfishOS-armv7hl] user@host alice-embedlite-components # ssu er apps-mw
   # she refreshes the available package list
   [SB2 sdk-install SailfishOS-armv7hl] user@host alice-embedlite-components # zypper ref -f
   # she installs the required packages into her SDK target
   [SB2 sdk-install SailfishOS-armv7hl] user@host alice-embedlite-components # zypper in xulrunner-qt5-devel nspr-devel
   # now she is ready to go back to the MerSDK chroot and use mb2 to build the package again.
   [SB2 sdk-install SailfishOS-armv7hl] user@host alice-embedlite-components # exit
   MerSDK ~/alice-embedlite-components $ mb2 -t SailfishOS-armv7hl build
   # success!
   MerSDK ~/alice-embedlite-components $ exit


8) Now that she has successfully built the package locally, she tests deploying the package.

   ~/alice-embedlite-components $ scp RPMS/embedlite-components-qt5-1.0.0-1.armv7hl.rpm nemo@device:~/
   ~/alice-embedlite-components $ ssh nemo@device
   [nemo@SailfishOS ~]$ devel-su rpm -Uvh --force embedlite-components-qt5-1.0.0-1.armv7hl.rpm


9) After rebooting the device and opening browser, she sees that it works. She is now ready to begin development. She makes changes, adds debug statements, deploys the package to the device, and retrieves the logging information from the journal with `devel-su journalctl -af | grep browser`.


10) After some hours of work, she has resolved the issue, and commits her fix locally. She then pushes the commit to her personal fork, and creates a merge request to the upstream repository. This will look something like https://git.merproject.org/mer-core/embedlite-components/merge_requests/7


11) She now updates the bug report with her findings, and a link to her merge request.


12) After receiving some review comments, she updates her commit with `git commit --amend`, force pushes it to her personal fork via `git push origin master:master --force` and comments in the merge request that it has been updated.


13) The maintainer merges the patch, tags it, and comments on the original bug report the tagged version which should be in a future release.


Contacting Developers

Developers of Sailfish OS (including paid employees of Jolla or other companies contributing to Sailfish OS, and community members who contribute to the development of Sailfish OS) can generally be contacted in #sailfishos@freenode IRC, via email, via bugs in the Mer Project Bugzilla, or via comments and issues in the Mer Project Gitlab repositories. The email address of a given developer can be discerned by reviewing the commit history in a source code repository to which they contribute code.


Here is a brief list of core areas and some of the developers who have contributed significantly to those areas, as well as the IRC nickname of each developer so that they can be contacted on IRC. The list is not intended to be exhaustive or complete, but should give newcomers to Sailfish OS a starting point on who to contact if they wish to contribute to a certain area.


  • Hardware adaptation: Marko Saukko (sage), Simo Piiroinen (spiiroin)
  • Mer Core and Hybris: David Greaves (lbt), Carsten Munk (Stskeeps), Simonas Leleiva (sledges)
  • Multimedia: Mohammed Hassan (msameer), Andrew den Exter (denexter)
  • Connectivity: Slava Monich (slava)
  • Telephony: Jarko Poutiainen (jpoutiai), Martin Jones (mjones)
  • Audio: Juho Hamalainen (jusa)
  • Bluetooth: Bea Lam (blam), Jarko Poutiainen (jpoutiai)
  • Systemd: Igor Zhbanov (IZh)
  • Qt: Robin Burchell (w00t), Martin Jones (mjones)
  • Homescreen and Compositor: Joona Petrell (jpetrell), Andrew den Exter (denexter), Gunnar Sletta (sletta)
  • Alarms: Pekka Vuorela (pvuorela), Slava Monich (slava), Simo Piiroinen (spiiroin)
  • Calendar: Pekka Vuorela (pvuorela)
  • Text Input and Localization: Pekka Vuorela (pvuorela)
  • Email: Matt Vogt (mvogt)
  • Contacts: Matt Vogt (mvogt), Chris Adams (chriadam)
  • Notifications: Matt Vogt (mvogt)
  • SDK: Jarkko Lehtoranta (Jare)
  • Synchronisation: Chris Adams (chriadam), Bea Lam (blam)
  • Web Browser: Raine Makelainen (rainemak)
  • Web Services: Niels Breet (xfade), Pami Ketolainen (pketolai)
  • Packaging, Promotion, QA: Matti Kosola (mkosola), Wouter van Heijst (LarstiQ), Jari-Pekka Walden (jwalden)
  • Translations: Jarkko Ahonen (jahonen), Simonas Leleiva (sledges)


Specific Collaboration Efforts