In our last blog post, we examined some of the factors that make mobile app releases difficult and we began to consider how teams can optimize their mobile release process in light of these complexities and constraints.
In this post, we’ll look more concretely at all of the steps that compose a typical mobile release pipeline and we’ll explain the importance of each. By the end of this post, you may start to recognize relative strengths and weaknesses of each step in your team’s current process, or come to realize you are lacking some steps entirely.
In subsequent blog posts in this series, we’ll dig into each step in turn, allowing you to address any shortcomings in your current setup and move towards a more ideal mobile release process.
A note on release styles
Most teams adopt one of two styles when it comes to defining their mobile app releases:
“Feature-based” - Some particular set of feature work is designated for a given app release, and that release only goes ahead when all of the designated work is ready to ship.
“Scheduled” - App releases happen on a fixed schedule and aren’t held up for any feature work; whatever code happens to be ready and merged by the time the release is cut will go out. This style has also been described as the “release train".
There are advantages and disadvantages to each, and a team’s choice often depends on their size and on the kind of app they build. Although release style affects when releases kick off, the release process itself looks very similar for both styles post-kickoff.
Release branching: Kick off the app release
The first step in the mobile release process is the one which brings a release into existence: release branching. Fundamentally, when starting a release teams should be isolating production-ready code from any code still under development. Often this is accomplished by creating a dedicated “release branch” for each release. In some cases, teams will use their “master” branch as a snapshot of the code slated for production, while continuing everyday development on a separate development branch.
By isolating production-ready code destined for an imminent release, you limit changes to code that’s about to ship (de-risking the release) while allowing other development to continue in parallel. Generally, only small additions or fixes should be merged into the release branch, while developers working on bigger changes or unrelated features can continue merging elsewhere. Usually this distinction is safeguarded more formally using branch protection. Post-release, the use of release branches allows teams to more easily push hotfixes without pulling in lots of unrelated, new code that has been merged in the interim.
We’ll cover the different ways to approach your mobile release branching strategy in more detail in a subsequent post.
Continuous Integration & Delivery (CI/CD): Build, test & distribute
Continuous Integration (CI) – Build & Test
With a release now defined and its code isolated, the team needs some way of validating any changes to the code in the release branch. Enter Continuous Integration (CI).
CI is the practice of regularly merging changes into shared code, and then automatically building and testing the result. It is usually leveraged throughout the mobile app development cycle, but during releases it is especially valuable because the cost of inefficiencies and mistakes is higher. CI allows teams to collaborate on and finalize releases with more confidence, by catching any conflicts or integration errors that arise during merges into the release branch.
A robust CI setup further improves stability through the app release process by standardizing the build environment across each build. Whether for iOS or Android, release builds require a specific configuration (build settings, provisioning profiles, signing certificates, etc.) and a consistent integration environment; if these requirements aren’t met, flakiness and risk increase. Often teams turn to third party, cloud-based CI services which take care of spinning up consistent, reproducible integration environments. Such services also manage the hardware behind the build servers, which can be cumbersome to set up and maintain in-house (iOS integration servers are especially tricky, with unique hardware and operating system requirements).
Generally, teams should set up their CI pipeline to build and test on their release branch any time code is committed to the branch. Most CI platforms offer integrations with version control systems (e.g. GitHub) to handle this automatically.
Continuous Delivery (CD) – Distributing a Release Candidate
Continuous delivery (CD) describes the process of automatically generating and distributing a build (usually for testing) after each successful integration with your base branch. Most cloud-based CI services also cater to this CD step, and many leverage developer tools like fastlane to help connect the pieces of the CI/CD pipeline.
An important rule enforced by most distribution destinations (including Testflight and App Store for iOS, and Google Play for Android) is that no two release candidates can have the same version number and build number. This means build numbers must change with each new release candidate the CI/CD pipeline delivers. This incrementation can be done manually, but it’s often executed automatically as a step in the team’s CI/CD workflow.
Testing the Release Candidate
Arguably, the most important step of the mobile app release process is testing. Of course, testing should be undertaken throughout the app development process, but testing of a final release candidate is especially crucial. Because mobile releases can’t be rolled back or hotfixed easily, teams should do their utmost prior to shipping to ensure that new code functions properly and that no other areas of the app have broken as a side effect of changes.
Because the mobile app team itself is so close to its product, team members are inherently biased and it can be more difficult for them to spot bugs or areas of friction for users. For that reason, during testing it’s very valuable to put release candidates in front of an audience that’s further from the product. This lends itself to gathering more open-ended feedback in addition to basic bug reporting. Distributing builds to a controlled group of users for testing in this manner is referred to as “beta testing”.
Internal Beta Testing
A release build is distributed to a group of users within the company. These are people that are familiar with the overall product but may not use the app on a regular basis. Internal beta testing also promotes transparency, because it keeps the rest of the company in the loop on new features and changes to their app.
External Beta Testing
A release build is distributed to a pre-selected group of users outside the company. These users may be sourced from an internal list of “power users”, or randomly selected from your general user base. External testers have the least insight into the inner workings of your company and will therefore provide the least biased feedback, and generally interact with the app in the most “real world” way.
Dogfooding
Dogfooding is a specific subcategory of internal beta testing: users inside the company who are otherwise regular users of the (production) app, or a competing product, use the release build for its intended purpose. Instead of testing contrived flows, they use the app just as they otherwise would. Dogfooding is used heavily by teams at Microsoft and Google where, for example, developer tools they make can be used by their own development teams.
Regression Testing
Regression testing entails execution of a set, or “script,” of specific functional tests on a release candidate. Usually these scripts are designed to cover core and critical flows in the app (e.g. sign-in, or checkout in e-commerce) to make sure they haven’t been unexpectedly affected or broken by changes in the release. Regression testing can be carried out manually, usually by quality assurance (QA) analysts on the team, or automatically, with UI testing on virtual devices or via automated device farms.
Metadata, release notes & screenshots: App Store and Google Play presence
With every release, both the Apple App Store and the Google Play store allow teams to share release notes, update their app screenshots, and modify various other aspects of store presence (e.g. description and search keywords). Choices here can affect your app’s visibility in search results and its appeal to prospective new users.
With more and more users opting into automatic app updates, some teams feel that it’s not worth spending much time or effort on release notes and screenshots. For example, Facebook has controversially chosen not to write descriptive release notes. Others maintain that more involved release notes and screenshots are an extension of the app’s branding and can offer an additional channel through which to connect with and excite users.
Approvals
Many mobile app teams have an additional “gate” in their release process that calls for sign-off by one or more stakeholders. In some cases, this is as simple as a final “O.K.” from the overseeing parties. In other cases, a more involved audit takes place (e.g. security or privacy compliance). Often this entails some combination of release build, code diff, and changelog being packaged up and delivered to a different team within the company, or to an external partner.
Submission & app review
Upon successful completion of all of the preceding steps, the team is finally ready to submit the app! The final release candidate build is uploaded to the store, the metadata and screenshots are added in, and the finished product is packaged up for subsequent review by the store’s moderators. Turnaround time for app reviews can vary from hours to days to weeks, and rejection is not uncommon. Apple kicks back 40% of submissions; Google rejects around 55%. The potential reasons for rejection are diverse, and even experienced teams with an established store presence should be prepared for occasional rejections. Some app rejections will require the team to address issues in the build and restart the entire release process from the beginning. Others just call for changes to metadata or answers to questions from the review team.
Assuming all goes well and the app is approved, teams have options for how to release the new version to their users. Apple and Google both support immediate or phased rollout styles, and many teams like taking advantage of the latter to ensure the new version is behaving as expected before it’s in front of their entire user base.
Putting it all together: how to ship mobile apps with less friction and more confidence
Clearly, the full mobile release process is quite involved and there are many dependencies and potential blockers between steps in the process. Tools and team members involved in any one step are often siloed from those involved in other steps, preventing the existence of a source of truth and making it difficult to monitor the state of a release at a glance. Careful coordination and effective collaboration is required, and it must occur across disciplines: throughout the app release lifecycle, engineers, product and project managers, designers, QA analysts, and stakeholders external to the team or even the company are all called upon.
As we continue this series, future blog posts will cover each step in the mobile release process in turn and consider how they can be streamlined and more effectively connected to other steps in the process.
