No one is immune from shipping critical bugs to production. No matter how extensive your test coverage, no matter how thorough your QA, no matter how much you put behind feature flags — your mobile team will ship irreversibly-bad updates that break things.

This is especially problematic for mobile because you can’t roll back immediately like your web or backend friends can. Rather, once you detect a critical bug, your team has to triage the issue and assign it to the right team members, implement fixes (or carefully revert the offending changes), compile one or more RCs, upload to the stores, submit for review, then cross your fingers and pray to the store review gods for a speedy review. All of this takes time — hours in the best of cases, but potentially as long as days. And the negative impacts of even just a few hours with a critically broken app can be eye-watering: tens or hundreds of thousands of dollars in lost revenue, a flood of negative user reviews, and overwhelmed CX or ops folks.

But what if it didn’t have to be that way? What if you could instantly resolve a critical issue in prod, with a single click? We set about building the answer. Runway’s new rollback feature means your team is always just one click away from resolution of critical bugs in prod. We handle the busywork needed to prepare and stage rollback builds, each and every release, enabling a “break glass in emergency” action that is always there when your team needs it. Read on for the full story on how rollbacks work in Runway, and the kinds of problems we’re solving for. 

Recover from production issues in zero time — literally

With Runway’s rollback automations, we ensure all the pieces are in place to allow you to roll back instantly if a critical issue is discovered in prod. In the background, for each and every release cycle, Runway automatically re-signs the previous stable, live build and gets it prepped for release — including submission for review as soon as possible (iOS only) — to get all possible sources of delay out of the way. Then, if something goes terribly wrong in production with the release you were just working on, you have the rollback build ready and waiting for a single-click release in Runway.

Here’s how the rollback flow works in a bit more detail:

1. Your team is busy getting your next regularly-scheduled release out the door. Let’s call it version 1.2.0.
2. In the background, Runway takes your latest live build in production — say, 1.1.0 — and re-signs it, bumping the version to 1.2.1 and incrementing the build number as necessary.
3. Your team releases 1.2.0 as scheduled. Nice!
4. The 1.2.1 rollback release is surfaced in Runway, just in case. For iOS, we also preemptively create a new version for the possible rollback in App Store Connect and submit the rollback build for review.
5. Your team is monitoring the rollout of 1.2.0 and identifies a critical issue 🚨You halt the rollout.
6. Right next to the button you used to halt the rollout, there’s another button to release the rollback build that’s ready and waiting. Click that, and you’re all set.

With rollback builds primed and ready to go, your team can address critical bugs in prod with a single click — cutting out all the lead time needed to investigate and implement a fix, get builds compiled and uploaded, and make it through the store review process.

Take the pressure out of hotfixes, so you can triage more fully and fix more confidently

Even if you decide to leverage Runway to quickly ship a rollback to address an issue in prod, it doesn’t mean you’ll necessarily wait until your next regularly-scheduled release to roll forward and issue a fix for the original issue. In fact, it’s likely you’ll often use rollbacks in conjunction with roll-forward hotfixes, and the two complement each other. Because rollbacks allow you to instantly stabilize the situation when you detect critical bugs, your team doesn’t have to scramble to get a hotfix prepared and out the door. Instead, you can take your time in triaging, investigating the root cause, implementing the necessary fixes, and then working through your hotfix release process. Removing the commotion and pressure that’s typically involved in hotfixes results in better hotfixes, and helps you avoid the hotfixes-on-hotfixes(-on-hotfixes) rabbit hole that can swallow teams whole.

More predictable resolution timelines, and fewer unanswerable “when??” questions from stakeholders 

Every team knows the feeling of dread that hits when you’ve shipped a breaking change and leadership shows up in your #mobile Slack channel. Mistakes happen, sure, but once a mistake has been identified, stakeholders are often breathing down your neck wondering how long until your team can get it addressed. Is the fix in yet? Is the build approved by Apple? How about now? With rollbacks in Runway, you can actually respond to these kinds of asks and give stakeholders confidence that the situation is under control. When you choose to ship a rollback, the answer to the inevitable “when” question can be “already”, and stakeholders can rest assured that the situation has been stabilized.

When you need a rollback, it’s there for you (batteries included). When you don’t, we’ll handle all the cleanup.

Runway’s rollbacks functionality encompasses quite a few dependencies and steps which, if approximated manually each and every release, would require a lot of work — not just to prepare each rollback, but also to clean up unused rollbacks along the way (to unblock subsequent releases). In handling all the heavy lifting, Runway unlocks the massive upside in situations where your team would want to deploy a rollback, while eliminating the continuous manual wrangling that is required along the way just to make that possible. As a result, you no longer have to choose between investing huge amounts of time and effort into setting up rollbacks manually or else forfeiting the safety net instant rollbacks provide.

With rollbacks, Runway is giving your mobile team the quickest possible way to respond to critical issues in production. Instead of scrambling to identify, implement, and ship a fix, you can address the situation with a single click. Stakeholders won’t hover anxiously wondering when a resolution will land, and your team will be able to triage and implement fixes more confidently, without all the pressure that typically accompanies hotfix situations. Plus, rollbacks mitigate the serious negative impact to revenue and app store reviews that comes with a buggy app being in users’ hands for even just a couple of hours.

As always, we would love to hear your thoughts! Get in touch at hello@runway.team or, better yet, schedule a live tour of rollbacks with our team!

‍

No one is immune from shipping critical bugs to production. No matter how extensive your test coverage, no matter how thorough your QA, no matter how much you put behind feature flags — your mobile team will ship irreversibly-bad updates that break things.

This is especially problematic for mobile because you can’t roll back immediately like your web or backend friends can. Rather, once you detect a critical bug, your team has to triage the issue and assign it to the right team members, implement fixes (or carefully revert the offending changes), compile one or more RCs, upload to the stores, submit for review, then cross your fingers and pray to the store review gods for a speedy review. All of this takes time — hours in the best of cases, but potentially as long as days. And the negative impacts of even just a few hours with a critically broken app can be eye-watering: tens or hundreds of thousands of dollars in lost revenue, a flood of negative user reviews, and overwhelmed CX or ops folks.

But what if it didn’t have to be that way? What if you could instantly resolve a critical issue in prod, with a single click? We set about building the answer. Runway’s new rollback feature means your team is always just one click away from resolution of critical bugs in prod. We handle the busywork needed to prepare and stage rollback builds, each and every release, enabling a “break glass in emergency” action that is always there when your team needs it. Read on for the full story on how rollbacks work in Runway, and the kinds of problems we’re solving for. 

Recover from production issues in zero time — literally

With Runway’s rollback automations, we ensure all the pieces are in place to allow you to roll back instantly if a critical issue is discovered in prod. In the background, for each and every release cycle, Runway automatically re-signs the previous stable, live build and gets it prepped for release — including submission for review as soon as possible (iOS only) — to get all possible sources of delay out of the way. Then, if something goes terribly wrong in production with the release you were just working on, you have the rollback build ready and waiting for a single-click release in Runway.

Here’s how the rollback flow works in a bit more detail:

1. Your team is busy getting your next regularly-scheduled release out the door. Let’s call it version 1.2.0.
2. In the background, Runway takes your latest live build in production — say, 1.1.0 — and re-signs it, bumping the version to 1.2.1 and incrementing the build number as necessary.
3. Your team releases 1.2.0 as scheduled. Nice!
4. The 1.2.1 rollback release is surfaced in Runway, just in case. For iOS, we also preemptively create a new version for the possible rollback in App Store Connect and submit the rollback build for review.
5. Your team is monitoring the rollout of 1.2.0 and identifies a critical issue 🚨You halt the rollout.
6. Right next to the button you used to halt the rollout, there’s another button to release the rollback build that’s ready and waiting. Click that, and you’re all set.

With rollback builds primed and ready to go, your team can address critical bugs in prod with a single click — cutting out all the lead time needed to investigate and implement a fix, get builds compiled and uploaded, and make it through the store review process.

Take the pressure out of hotfixes, so you can triage more fully and fix more confidently

Even if you decide to leverage Runway to quickly ship a rollback to address an issue in prod, it doesn’t mean you’ll necessarily wait until your next regularly-scheduled release to roll forward and issue a fix for the original issue. In fact, it’s likely you’ll often use rollbacks in conjunction with roll-forward hotfixes, and the two complement each other. Because rollbacks allow you to instantly stabilize the situation when you detect critical bugs, your team doesn’t have to scramble to get a hotfix prepared and out the door. Instead, you can take your time in triaging, investigating the root cause, implementing the necessary fixes, and then working through your hotfix release process. Removing the commotion and pressure that’s typically involved in hotfixes results in better hotfixes, and helps you avoid the hotfixes-on-hotfixes(-on-hotfixes) rabbit hole that can swallow teams whole.

More predictable resolution timelines, and fewer unanswerable “when??” questions from stakeholders 

Every team knows the feeling of dread that hits when you’ve shipped a breaking change and leadership shows up in your #mobile Slack channel. Mistakes happen, sure, but once a mistake has been identified, stakeholders are often breathing down your neck wondering how long until your team can get it addressed. Is the fix in yet? Is the build approved by Apple? How about now? With rollbacks in Runway, you can actually respond to these kinds of asks and give stakeholders confidence that the situation is under control. When you choose to ship a rollback, the answer to the inevitable “when” question can be “already”, and stakeholders can rest assured that the situation has been stabilized.

When you need a rollback, it’s there for you (batteries included). When you don’t, we’ll handle all the cleanup.

Runway’s rollbacks functionality encompasses quite a few dependencies and steps which, if approximated manually each and every release, would require a lot of work — not just to prepare each rollback, but also to clean up unused rollbacks along the way (to unblock subsequent releases). In handling all the heavy lifting, Runway unlocks the massive upside in situations where your team would want to deploy a rollback, while eliminating the continuous manual wrangling that is required along the way just to make that possible. As a result, you no longer have to choose between investing huge amounts of time and effort into setting up rollbacks manually or else forfeiting the safety net instant rollbacks provide.

With rollbacks, Runway is giving your mobile team the quickest possible way to respond to critical issues in production. Instead of scrambling to identify, implement, and ship a fix, you can address the situation with a single click. Stakeholders won’t hover anxiously wondering when a resolution will land, and your team will be able to triage and implement fixes more confidently, without all the pressure that typically accompanies hotfix situations. Plus, rollbacks mitigate the serious negative impact to revenue and app store reviews that comes with a buggy app being in users’ hands for even just a couple of hours.

As always, we would love to hear your thoughts! Get in touch at hello@runway.team or, better yet, schedule a live tour of rollbacks with our team!

‍