Solving a problem is hard when people don’t realize it’s a problem. It’s also hard when people do realize it’s a problem, but they believe they’ve already solved it. In telling the Runway story, we have to contend with both perspectives, each one a different side of the “category creation” coin. And we understand why this is. It took my co-founders and I quite a while — our entire careers in mobile, really — to realize that there was a problem, and that it wasn’t, in fact, solved.
On most of the mobile teams we were a part of, we tacitly accepted the wasted time and crossed wires that came with each release cycle. Bouncing back and forth between different tools, chasing down marketing for updated copy and stakeholders for required sign-offs, clicking around gingerly in App Store Connect and Play Console… Productivity, happiness, and general sanity suffered, but we all accepted that this “releases problem” was just the cost of doing business when it came to shipping apps.
Everyone dreaded their turn as release pilot/driver/manager/captain, and when their turn did roll around, they bid farewell to their product work and resigned themselves to a week or two of boring admin, cat-herding, and manual tasks. Occasionally, a disgruntled release driver would have an epiphany and jump into Slack:
<chat-blue>"That step is something we should really automate."<chat-blue>
<chat-red>"Agree. It seems silly to do it manually."<chat-red><giant><giant>
<smallen>this and the exchanges below happened in real life, on Slack circa early 2020s<smallen>
And then everyone would carry on with their day, and nothing ever changed.
At some of our former teams, a collective recognition of the mobile release problem did eventually coalesce. And so, vague attempts at solutions were made. Checklists were cobbled together in massive spreadsheets, and build pipelines were spun up and crammed with increasingly more stuff. Some of the low-hanging fruit was taken care of, the most egregious manual tasks were automated, and people were satisfied that the problem was solved. But it wasn’t.
<chat-green>"I’ve never been on call during a release before. Can someone help me out with the checklist?"<chat-green>
<chat-gray><Thread with 20 replies follows><chat-gray>
<chat-blue>“We’ve been approved, now just waiting for QA and deploy approval.”<chat-blue>
<chat-red>“I finished QA approval yesterday and marked it as done. Was there something I missed?”<chat-red>
<chat-blue>“Oh okay, I didn’t look. Lol. I assumed it was still going.”<chat-blue>
<chat-blue>Person A: “Hello Person B! Person C is preparing the release and we need release notes for the App Store. Can you help with that or point us in the right direction? <Link to massive spreadsheet containing release notes>”<chat-blue>
<chat-red>Person B: “Person D is your person for release notes!” <chat-red>
<chat-blue>Person A: “Person E, any highlights for Person D for the release notes?”<chat-blue>
<chat-green>Person D: “Sorry!! Seeing the release schedule on my calendar now. I’ll be on top of the next one. And yes, let me know what we want to talk about in the notes”<chat-green>
<chat-orange>Person E: <Proceeds to list out by hand everything that’s shipping with the release><chat-orange>
With each recurring exchange on Slack, with each delayed release and each unnecessarily buggy binary shipped, our eyes began opening and we finally took a step back. The problem we thought we had addressed with checklists and CI/CD was much broader, more fundamental, more human — and still unsolved. It was also a bigger problem than we had realized: the true cost of release overhead is often hard to detect, because it’s a slow but steady drip of wasted time and resources that only becomes acute when you zoom out beyond a single, isolated release. It’s the kind of thing that’s easy to overlook or ignore against the backdrop of overall mayhem that is a product team executing at a high level and shipping lots of cool stuff.
As we started talking to more mobile teams about their own experiences with releases, we found that they largely reflected our own. People’s thinking ranged along that spectrum from not recognizing the problem in the first place, to passively accepting it as an intractable part of app development, to actively engaging with it but feeling it was solved by an existing patchwork of spreadsheets, CI/CD, and sheer humanpower. We wondered how to best convey our learnings, and how to make clearer the fact that there could be a better way.
As the Runway product has matured and proven itself in the field, it’s gotten much easier to show-not-tell what a real mobile release solution looks like. But in order to better frame and talk about the releases problem, and to flesh out the shortcomings of what some perceive as adequate existing solutions, we’ve also spent more time probing the status quo and thinking about things a bit more academically.
Here are some highlights of that exercise in discovery — a handful of things we wished our old selves and teams had insight into back then.
DevOps is a north star, but is not itself a solution
Release process isn’t a particularly exciting domain for most product teams. DevOps isn’t either, but at least it’s buzzy and talked-about. Those are helpful traits when you’re approaching a problem space that’s often overlooked and which lacks much real prior art. We found that the philosophies and principles underlying the DevOps movement capture a lot of the feelings and opinions we’ve formed around the mobile release problem. And, the DevOps framework has helped us talk about and tackle the releases problem space. I wrote about DevOps in more detail previously, but here again are some of the most relevant tenets:
- Reduce silos within teams and across organizations. Bridge gaps between roles and disciplines. (Extremely relevant to releases, which play out across entire cross-functional teams and require visibility for, and input from, many different players.)
- Break up long-running tasks and make linear processes more parallelized and asynchronous. (A lot of the wasted time, overhead, and failure points in release cycles arise when dependencies and gates force a delicate, synchronous chain of events.)
- Empower teams and individuals to handle themselves those tasks they would have previously had to bug other people for. (See also: designers throwing new screenshots or marketing passing release notes over to engineers to then update in the app stores.)
- Automate as much as possible. (Enough said.)
You’ll notice that this list represents a collection of goals to aim for, but not concrete methodologies to implement. DevOps is a north star, but it is not itself a solution. It’s up to you and your team to figure out the how, in tangible and executable form. That’s the hard part, and many teams don’t get that far (which is why, for many, DevOps remains a buzzword and not much more than that).
CI/CD is not enough
Among teams that do recognize the releases pain point and who mobilize to address it, there’s a tendency to look at CI/CD as a catch-all solution. Teams spend time and resources stringing together (and maintaining) Swiss army knife build pipelines, and they feel that the problem is thus solved. Often, there’s a sense that CI/CD somehow equals DevOps. Avoid this trap. Robust CI/CD can certainly play a big role in a team’s release practice and overall DevOps success, but it doesn’t check all the boxes. CI/CD exists squarely in the engineering domain and is, by nature, build-centric; it fails to address the interdisciplinary, human and organizational aspects of the problem space.
Over-emphasis on CI/CD can actually have a net negative effect: the more teams try to cram into their build pipelines, the further they stray from sound DevOps principles. Overcomplicated and over-featured CI/CD is difficult to maintain and it becomes an esoteric domain unto itself. Even engineers feel daunted by it and ill-equipped to interact with it, and so it’s often exclusively attended to by one person or a small handful of specific and specialized team members. Rather than breaking down silos, this builds up a new one. Instead of affording more team members more agency, it creates more dependence. And that’s not to say that teams shouldn’t have dedicated release managers or release engineers — rather, overwrought CI/CD can end up tying those team members up and preventing them from doing more impactful things.
The transparency problem of release coordination is also not well served by CI/CD. CI/CD tools are much better at doing things than showing things, so as you add bells and whistles to your pipeline, you’re creating a bigger and bigger “black box” which does a lot of stuff and has a lot of side effects in a way that’s not visible to or auditable by the wider team. This can lead, somewhat paradoxically, to more human check-ins, more retracing of steps, more retro’ing when things go off the rails. Interdisciplinary teams need a real place to come together around the dev and release lifecycles, and CI/CD tools are not that place.
All of those Slack exchanges I highlighted above played out at teams with pretty solid CI/CD in place, in some cases even looked after by dedicated DevOps engineers. A build pipeline is good at moving builds from A to B, but it’s not going to help you herd cats. When it comes to CI/CD, stay lean and focus on core principles — creating, testing, and distributing builds — and avoid that tempting “CI/CD == DevOps” misrepresentation.
Take care with dedicated release teams and in-house tooling
For companies with resources that allow for it, adding headcount for release managers or TPMs, or even putting together entire dedicated release teams, can be an attractive approach. Throw people at the problem! It can certainly help cover some of the bases that CI/CD alone cannot, and we’ve had the privilege of watching and working with awesome, high-performing release teams. (We’ve also seen firsthand how expensive they can be!)
The humanpower approach is not without risks: at best, it’s often inefficient, and at worst, it can even do more harm than good. First, there’s the idea that adding humans to a task leads to diminishing returns — the infamous (if somewhat dated) “Mythical Man-Month” — and the fact that too much of a release team’s time and energy ends up spent on bookkeeping and mundane tasks. But also, creating yet another team within the org creates yet another silo situation, which is at cross purposes to the DevOps ideal of breaking down silos and bridging gaps between teams. Besides the general negative impact on visibility and ability to coordinate and collaborate effectively, there are also specific risks that come with distancing the product and release functions. Things can start deteriorating when the release side loses touch with the day-to-day needs, realities, and constraints of the product side. In order to successfully navigate the addition of dedicated (human) resources for the release function, teams need to already be exercising sound DevOps practices at a level that’s difficult to achieve for many.
Some teams — a very small minority — come to acknowledge the release problem and the inadequacy of existing band-aid solutions so much so that they assemble a dedicated team to build and maintain an in-house release management platform. The handful of successful examples of this come out of mature, high-performing and resource-rich tech orgs. Of course, success here isn’t guaranteed. And, whatever the outcome, it’s an undertaking that involves huge up-front cost and, crucially, an equally daunting amount of ongoing investment. Conversations with Sasha Friedenberg, the lead behind Etsy’s Ship release platform, revealed the difficulty Etsy had in continuing to allocate the necessary engineering and product resources to maintain the tool — let alone to expand it beyond its initial feature set. And this despite the fact that Ship was a successful and much-loved project, and despite the nurturing backdrop of Etsy’s strong “do it yourself” culture.
One of the attractions of the in-house approach — that it gives teams the ability to finely tailor a solution to their specific needs and existing processes — is another one of its risk factors. Namely, it can trap teams in an inward-looking exercise of doubling down on idiosyncrasies and reinforcing existing practices which may have been arrived at randomly or have diverged from industry best practice. Sasha identified this “conceptual tug of war” as a design dilemma the Ship team struggled with.
Both approaches to the release problem space — adding humans and building in-house tooling — represent a deliberate trade-off from a business perspective. Simply put, headcount and resources allocated there mean headcount and resources not allocated to real value-creating product work. Of course, platform teams and tooling allow product teams to do their jobs better, but what if you could get the same effect at less cost to the business?
What changes in the remote world
Pontificating on the challenges of our remote reality has gotten a bit trite, so I’ll keep this brief. In short: remote work amplifies some of the friction and problem areas that teams are already grappling with when it comes to release management. One recent, widely socialized study in Nature found that “firm-wide remote work caused the collaboration network of workers to become more static and siloed, with fewer bridges between disparate parts.” More silos and fewer bridges is a recipe diametrically opposed to the DevOps ideals I highlighted earlier. And so, in this new remote world, teams have to work even harder to tackle the releases problem and to set up a robust framework for DevOps success. There’s less room for error, and missteps become more costly. Teams need all the help they can get!
Towards mobile release, and DevOps, greatness
So, let’s say you agree that mobile releases are a pain point that costs your team time and money, and that keeps you from delivering value as quickly as you could be. Perhaps you now also see how your snazzy CI/CD pipeline isn’t covering all the bases or is even making the problem worse, and you’re wary of the complications that come with human-powered solutions or bespoke tooling. Where to, then?
We weren’t sure, either — and that’s why we decided to build Runway. It’s the solution we wished we had. And, although our past selves don’t get to put it to the test, it’s been extremely heartening to watch other awesome mobile teams do so, proving out our hypotheses and reinforcing our convictions in the process.
And so, finally, here are some of those hypotheses and convictions. These are the fundamentals that the team and I believe are necessary for mobile DevOps greatness, and which we’re striving to build into Runway’s DNA:
- Create a shared space where the entire interdisciplinary team, and the wider org, can come to participate in, weigh in on, and observe releases and the mobile practice more broadly. This should be a dedicated space — not just a Slack channel, with all its extra noise; not just a build pipeline, with its opaqueness and relative hostility to non-engineers; not just a massive spreadsheet, with its — well, you know how it goes with spreadsheets, the foremost code smell of the DevOps world!
- But also, meet every team member and stakeholder where they live in their normal day-to-day. Even as you create shared spaces, recognize that different players do still spend a majority of time in their own domains, and work to create bridges between those domains. This means facilitating and automating cross-pollination of data between the different tools each function is most comfortable with and in which they spend most of their time. A classic example of this exists at the intersection of product and engineering, with the labeling of work on the project management side based on code destined for, or actually shipped with, each release.
- Give agency to everyone involved, and not just by making participation possible, but by also making it approachable. Design processes and tools in such a way that team members across all disciplines can do — and are comfortable doing — what they need to do, without having to involve an intermediary. Consider again the classic game of telephone that plays out just to get release notes populated or app store assets updated. Not only should product/design/marketing be able to make updates directly, their means of doing so shouldn’t be daunting — in other words, don’t just set them loose in App Store Connect or Play Console, and don’t make them fire up Terminal and commit stuff under version control.
- Democratize and cultivate empathy for the release function. In organizations without a dedicated release team, share responsibility for the release process. Not only does this prevent there being a single point of failure or unnecessary bottleneck, it also reduces cognitive load and stress for whoever’s in charge of a given release. Releases are less taxing when there’s a supporting cast that’s equally capable and competent when it comes to releases. Even in orgs with dedicated release teams, look to cultivate empathy for the release function among product teams. Consider regular meetups — or even try a sort of “exchange program” — between release and product teams, to facilitate understanding of and empathy for the needs and constraints of the other.
These pillars are some of the biggest, most crucial building blocks for mobile DevOps success. For us, they’re also just the beginning. Our understanding of the problem space keeps growing, and along with it our vision for the solution. Although we’re no longer developing apps ourselves, working closely with so many different mobile teams now has only strengthened our resolve to get to the bottom of the mobile releases problem and to solve it in a holistic way. We’re excited to continue sharing what we learn, and what we build, and we hope you’ll join us for the ride.
