Yes, staging can be a pain — but here’s why you shouldn’t skip it

Programming’s big dirty secret is that everyone has, at some point in their career, broken something big.

Big like, wiping the full production database on their first day of work, killing the live application while multiple sales people are showing it to clients, or bringing down half the internet by passing on the wrong piece of data.

Oops.

There are a million reasons why things go wrong, and not all are completely avoidable. But having a good setup for testing new code can at least mitigate some of the worst errors.

Traditionally, deploying new code to a website or application follows three steps. First, an engineer writes code and tests it on their local machine — i.e. their own computer.

If that works, that code is deployed to a so-called staging environment, which is pretty close to the real deal. The staging environment allows new features or updates to be tested with real or anonymized data as a last step before finally pushing the code to production.

However, it’s slightly more complicated than that with staging environments being split up into multiple branches or forks that allow different features to be tested simultaneously. These forks are then merged back into the staging environment’s code and sent to production as a batch once they’re all proven.

Now this sounds like a pretty good setup right? Kind of like how new airplanes are first tested many times on the ground, then flown without passengers, and finally get the green light to transport people.

Nonetheless, lately, many engineers have stopped using staging environments entirely, arguing that it’s better, faster, and cheaper to skip that stage and just roll out new updates and features straight away. But could this be hurting you more than helping?

Let’s take a deeper look at why some engineers are ditching staging environments and why these arguments might not add up.

Facebook does it…

Many point to Facebook, which famously doesn’t use a staging environment but rather pushes changes to a tiny (but still huge) subset of users — e.g. 1% of users in Hungary — and, if that works, rolls out updates to a wider audience. But not every company has billions of users like Facebook.

Sure, depending on one’s tolerance for downtime and doing hotfixes, an argument can be made for simply deploying new releases out into the wild and seeing what happens. For most companies, there are no life and death dependencies on their product. And simply allowing a website or app to be unresponsive, for however long it takes to roll back changes and check what caused the disruption, can be an option.

Then again, this might not be the most user-friendly approach. Also, there might be agreements in place with customers that limit the threshold for downtime before sanctions are levied against the service provider (you).

Rather than removing accountability, it shifts it to the wider team.

It’s important to be sure that updates or changes that are meant to benefit the customer — and might even be a direct request from them — actually work as intended and don’t require your account managers to beg for forgiveness.

Staging environments are too costly

Depending on the size of your website or application, keeping extra servers running for staging can lead to a sad CFO paying double AWS bills.

Let me make an analogy to a restaurant. Yeah, if you’re keeping your kitchen fully stocked and staffed to test out new menu items for a restaurant, that’s going to eat into your bottom line. Or all of it.

But a new recipe can also be prepared and tested at home, on a normal stove, with smaller amounts of ingredients.

The same goes for staging. You don’t necessarily need the same hardware capacity to test out code for production. As long as you take into account the relative resources you’re using, it should deliver useful results. And, if you make sure staging mirrors production rather than your home setup, you should find out soon enough if you’re missing any ingredients.

IONOS’ Deploy Now feature, for example, takes care of that by spinning up and managing feature branches at the staging phase, without breaking the bank.

It worked on my machine

Staging your code isn’t just about making sure it works during production. Staging is also about getting other stakeholders on board.

Anti-staging advocates argue that having a staging environment dilutes responsibility because developers throw their code over the fence to the people managing it.

That might be true, but it also helps break down silos amongst different product teams.

A staging environment allows designers to test how things look, it allows quality assurance (QA) engineers to test the new code on different platforms, product managers to keep track of changes, and clients to get a feel for a new feature before it goes live.

It’s basically like running two parallel but identical restaurants.

Rather than removing accountability, it shifts it to the wider team, which in turn encourages more involvement and engagement with the product.

Staging leads to large queues

If you’re a larger organization with many teams working on different parts of an app, having a single staging environment can lead to longer approval processes — this not only slows things down, it also holds engineers back from starting on a new task or makes them have to go back and fix elements of a previous feature after they’ve started on a new project.

This can be quite tedious.

Ideally, what you’d want from staging is to automatically have multiple feature branches with corresponding staging environments that allow code to be tested immediately and pushed live after it passes all the checks.

Again, managing this can be a slog, but IONOS’ Deploy Now feature can pretty much do this for you. In principle, that should mean you can continuously deploy, without building up large queues — or losing productivity due to task switching.

Staging isn’t at parity with production

Another reason some people dislike staging is that it takes a lot of work to keep the staging environment in a state of carbon copy with production.

This doesn’t just go for all the dependencies, but also for the data that’s used during production but can’t be used during staging because it contains things like passwords, personal identities, or credit card data.

This means you need to maintain a dummy dataset and also have the schema of that data correspond with the information you have at the staging and production phases. Developers also need similar dummy or test data for local development (or the review) so this can also be used for staging.

It’s basically like running two parallel but identical restaurants.

Building this from scratch might seem like a mighty task. But, then again, if you start out with a proper staging environment and make sure it’s regularly updated to reflect production, it just takes a little bit of extra time to keep it up.

And having that habit in place might just help your company keep engineers productive and, more importantly, keep your customers happy.

So if you’re thinking about ditching staging environments, think again. Rather than a cumbersome process, keeping this product development phase will help keep the customer experience running smoothly and allow you to involve the wider company, without breaking the bank.