Development slowness in big and legacy applications [and how to hurry it up]

Software companies come in many sizes. There are small startups, medium-sized companies, and huge enterprises. As you might expect, startups are usually lean and fast. Big companies that develop large applications move much more slowly. Those might be systems developed for many years or decades by hundreds of developers. I’m talking about products like the Amazon marketplace, AutoCAD, or any operating system. These products take a very long time to release new features or fix bugs, considering the amount of engineers they employ. Take Apple, for example. I’m sure thousands of developers are working on iOS, but the differences between each version are relatively small.

Some big projects are legacy applications because they use older technologies, include a lot of technical debt, and are hard to change. But projects that move slowly and have a legacy feel aren’t limited to enterprises, old systems, or even big development teams. Many modern applications suffer from the same symptoms, even if they are new and use the most modern technologies. They might have a convoluted business logic that makes it hard to change anything without breaking something else. Or the org might be full of policies and bureaucracy, which makes engineers spend more time waiting than writing code. Or you might have a flawed CI/CD process, so lots of time is wasted dealing with build problems and conflicts rather than developing new features. Many issues can turn a project into a legacy project. So why or how do projects become slow to develop? Is it the fate of any old product? And can we prevent or fix some of the problems?

Let’s discuss eight reasons for slow development processes in big and legacy applications.

1. Big Project Complexities

Even though it makes sense that big projects are more complex, let’s define this complexity. What exactly about such projects makes them complex and slows development?

Nobody knows the product, so it’s easy to break - In a huge product, there are so many features and subtleties that nobody knows everything. As a result, when you add a new feature or make a small change, you can easily break something. In the best case scenario, that will be a broken test. But sometimes it will result in a customer complaint or some problem discovered in production. The later the stage of finding a bug, the longer it takes to fix it. The worst-case scenario is when a customer finds it in production. When you have thousands of features and tens of millions of users, this happens a lot and takes up a lot of time. A big chunk of a big project’s work is spent on fixing bugs.
Convoluted architecture and abstractions - Over the years and decades, product managers come up with new features. Those are often added with minimal effort and without refactoring the system to make the abstractions reflect the new feature set. So these additions, hacks, and patches are added on top of each other until the codebase becomes a big ball of mud. Sure, there are techniques to minimize that. You can split the project into microservices, allocate more time for refactoring, and practice TDD to design a loosely coupled system. But reality and theory don’t always meet. When a developer faces the choice of completing a task in 1 day with a patch versus asking for two additional weeks to refactor, the result will usually be the first. Multiply that by 1000 over five years, and you’ll get a typical enterprise software project.
Custom builds and automations - Custom automations are introduced to project builds all the time. Those might be a linter, a code generation mechanism, file signing automation, etc. Theoretically, they will be part of the automatic build process, and you won’t have to think about them. In reality, such mechanisms break all the time. Engineers need to know how to fix them or work around them. New engineers spend a lot of time dealing with these build customizations, and when the project is big enough, you can stay “new” for several years.
Tooling overflow - Besides custom build processes, a big project is accompanied by a sea of internal tools. Those include, but are not limited to custom testing utilities, custom deployment tools, linters, transpilers, static analyzers, configuration controllers, CI/CD systems, proprietary source control systems, documentation engines, code review tools, security tools, monitoring tools, and license builders. As you might imagine, getting to know all these is a full time job. The onboarding of new engineers becomes longer with each custom tool. Besides, these tools are not exactly production ready. It frequently happens that you need to stop development to fix a bug in one of them or wait for someone else to fix it.
Conflicts - The more developers work on a project, the more conflicts you will have. Those conflicts might be an actual merge conflict or a bug introduced by another team. For each of those, you need to stop your work and fix the conflict, often needing cooperation from another team, which can come with another set of delays (see the next section).

There are branching strategies that allow you to “branch-off” for a while and work without interruptions. But those aren’t magic solutions. If you branch-off for a long time, the merge will be more painful. If you branch-off for a shorter time, you’ll have to deal with problems more often.
Long build times - As you might expect, the bigger the project, the longer the build. And when you have a humongous project, the build time is similarly huge. You can argue that build times can be optimized in a million different ways, and I would agree. You might even have a dedicated team to do just that in a big project. But optimizing build times is [usually] not the first priority, so relatively few resources are thrown at it. Besides, working on a moving target, a project that’s actively worked on, is hard, and the optimization rate is usually slower than the rate of new problems appearing.

So, how do we deal with those issues? Some of them seem unavoidable when an application gets big enough. A lot of those problems aren’t going to go away no matter what, but you can mitigate them somewhat. Here are a few tips from my own experience:

Treat inner-development processes as a first-class citizen. Place some of your senior engineers on dedicated teams dealing with build optimizations, internal tooling, CI/CD, and overall developer experience.
Treat engineering tooling like public tooling. For example, if you’re creating an internal API, do what you would do for an API for your paying customers. That means high up-time, great documentation, backward compatibility, etc.
Create an amazing onboarding process for new engineers and maintain great documentation.
Don’t neglect technical debt (within reason). That might mean big refactor efforts from time to time or an occasional feature freeze to deal with tech debt.
Allow hackathons and creativity weeks for engineers. Sure, a lot of the projects will focus on the product, but every company has passionate engineers who will take on improving build times or fixing internal tooling.

I feel like I covered just the tip of the iceberg of big projects' complexities. I’m sure each company has its own pain points, and if you have some interesting case, go ahead and write it in the comments.

But big projects have upsides that can also speed development up. When something has been worked on for decades, it usually includes many systems that make your life easier. That might be useful internal packages, a great testing environment, internal configuration tools, or tailor made IDE extensions. For example, you might get a new application up and running very quickly because you can access a cloud account with infinite compute, a way to quickly set up CI/CD, and a monitoring framework you can use automatically.

2. Cross Team Dependencies

A big project might have thousands of developers, DevOps engineers, project/program/product managers, etc. Often, the teams are spread across the world. You can’t possibly know everyone, or even close. When you need to change something that’s under the responsibility of a different team, you either need to ask them to change or do it yourself and get their approval. Since priorities don’t always align, this can take a while. With a different organizational hierarchy, you might have entirely different incentives and goals. This means you’ll need to do some politics to get things done quickly. Establish relationships, do some convincing, escalate when necessary, and occasionally beg for help.

If the other person lives in another country, it becomes harder. Cultural differences might become an issue. Engineers in India, for example, are very different than those in western countries. An Indian engineer might agree with something to appease a western colleague, even though they have objections, didn’t get the task done, or didn’t even understand it properly in the first place.

The time zone issue is another matter. Consider that if one team is working while the other team is sleeping, each message roundtrip will take a full day.

Some things can help with cross-team collaboration.

Limiting development to a single location is impractical for a big company. There are many good reasons to open shop in a different country, like a cheaper work force, diversity, taxes, or to help sell to the local market. But I’d suggest minimizing the number of locations in each product vertical. For example, if you’re working for Google, you might limit Android development to India and England, the search engine to the U.S. and Israel, and YouTube to Canada. This way, most collaboration happens locally or in as few places as possible.
Encouraging a culture of helpfulness is, well, helpful. If you incentivize developers to assist each other, they will do so. On the other hand, if you encourage a culture of competitiveness, they might feel less likely to help. They might even sabotage each other’s efforts. One idea to encourage helpfulness is to include it in a company’s mission statement. Or better yet, add helpfulness to every employee’s performance review goals. If money depends on it, you can be sure people will make an effort.

Some cool projects gamify team cooperation, like bonusly and Kudos . Bonusly, for example, allows employees to grant each other reward points. You can then give prizes or higher bonuses to employees with high reward points. In my opinion, it also opens the door to some corruption. Friends give points to each other without merit and quit pro quo agreements.

3. Security, Privacy, and Compliance

You know the old saying, “more money, more compliance concerns”. Well, it’s certainly true for tech companies. In startups, privacy, security, and compliance concerns might be an afterthought, but in an enterprise, they are a first-class citizen. Big companies go to great lengths to stay compliant, which has a price. Let’s talk about that price for each compliance category.

Security

How exactly do security concerns slow down development?

When creating new features, big companies will have policies around security. The point is to prove beyond any doubt that your new feature isn’t creating a vulnerability. That might require filling out forms with a hundred questions, drawing many UML charts, and facing a security review. Often with a lot of waiting in between each step.
New features are the biggest hurdle, but simple code changes can also add vulnerabilities. That means whenever you change anything near the attack surface area, you’ll need a security sign off from a certified security officer, which might be a simple ask from Joe down the hall or a long process full of bureaucracy.
Big companies love their security policies, and some of their favorite ones are restrictions on community projects. Think again if you thought you could use your favorite open source package. Is it secure? Is it compliant? What about its license? Time to fill out some forms and schedule committee reviews.

Privacy

If there’s one thing more important than security in today’s consciousness, it’s privacy. In 2023, we have had the pleasure of going through multiple public privacy related scandals in recent years (Meta, I’m looking at you). Nowadays, every developer has to learn about data taxonomy, GDPR rules, and your company’s policies, of which I’m sure there are plenty. This possibly means that pull requests need to go through privacy reviews. Or that you have limited access to customer data like logs and telemetry. Or that telemetry data is available but only from a single region in the world. It’s complicated. It’s certainly much easier to work in a startup with no customers.

Other compliance concerns

Compliance means adhering to standards, regulations, and internal policies in various aspects. Security and privacy are two of the main ones, but that also includes meeting standards like SOC and HIPAA, legal requirements in your operating location, commitments to your customers, and accessibility. Depending on your domain, it can mean different things, but it always means some amount of red tape you have to work with.

So what’s the solution?

Ignoring privacy and security concerns is not a great solution, right? Assuming those got to stay, here are some ideas:

Have engineers in your own team or group who can sign off on security and privacy reviews.
Make the compliance bureaucracy and processes as easy as possible. If there are forms to fill, make them short and easy. If a review needs to happen, have an easy way to schedule it.

4. Coding standards and code reviews

When a company grows beyond a certain number of programmers, you naturally start having major differences of opinion. Some developers prefer tabs, while others prefer spaces. Some like to prefix private members with m_, others with just _, while others don’t like prefixes. You can allow everyone to do what they want, but it will lead to conflicts and make nobody happy. Not to mention that it’s much easier to read code with a unified style.

Some companies have style guidelines, like this one from Google. To avoid any sort of confusion, Google provided rules for everything. You can take a look at this brief 21-page guidelines for Java , or this 82-page guidelines for C++ if you’re having trouble falling asleep. Every code review should follow these guidelines; that’s why Google has Certifitied Readability Reviewers , and you need every code review to get a sign-off from at least one such reviewer.

As you can imagine, big companies' code reviews take much longer. You must go through readability, security, and privacy reviews. In addition, I found that big companies have a different culture to code reviews than small companies. In a startup, you rush to create an MVP or meet some deadline. A big company isn’t rushing anywhere. Quality and compliance are prioritized because there are millions of existing customers, and every small mistake will cost a lot.

But things aren’t that bad. There are many ways in which you can hurry up code reviews. One such way is to invest in static code analysis tools that auto-fix your code to follow the company’s coding style. As an example, see ESLint for JavaScript and StyleCop for C#. Another idea is that if you have certified reviewers, have someone certified close by, preferably in your own team.

5. Meetings, meetings, meetings

Big teams need meetings for successful collaboration. That’s just reality. Meetings are necessary to decide project timelines, resource allocations, design reviews, status meetings, retrospectives, etc. Some collaboration can be done via Email or chat, but nothing beats a real-time meeting. An immediate-response dialog is crucial in complex discussions. It allows you to quickly clarify, ask more questions, and challenge ideas more easily. Meetings provide richer communication with many nuances via body language and tone of voice. They help build relationships, which is priceless for a healthy and productive work environment.

It seems there’s no getting around it. If you work for a big company, you’re going to be in a lot of meetings. More than half of the time, if you’re a manager or a senior individual contributor. But there are some ways to make the most of it. It helps to have a healthy meeting culture. That means starting on time, setting an agenda, timeboxing the meeting, and inviting only essential participants. You can send pre-meeting preparation documents to avoid spending the first 30 minutes explaining the problem. It helps to have someone act as a moderator. The moderator will allow everyone to speak their turn, staying on topic. Finally, follow up each meeting with a summary of the decisions made, or you risk needing another meeting.

6. Canary Trouble

In a startup, when you start with 0 customers, you can get much progress done with this convenient number. No customers means no complaints. It takes a while until you have a few customers and more time to gain dozens of customers, then hundreds, then thousands, and so on. Each jump in a magnitude of scale comes with new safeguards. In modern software shops, the trend is to reduce manual testing and instead implement automated tests and canary deployment. A canary deployment means each version update is deployed to a small set of customers first, then to a larger set, then to a larger set still, and if nothing goes wrong, it will eventually get to everyone.

Sounds great, and it has a lot of pros, but there’s a price for everything. The price, in this case, is development times. Often, you can’t continue with development until some change reaches all of production. The reason might be that you need telemetry data from production because of an API change or whatever. Besides that, there are occasional rollbacks. Imagine that your change from 3 weeks ago was reverted because of a bug that was discovered in a later stage. By this time, you might have written a bunch of new code that relied on that old code and even deployed it. Now, you need to make a smart fix that untangles this mess, deploy it, and wait three more weeks to ensure everything’s fine.

There are a few tricks to improve productivity in this reality. Most companies use feature flags, meaning code changes can be turned off remotely and immediately. You can set feature flags to “off” by default and then activate your feature at your own pace, not along with the canary deployment. But those are mitigation tools. Each change will still take a while to reach customers, and you’ll spend time waiting. The only way to avoid wasted time is to work on multiple things in parallel.

7. Data-driven decisions

Small companies tend to take more risks and come up with innovative products. Mature companies with a successful product don’t want to risk the golden goose. Every change is carefully considered, and decisions rely on customer feedback and telemetry data rather than the gut feeling of some product manager.

This kind of process is smart, but it’s also slow. Sorting out customer feedback or constructing a good A/B test takes time. Let’s consider running such a test. First, you need to plan it carefully in advance because if something goes wrong, you’ll have wasted a lot of time. Often, it’s necessary to add new telemetry events in code for the test and wait for them to be deployed. After the initial preparation, you need to run the test long enough to get statistically significant results and then analyze those results without falling into known pitfalls like bias or being affected by external factors. We’re in the ballpark of weeks and possibly even months. Sometimes, the A/B test doesn’t yield conclusive results, and you need to run another one. Other times, the stakeholders won’t sign off on any of the proposed changes, and all your hard work goes to waste.

8. Less time spent on “work”

The bigger the company, the less time you spend on actual work. I’m not talking about work-related meetings but events unrelated to your day-to-day job. That might be team events, group events, company events, town halls, all-hands, security training, privacy training, diversity and inclusion seminars, company culture days, hackathons, guest speakers, happy hours, performance reviews, 1-on-1s with your direct manager, 1-on-1s with your skip manager, and occasional 1-on-1s with the other five stakeholders you work with. Those might be important, but they add up to a lot of time of not developing software.

Final thoughts

Slow is usually bad in software, right? We want to be as productive as possible, but considering this list, I think big corporations get a lot of things right. Or at least right for them. Startups probably shouldn’t spend much time discussing coding styles, for example, because by the time they figure things out, they’ll run out of funding. In a big company, on the other hand, you have so many developers with different opinions that it makes sense to spend a ton of hours writing a guideline. The same goes for other items on the list, like canary deployments and feature gates. A startup with five customers doesn’t need five production environments or complicated feature-gate tooling. But applications with 10 million customers, where problems can cost billions, certainly do.

Some things are unpleasant for anyone. Nobody likes long builds. Most of us don’t like dealing with politics. Meetings can be nice up to a point, but sometimes whole days go by without getting anything done.

Although development can be slower in a legacy application or any big company, there are many upsides to big companies. For example, the compensation is better, there are more opportunities to find friends, great facilities, many people to learn from, and you get to work on some of the most important products in the world.

So how are things in your company? Please share in the comments! Cheers.