Engineering at Scale - 5 key learnings


As an organisation grows, scaling its engineering capability presents a number of challenges: How to maintain velocity? What is the right balance between autonomy and alignment? How do I measure the success of my teams? What is the leanest way to structure them?

In October 2018, we invited Greg Stewart, Head of Engineering at Pizza Hut; Sam White, Tech Lead and Joe Paice; Tech Director at Red Badger to discuss.

Here’s what we learned:

1. Measure early

DevOps metrics are important. Greg described how their visualisation is equally important. Displaying live dashboards with information builds both a sense of transparency and responsibility for improving the metrics that are displayed.

We encountered a good example of this when one of our Badger teams - working at a large financial client - displayed some key DevOps KPIs on large TV screens around the team area. Not only did this spark discussion between team members, but stakeholders would also take a keen interest when walking by.

Product metrics are also key to making sure we are actually driving business value in what we build. An example Greg gave from Pizza Hut was measuring the time taken between a customer landing on the site, to finally pressing “Place my order”. Another such metric was the proportion of people ultimately placing an order after landing on the site (drop-off rate). Displaying these metrics so prominently and closely to the engineering team serves as a great reminder that the software we build has a direct impact on business value.

2. Automate everything

We know that by deploying code more frequently, we're able to get value to customers faster. Fully automating deployments enables a high frequency, but often presents challenges for many organisations. Technology and techniques are available to us which enable the continuous delivery of value to customers. Examples include feature flags, canary deployments, A/B testing, and close monitoring of systems to allow us to gradually release new features while minimising risk.

High levels of automation are key to enabling not just deployments, but other parts of the delivery lifecycle, too. Both Greg & Sam discussed how human gates should be replaced with automated checks wherever possible. Quality Assurance (QA) activities, approvals processes, release notes, and health checks can all be automated, too; resulting in reduced lead times, less risk of human error, and more autonomous teams.

The approaches discussed above can help make continuous deployment to production a reality, providing value to customers as quickly and robustly as possible.

3. Loosely coupled systems - evolutionary architecture

Greg talked us through how utilising the principle of evolutionary architecture allows software systems to be developed independently, and evolve independently.

Neither monoliths, nor microservices, are necessarily the right answer. A monolithic approach creates complex behemoths which take weeks to understand, and sharing too early through microservices can create unnecessary coupling between code and teams. Instead, teams should focus instead on building simple, understandable services, meeting just the minimum required for the current point in time. Eventually, the right boundaries will become clear.

Something that can help make this a reality is containers (eg. Docker), and container orchestration, (eg Kubernetes). They encourage you to fit the software you build into commonly shaped boxes that can be moved easily. If you need to build an application and the pieces are Docker containers, it’s easy to fit them together and ship them to customers.

Once we have these built, identifying them is the next challenge, so service catalogs are a good idea. These are like web catalogues within a company.

4. Internal open source

Openness encourages collaboration between teams, and helps to prevent silos forming within organisations. Collaborating and helping others across projects is much easier when internal open-source is adopted. When code teams write is visible to the entire organisation, this again drives a behaviour of transparency and being proud of what you’re creating. "If you build it, you run it."

A common misconception with inner open-source is that software doesn't require explicit ownership, rather that the organisation as a whole will maintain pieces of software. For example, Sam discussed a client project where a library was internally open-sourced, with the idea was that it would be owned and maintained by all of its consumers. The reality was that without explicit ownership, things gradually got messy and inconsistent: a variety of different code patterns and approaches developed, and new joiners were unsure of how to contribute.

Ultimately, a second version of this library was developed but with an explicit group of individuals in charge of its development and maintenance. This second attempt was much more successful, but maintained the principle of everyone being able to contribute.

Embracing internal open-source helps encourage good behaviours, instills a sense of pride, and a mantra of collaboration.

5. Be social

A healthy organisation is always talking.

Sam talked through how we helped introduce Slack to one of our larger clients and without a tool like this, they wouldn’t have been able to scale as successfully. Slack gave us an incredible way to share ideas and a platform to ask questions, even in an engineering organisation that spanned globally.

In addition, on the same client we introduced the use of a monorepo (150k commits, 2k CI builds p/d, 200 active contributors). This made it much easier to discover what others are working on, and made it easy to start conversations because you have the insight to see what others are working on and spark new conversations. It discourages any siloing because the code is so close together.

Being social is just one aspect of an organisation's culture. Joe discussed how everyone talks about culture, and how to improve it. It’s difficult to have these conversations about what is culture and how to measure it? How can it truly be improved? The culture map is a tool that can be use to articulate this (Dave Gray - who coins the term “Culture Map”). It captures three different aspects of a culture: outcomes, behaviour & enabler/blockers.

Thanks to everyone who attended. If you missed it you can watch the recording of the event below.

Want to join our next event? Sign up now to stay in the know.

Sign up to Badger News