“Do What Makes Your Beer Taste Better”: Investing in Application Performance When it Counts

Jeff Bezos gave a famous keynote at Y Combinator’s Startup School in 2008. At the time, Amazon’s market capitalization was 22 billion dollars. Respectable, surely. But it was a tiny fraction of the trillion-dollar behemoth that Amazon would later become. In his presentation to startup hopefuls, Bezos challenged his audience to “do what makes your beer taste better.”

The background? Bezos was talking about European beer breweries in the 20th century. They, like many companies at the time, were adopting electricity as a new technology. The first breweries that used electricity had to build their own power generators, which were labor and capital-intensive. The breweries could use electricity to power new tools and machinery, but it came at a heavy cost.

Things changed with the next generation of breweries. Rather than build their own power solutions, they simply rented electricity from utility companies, which was cheaper and more efficient. As a result, these breweries outcompeted the first generation that built their own power generators.

Bezos argued that this lesson applies to businesses today, too. They should focus on what differentiates a company and not get distracted by what doesn’t. In short, “focus on what makes your beer taste better.” In the year 2023, we can pretty safely say that it seems to have worked out well enough for Amazon, anyways.

Working the Brew

We’re not a beer company here at FloQast, nor are we an Amazon-sized behemoth. But we certainly are a user-minded organization that has tried to follow Bezos’s advice and focus on “our beer.” We’ve tried to do this in a few different ways throughout our organization:

• Engineering & Product: We choose to “buy” instead of “build” whenever it makes sense, while investing carefully and intentionally when it doesn’t. For example, running our infrastructure on AWS (funnily enough, the modern-day cloud equivalent of the old power utilities) but investing heavily in engineering productivity and tooling.
• Success / Support: Rather than outsource a generic support service, focus on hiring fantastic people with deep accounting knowledge and backgrounds.
• Design: We started out by relying on a generic design library (Ye Old Bootstrap) and created our own when it made more sense

We haven’t always done it perfectly, of course, but a deep emphasis on user needs and company differentiation has worked out pretty well for us on the whole. Looking back at our past, it seems safe to say that the times when we’ve struggled the most were when we lost that focus and got lost in the weeds, thinking, “maybe I’ll just build a battery myself, not the whole plant.”

Investing in Performance When it Counts

A relentless focus on differentiation isn’t necessarily easy, even if it is important. The tradeoffs between “build” and “buy” decisions can be quite nuanced. Sticking exclusively to one side can be dangerous, especially for an engineering organization. Eventually, you’ll be mired down by maintaining custom-built solutions (“build everything”) or unable to meet nuanced customer needs (“buy everything”). You also have to be aware of the tradeoff spectrum — making it as fast might also make it difficult to deliver features and evolve your product.

We had some experience recently in dealing with this area in the form of a creeping performance degradation. We noticed that key latency metrics (API response time, UI time to interactive, etc.) had increased slowly over time to a point where some users were starting to notice. The beer was starting to taste a bit too much like Bud Light.

So, we responded. We chose to create urgency around improving performance rather than tackling the problem over several quarters. We recognize that often, we can better differentiate ourselves by focusing on feature completeness and correctness over optimizations. But in this case, slowdowns were noticeable enough to affect our core area of differentiation.

By creating urgency and focus, we were able to empower teams and allow them to focus more directly on the problem at hand. We also opened the problem up to the engineering organization more broadly – performance is everyone’s responsibility, after all. We focused on reducing latency of our systems in a few key areas:

• Database:
  • Re-visit queries that might need better indexing
  • Move to a read strategy that prefers secondary database nodes over primary ones
  • Launch an org-wide lunch-and-learn to improve long-term database skills
• API :
  • caching: caching certain network calls reduced network latency by a significant amount
  • compression: we noticed that certain API components weren’t automatically using GZIP compression. By enabling this, we reduced data sent over the wire by 10x or more
• UI:
  • Reduce the size of core FloQast libraries
  • Reduced client waterfall HTTP sequences that delayed interactivity
  • Reduce the overall size of FloQast client code
  • Investigate ways to further offload third-party scripts

The effort involved nearly every engineering team at FloQast, and our main efforts lasted for about a month. In the end, we were able to significantly reduce latency across FloQast apps, with more gains to be had in the future.

Staring Into the Distance

On reflection, we’re pretty happy with the balance we’ve struck. By investing in performance when it really counts, we can avoid many of the pitfalls of extremes of the tradeoff spectrum between optimization and feature completeness. There’s a huge pile of valuable features we never would have shipped if we had made performance our absolute number-one goal for years. There’s _also_ a significant number of users who wouldn’t be using our products if we had totally neglected to think about speed and performance. The key is to keep asking yourself, “does this make the beer taste better?” and focus on user delight and differentiation over everything else.

Mark Thomas

Mark is a Staff Software Engineer at FloQast. He enjoys good coffee, a solid gif, and beautiful systems.