Learnings from engineering in experimentation

I’ve spent the last 5 years building experiments.

The work isn’t too different from traditional frontend engineering. I still build interfaces, work with APIs, solve technical problems and ship code to production. What’s different is the way I approach the work.

Here are five things I have learned doing engineering in experimentation.

1. Understanding experimentation

It took me a while to understand what experimentation actually is.

Experimentation tests whether a change resolves a customer problem before fully committing to it. It begins with a problem statement and a hypothesis, followed by an experiment that measures the new experience or variation against the existing one.

A/B testing is the most common method. The idea is simple: make an assumption, test it, learn from the result and decide what happens next.

2. Simple doesn’t mean easy

Experiments often look simple.

Move something on a page, add a button or change how a product is presented. But a small change can require understanding a much larger system. Some questions I continually ask for each experiment:

• Where does the data come from?
• What happens when the page changes or updates?
• Is it a static page or SPA?
• What global components can be used?
• Are my styles scoped correctly?
• What additional analytics metrics need to implemented?

3. Learn how to hack safely

“Hack” isn’t usually a compliment in software engineering, but experimentation has taught me to see it differently. Sometimes you need to manipulate an existing page, work around a limitation or find a way to build something within a system that wasn’t designed for it.

Experiments are also temporary, so the goal isn’t always the perfect solution. It’s finding a safe and reliable way to test an idea without over-engineering something that might never exist beyond the experiment.

4. Work together, not in handovers

Experimentation works best as a loop between design, engineering and analytics, rather than one discipline handing work to the next.

Experimentation teams move quickly. For engineers, being part of identifying the customer problem and hypothesis enables us to quickly assess technical feasibility before design and build.

5. Temporary doesn’t mean careless

Experiment code is temporary by design, so it doesn’t always need the same level of abstraction or long-term maintainability as permanent application code. In my professional experience, the code is still running in production for millions of customers who use our website.

Because of this, accessibility, security, performance, and reliable measurement matter. We still need to write reliable and safe code for the lifespan of the experiment.

6. Build to learn

Engineering often treats shipping as the finish line, which has its pros and cons in large organisations. Experimentation shifts my mindset to one of continual learning and iteration rather than set and forget.