Living a Testing Culture

Great notes from Luke W from Max van der Heijden's lessons learnt from the A/B testing culture at Booking.com.

Bullet points that stuck out to me:

• Booking.com books 1.5 million room nights per day. That's a lot of opportunity to learn and optimize. More than 40% of all Booking.com sales happen on mobile.

• Booking.com A/B tests everything. If something cannot be A/B tested, Booking.com won't do it. There's more than 1,000 A/B tests running at any time.
• Teams are made for testing a hypothesis. They're assembled based on what resources they need to vet a hypothesis. They are autonomous, small, multi-disciplinary (designers, developers, product owners, copywriters, etc.) and have 100% access to as much data as possible.
• Booking.com's experiment tool allows everyone to see all current experiments to avoid overlaps and conflicts between testing.
• Failure is OK. Most of your learnings come from tests that don't work. 9/10 tests at Booking.com fail.
• Everyone needs to to live the testing culture. Ideas can come from anywhere and go all of them go through the same testing process. Everyone is part of this.
• Data only tells you what is happening. User research tells you why. These two efforts work together to define what experiments to run next and then test them.
• What worked before might not work now. Always challenge your assumptions.
• You can innovate through small steps.
• Delighting customers doesn't necessarily mean more profit. Short term gains may not align with long term value. You'll need to find a balance.

It's evident that in order to make a testing culture work it needs to be implemented at all levels (designers, developers, product owners, copywriters) and everyone needs to learn from each other in order to not make the same mistakes over and over. One of the reasons for their success is to allow mistakes (90%) but to learn from them and share insights within the entire organisation.

Not everything needs to be big steps. Refining the copy of a button can make as much different as redesigning the entire page.