Conversion optimization” and “testing” became the hottest topics on the strategic road maps of online marketing managers. But still, a lot of companies suffer from easy to solve problems that probably cost a couple of million dollars.
In this talk, André Morys shows some unkown conversion barriers that are not visible on the website. He shows some practical examples and cases that show how effective optimization programs gain a lot more momentum and drive more optimization ROI than usual testing efforts do.
Full transcript below:
Yeah, Peep asked me to do a presentation about strategy. How boring is that? So I’m really sorry. I’m really sorry for talking about strategy. But I thought I could add some fancy animations like this. Maybe it makes it better.
In Germany there is a big e-commerce shop, it’s called Zalando. And when I’m talking about strategy, I want to talk a little bit about this. So I think in the U.S. you don’t know Zalando, right? In other countries you know what’s Zalando? Quite a little bit. So they’re kind of an aggressive start-up in e-commerce. We will talk about this.
But first, who am I? My name is Andre. I’m from Germany. I’ve written this book Conversion Optimierung, which is actually in German. So, sorry for you guys. Anybody speaks German? A little bit. I also do a conference in Germany it’s called Conversion Summit. And I have a blog that is called konversionsKRAFT, which sounds really German I’m sorry for that.
We have all these words. You know the saying with the butterflies. Butterfly is such a nice animal, it’s beautiful, that’s why it’s called butterfly. And in French it’s called papillon. What a nice word for a nice animal. And in German it’s called schmetterling, schmetterling. Yeah, so it’s konversionsKRAFT. Schmetterling.
Some marketing blah, blah, blah. Let’s get to the point. What I want to talk about it. I’m sorry that now that you named your conference Conversion Excel. My conference is called Conversion Summit. It’s a shitty name. I’m so sorry, but Conversion Rate Optimization.
This is what some ugly people do with sitting in the cellar, moving some buttons, AB testing them. No, it’s not about conversion rate optimization. And another thing a lot of times, so thanks for mentioning that Peep, it’s not about testing optimization, is not testing. And I will tell you later more about this. And I would say 90% of all test results are a lie. I’m sorry to say that, and I will explain why, and what you can do against it.
So first of all, I don’t know if you can realize what’s on this picture. If you do, please be quiet don’t tell your neighbors. Anybody knows this picture already? Okay. Some of you know it. So of you who doesn’t know it? Who can see clearly what there is on this picture? Please raise your hand. Okay. Three, four, five, six people. That’s a bad conversion rate. So I will change contrast a little bit.
Some people think… Actually this is Spain here, so this could be a satellite photo, or this could be a duck, or whatever, a bush. But no, it’s a cow. This is a cow. Who sees the cow right now? Quite more, that’s conversion optimization. Explain people what to do. Anybody else? Here’s the eye. Here’s the nose. Who sees the cow right now? More people. For everybody else, you’re normal. If you don’t see the cow you are still normal. Okay. Everything’s okay. Here’s the cow, you see? This is the cow. Who still doesn’t see the cow? Okay. Thank you.
Why do I show this picture? Because it’s all about patterns in our brain. So if something is not inside our brain, we can’t recognize this, or recognize it. Our brain needs a pattern to realize something. So it’s like seeing the other side of something. And today I want to talk about like the other side of conversion optimization.
To come back to the point, this is the shop, Zalando. And if you want to shop for shoes. That’s why Germans, they’re not funny. This is so creative. In German, boot, stiefel, or boat, a boat is kind of similar. They call this teaser in German boats parade. Nobody’s laughing because it’s not funny. And if you want to shop shoes look at that. I just want to buy some shoes. This is not Photoshopped, this is real. So, yeah. I’m sorry for that.
This is page 1 of 141. Sorry you can’t see it. So you can shop a lot of shoes there and that’s kind of a problem. If you have to do conversion rate optimization for this shop what would you do? Nobody knows. The funny thing is, if you look at those stats. I don’t know why the titles of my slides are not showing. Can you change something? No, that was an honest answer. Thank you.
These are the growth curves for top 10 e-commerce websites in Germany. How sad is this? Look at that. That’s Amazon in Germany. And this is the second biggest player called Otto. Very German. And this is all the rest. Number three, four, five, six. And this could continue like this and nothing will happen. How sad for these people. And this is not some bullshit venture, it’s a big company. 300, 500, 600 million Euros revenue each year, that small ones. There’s only one that stands out of this and this is Zalando. Look at this. This is their growth curve.
My question is, what is the difference between these companies? And it’s very easy. Zalando has a big team of around 80 optimizers testing the shit out of this website each day, and all the others don’t have, except Amazon.
So what I want to say is, conversion rate is maybe not the best metric we could use. We’re not doing this for fun. Its not that we say, “We have a testing tool, so what could we test today?” I don’t know. It’s about growth. We do this for improving our business each day. And Peep, already talked about process. No, you can leave. It’s okay. He talked about the process, and that’s the core of it. So we need a process? You know, there’s the head of this guy, its cutoff. I’m sorry for that.
So if it’s a process and we can invest in this process. And the process has an impact on the growth of our business, which means its money. And the process costs money also. We need a testing tool, we need people who use it for researching, and so on. So we have kind of a cost benefit ratio. So we should do a business plan for optimization. We should not test around. We should be sure what’s the optimal point between investment and impact. Anybody has a business plan? So you should start working on this. Why? This is the boring part. No, I’ll show you some Excel sheets. Sorry.
This is this is a simple e-commerce business model. So if you had like two million net sales per month, you have all this fixed numbers that stay the same. And if you compare two scenarios, one without any conversion improvement, one with 5% improvement per year, you can easily calculate what happens. If you would be able to have a global uplift of conversion rate of 5% per year, at the end you would have a little bit more money. Okay. That’s what this diagram says. But if look at profit margin, then you see, “Oh, that’s already a big amount of money.”
So the cost of not testing properly and not getting this global 5% uplift is already half a million. The problematic thing about opportunity cost… Are you familiar with this word? Yeah. So the problem is, it’s real cost, it’s there. You just don’t see it, it’s not in your accounting, it’s not in your annual report, or something like this. But it’s there. Because if you would have the money you could buy this new Lamborghini or whatever, but you have not the money, because you’re having real opportunity cost.
And why? Because you’re doing the three, or four, or five test a year but you’re not getting the 5%. And I have to tell you something else, 5% is not enough. If you want to have a growth rate like Zalando, you need to do more. So we need a challenge. But look at this boy. If you look like this boy with a challenge you already lost. You should look like this boy. Yeah.
So let’s compare a 5% improvement with a 25% improvement. So that’s where the fun begins. This is what happens in revenue, and this is what happens in profit margin. Okay? This is already 2.2 million Euros or dollars, it doesn’t matter in opportunity cost. Two-point-two million it’s profit not revenue, its money you could have. So when Pete is talking about discipline and real hard work, he’s right. That makes the difference. And I would say that makes the difference in successful organizations, that are optimizing properly, and organizations that test around a little bit.
The thing is for testing properly, you need resources, you need a lot of money, you need a lot of trust from the people that invest in this program. Okay. So if you do it, you could keep the 2.2 million, or you could re-invest this in your optimization and that’s exactly what Zalando is doing. So they’re dominating their market. All the traditional retailers in Germany they’re really afraid about them. They fear Zalando.
So first lesson. It’s not about conversion rate, it’s about growth. We have a bigger goal. And that’s what I would like to add to Peep’s slide. I would say, “You not only need a process, you need a goal.” A real goal. And management should believe that this goal is reachable. And they should give you the money you need to reach this goal. Set this goal properly.
You can tell somebody, “Yes, we can get 15%.” So if you get at the end of 25%, people will love you. But if you’re going for the 25% and you reach only 15, people will hate you. So be careful with this and measure the right things, measure everything is right, but you should measure more, measure your process. To have a process is the beginning, but measuring the efficiency of this process is the base for improving the process. And that’s why we are optimizers. Okay. We should even optimizer optimization process. So at the end you should ask yourself, “Where do the best hypotheses come from, from user research persona’s web analytics?” So you can control your investment, things like this.
And this nice guy, also very loved in Germany for his success from all the other retailers, not really. He already told his secret success formula. And he did this 10 years ago, 10 years ago. And there are all these conferences where classical traditional retailers are sitting and they’re really depressed. So if I show this picture, they start to cry nearly. And I say, “Come on guys, he told you his secret, you just have to do it, it’s so easy.”
And this is the real Zalando effect. Look at this, green is a traditional shoe retailer, making nearly 300 million a year. Somebody said, “Yeah, come on lets get the 300 million. Yeah, we did it.” Zalando blue did not exist. 2010 even better. But who? Zalando, who? Who are these guys? Oh, startup maybe? First they have to earn money, so until then they’re not a threat. And then look what happened, 2011, bam. I think that was the first year where the management of this traditional retailer was kind of shocked. And this is what happened 2012. Okay.
That’s the difference between having a plan, or having a process and a plan to having no plan at all. That’s the difference. And that’s why I think conversion rate optimization is a word that sounds much too small for what we really do.
So what’s the problem? Why is not everybody doing his 20, 30, 40% increase each year? Sounds easy? What’s the problem? I tell you, you could do, for getting 25% accumulated global conversion rate uplift, you could do 10 test each with 2.5%. Right? Does it accumulate like this? No, it doesn’t. Of course not.
So let’s get on the security side and let’s do 10 test with 5% each. If we add this numbers it’s 60%. Also it doesn’t work because of course not all the test are successful. So maybe we should do like 20 tests with each 5% uplift. So we’re really on the secure side of it. What’s the problem with this? I found it interesting that you had this questionnaire at the beginning and people said how many tests they did. So who’s actually doing more than 20 tests a year? Oh, Lucas, you don’t count in. I’m sorry. That’s good. That’s good. And what’s your success, right? Do you get the 50%? Who gets more than 50% with a significant uplift? There’s a problem. That’s it.
So I would say first, congratulations for everybody that does more than 20 tests. But to get them also significantly with a positive uplift, that’s the hard thing to do. Because doing all this test is really expensive and complex. And you know what organizations hate? Cost and complexity. That’s much more difficult than turning the Edwards wheel. So this is what most people do. So let’s buy 25% more traffic. That’s the problem.
So at this point, most of the people ask, “But why is he talking about 5%? Five percent in the last case.” I was so happy Peep that you were talking about a 5% uplift. Most case studies they talk about 45, 80, 250, 800% uplift. So 5% sounds kind of, yeah, really unsuccessful, right? The thing is, if you look at case studies. First, not everything is measured. Right? And second. A lot of people optimized for email subscriptions, download, things like this, maybe you also. If I’m talking about 5% uplift, I mean, a global revenue uplift. So that makes a different.
And the second thing a lot of people stop their test as soon as the tool said, I hate these tools, why do they do that? They say, “Woo-hoo, winning variation found. Now, you can stop your test and run to your boss.” This is like a seppuku. You know this, Japanese ritual self murdering. Why? It says AB test. This is your AB test on the left, on the right is reality. That’s the problem.
You go to your boss saying, “We have a 70% uplift.” And your boss will trust you. You will say, yeah, because you’re a good employee and he’s a good boss. He will say, “Well done, let’s roll out this stuff directly.” And then he will sit on his monitor pressing F5 to see in his business intelligence system. The lift he wants to see the 70%. He’s waiting for the 70%. And maybe after several days he will say, “I’m sorry I can’t feel the uplift.” That’s what happened. Then you’re dead. Okay.
So that’s not the way to get the big budget for optimizing. Why? You have your two variation or your control blue, your variation. And after one week it says, “Yeah, 10% uplift.” You’re done, significant. Stop it.
So what would happen if you run any longer? Let’s see, it goes like this, and it says, “Only 3% upflift.” That’s not significant anymore. So you should maybe even run it longer. Sorry, the screen is finished there. And that’s what I want to say, “You need to run your experiments much longer, than you’re testing tool tells you.” This is really important. That’s why I say, “90% of test results are all lie.” Look at that.
That’s a test with VWO old version, that a client wanted to do. And it had 106% uplift after one hour. And guess what happened? Client called us, “Please stop the test.” And why? “We need to report good numbers to the management.” Said, “No, that’s not what it’s about. Let’s wait for it.” Look at the numbers, it’s 12 versus 25 conversions. From a mathematical point of view this is a significant difference. But this does not mean that your data is valid at this point.
So look what happens after one day it’s only 9%. The client was really sad, really sad. And I said, “Don’t be sad.” This is a 360 million euro business and 9.6% is still 36 million Euros each year. And guess what, I still think it’s not valid because it’s only one day.
So after two days the client said, “I told you it’s around 10%.” So now the test is calibrating, something like this. People have their fantasies. I would say this is like cognitive dissonance because they made the variations or they believe it better, so the data has to show that it’s better. So let’s stop the test until the opposite comes up.
This is what happened on day three, only 3% and we say, “No, keep it going.” Still 10 million Euros, a lot of money. And after seven days it recovered and VWO said again, it’s significant, 8.68% uplift. Who would stop the test right now? Oh, sorry. My questioning was maybe too obvious. Yeah, that’s not the end. After 14 days 2.4% of the four weeks nearly 1%. So this is what happens when you stop your test too early, and why? I would say from a market research point of view your sample size is not representative. You don’t have enough weekend traffic. People have more money on the beginning on the month than on the end. So they change their behavior, things like this. So you need more time.
And I compare this to this story that Bryan Eisenberg told once. That’s how it sounds to me when people are doing AB testing. It’s like the Underpants Gnomes. Do you remember them from South Park? Anybody? Yeah. They had this plan, the secret plan. That is collect underpants, phase one. Phase two and phase three profit. Okay.
So for most people AB testing is like this, “Yeah, let’s AB test something.” We got some profit. No. Okay. So I would say 70% don’t know that there is a phase two. How you turn AB test results, so that you get profit. And 20% have no answer, although they don’t know that there’s a question mark. Only 10% have an idea that there is a problem. And that what you measure on AB test is not the thing you have on your bank account, right?
So this is something you could expect when you do e-commerce testing. These are average uplift from e-commerce optimization. But because we are in an agency we exactly suffer from the pressure that Peep told in his talk before. Of course, people have high expectations. So we invest even more in building hypothesis and things like this.
Second learning I would say, it’s about effectiveness. Grove is not only about efficiency, it’s about effectiveness, which means do the right things. So if you really want to move something, you need great hypothesis, you need to have good test results. So, do the right thing. Invest more in research and find out what really happens on your website.
I know too many people that are testing around. And you know this is our template, we could move the box from this to that position. How should that affect user behavior? It won’t. So don’t do it too fast with stupid ideas quickly test, too early to stop the test. That only will burn your ideas. Okay.
So part two. Are you serious? Damn it, sorry, out of time. So where do good optimization ideas come from? This is a website we analyzed like 10 years ago, a troubled website. And I said, “Yeah, our conversion rate is nearly 0%.” So we have the feeling that this thing here is much too complex, can you do usability research. And yes, we did it. But like 20 people and 20 people could do it, like they had a conversion rate of 100%.
So we change the setting and made a difference. And these are three travel websites. Where would you buy? This is a completely different question. And then the website of our client had 0% because everybody would like to buy somewhere else and we couldn’t find out why. What are the triggers that make a decision in the users mind. So we asked, “Why would you buy here or there?” And this delivered a really good hypothesis for us.
So this is the usual euro marketing slide that should mean, people decide unconsciousness which of these websites is the best. They have a feeling and sometimes they have a story in their mind. And this is why I would like to say, to do conversion optimization is like a flat tire. If it’s flat you don’t need to measure the pressure to find the reason why it’s flat. You have to find out why it is flat, and this means you have to do qualitative analysis. And I would say, too few conversion optimizers do really good qualitative research. If you have both, that’s like the sweet spot. If you really understand people and you can prove it with data.
Most realities of people is like that. We send traffic to a website. We measure what comes out of it with analytics. So this is like a feedback loop, right? And outline is happy we looking at the website, credit card ready to buy. That’s what we believe in our naive fantasy.
That’s not how it is the reality is like this, you see. People watch your website, ask themselves should I click here, then they decide if they click or not, move their hand, click. And that’s what we measure. And these happens 100,000 times each day, or week, or year, whatever. Way too complex to understand. If we’re doing AB tests, we are not measuring changes on the website. We are measuring that human behavior changed for a group of people. And we forget this because we all the time sitting in front of a website. So we know 99% about this and only 1% about the people and why they do this.
So this is a test of Facebook. I’m sorry. You don’t see it. So Facebook deactivation page. And if you ask people how to optimize this page, most people say, “Yeah, we could remove the confirm button.” Or “Put it to the left.” Or “Make a little text link.” Or “No, we could hide the complete website.” Whatever. It doesn’t work.
What Facebook did is change the motivation of the people. They asked themselves, “Why are they at Facebook.” “How can we keep them there.” So we can work with their motivation being in contact with their friends and that’s what they did. They said, “If you leave Facebook your friends can’t be in contact with you. Tim will miss you.” You see? But they don’t know that Tim is the reason why I want to leave. Or looks like a stalker a little bit. But you know what I mean. They look on the motivation, not on the template. That’s the difference, don’t think in templates or design. Think in user’s motivation and how to change.
So this is what this model should show. Don’t work on the lower level. Our people are able to buy here. It’s not all about accessibility, things like this. Maybe if you’re working in mobile website. But work on this medium level on the motivation of people because this has our eye. And if you’re really good to it like Amazon. Amazon is a habit, that’s why it’s that successful. Or Apple, it’s a brand that’s why it’s successful. So this is our eye killer.
We developed the framework with all these factors that change motivation of people. So this is like a questionnaire that asks what would happen in each step you could use it. So for example, one question is, is that the right site for me? Every user asks itself, “Is that the right site for me?” “Is that for people like me”
And I’ll show you some Scandinavian examples. And all these Scandinavian guys here, please be quiet. What do you think all the people, what’s that website all about? What is it? Amusement Park. Yeah. Somebody said bank. Yeah, that’s good. Why do you think it’s a bank? Risk. That’s also my first association with a bank. If I’m searching for a new bank, I’ll say, “Bank, risk, great bank. Yeah, I take this.” No. Oh, look at that, that’s also a bank. Yeah. I feel like a director of it today. Go to this bank. This one I really like. The relationship. Who wants to have a relationship with this bank? Nobody wants to have a relationship at this bank. This guy wants a relationship with this girl. Okay. I don’t know who is the bank and who is the…
Anyway, so much hidden messages in website design. Okay. Oh, sorry. This is in German. Or more questions like, we know all this psychological things, we could use them, like the reason why. This is the second best German e-commerce website, why should I buy there? Anybody thinks it’s cheap. Anybody thinks it has a high customer service, maybe quality leader. Yeah, you’re right, nothing of that. You don’t know anything about them. They are nothing. They’re not cheap. They’re not good. They’re nothing. So the thing is they have a 24/7 hotline, you could call them on Christmas Eve. Ask, “Where’s my package?” They will answer your question. But where is it? It was not on top of the website, it’s below the SEO text, there is the hotline. And of course, this is crappy, nobody thinks that this is a good shop for me.
And this is why I like Zappos so much. This is Zappos and look how many square pixels they are using for value propositions. And I’m sorry you can’t see 20% of them. It’s all full with customer service and full with value propositions. Even in the footer, I like that they have their corporate values in the footer, and you click on it and see it. I think that’s a good job.
But most websites, Dell could be German. Dell could be a German website. Look at this. I’m sorry this is German. This is the German Dell website saying, “You made some mistakes in this form.” Amount of mistakes, 10. What the fuck. Yeah. And this is every single dollar has to go through, this form. And when I make a mistake it has all this red text saying, “Eh, eh, eh.” It says, “The selected payment method can prolong the delivery time of your smart selection system.” Maybe it’s nothing for you. Maybe you’re too dumb to buy here. Yeah. Let’s say this is why forms, it’s not about only its ability. It’s about corporate behavior, this is behavior. Imagine this is a real scene in real life. No, no, no. No, you can’t buy it.
So that’s why I love Booking.com. You will hear more about them, I think in this conference. Not only because they play this scarcity game very well. Tell me, “Andre, you’re a booking genius, honestly. You save so much money. You’re a booking genius, you get another 10% off.” I am a genius. Yeah, yeah, thank you. And they say, “Congratulations.” They give me good news. That’s it. They said, “You found the fucking cheapest hotel.” Yeah, that’s what I wanted to do. Okay. If I spell my name, they say, “Yeah, you can spell your name correctly.” How amazing is that? How amazing is that? So I really want to buy this.
So would say conversion optimization, it’s about growth, its value propositions. But at the end if you want to motivate people, it’s about getting a good feeling in their mind. Okay. That’s what it is all about.
So close to the end. Close to the end. Yeah. This is a process, this is our process. And of course, it means I would add define a goal to the process. It is important. Measure your stuff, analyze it. If you do this you will find a lot of things, you will learn a lot of things. So build a backlog like, Petrel and Excel. That’s how our backlog looks like. And if you want to rate the impact of your hypothesis, only test the best things. Ask yourself, “That what I changed.” Is it bold enough that people will see or realize the difference. Like you can deconstruct the factors of impact with this question. So you get a better rating.
You can ask yourself, “Does the variation have enough contrast to change the behavior?” And next time somebody says, “Put the button here.” And you say, “Yeah, but does it really affect behavior?” No. Then don’t test it. Do you use psychological principles? All these hundreds of biases and psychological principles, there’s not anything else that AB test psychologists already did for you that you can use. Its principles you can use, that gives an extra bonus.
And of course that means is it on the primary path of conversion. Is there enough traffic on the website that I’m testing? So prioritize your stuff. And last but not least, measure it correctly. Sorry, I’m fast, right? Yeah, yeah, yeah.
This is a website that sells shoes. Its 180 Euro shoe, it’s now reduced to 109. But at the end. Last time I told this there was somebody from Nike in the audience. I was really sorry because I always say, “It’s a piece of plastic. It smells like plastic, it feels like plastic, it does not feel like a 180 Euro shoe to be honest.” So what we did is we improve this design first, bigger picture things like this, things what optimizers do. And we realized, this is reduced. Why should I buy this shoe, right here? So if you’re having a good hypothesis, it should directly be connected like the users behavior.
First reason, it’s reduced, that it’s cheap. So we could enhance this by using color codes like this wiper of a price signal showing how cheap it is. This could be one reason why I would buy this shoe here. But another reason is, it looks like a piece of plastic, but it has play fast and play strong inserts. How amazing is that? Ever had a shoe that has that? It also has Flywire technology. I only thought airplanes had that. You can have a shoe with the Flywire technology. That’s amazing.
So this focuses on quality and value positions. We said people should buy this shoe because it has great features. To be honest what we wanted to do is we wanted to attract different people. This is why we showed these features. So we have the control, a basically optimized template, a version that focuses on discount, and the version that focuses on quality value proposition. And guess what happens? That’s why I say, most people only optimize the template.
Thirty percent uplift, 13 for the template optimization, 14 for the discount. So now you learn people don’t buy it because it’s cheap, because of course it’s still not cheap. And 43% uplift when you focus on value proposition. That’s why I say Flywire technology, imagine.
So our problem is, that’s why everybody’s measuring. Guess what happens? You see website, great photo, Flywire technology, all that crazy shit. It was delivered to your home, you open the package, and you look at the shoe, and what do you think? “Oh, it looks like a piece of plastic.” Right?
So this is why we also measure like the return rate. It’s easy just put the information about test and variation into the order. Order goes into your business intelligence system, data warehouse, whatever. And then you do a cohort analysis. And then you see control again 100%, template optimization 243% more returns. Ouch. That’s expensive if you’re an e-commerce, you know that. Discount not that much, but this quality stuff nearly three times higher return rate.
So that’s the hard lesson to learn. At the end, this had a happy end because a return is like [7 Euros, 50] in cost for this e-commerce guy. So when you look at the bottom line you see control again is 100%. Template optimization does not do anything significantly. Discount does not work at all because a lot of people didn’t pay, or want back the money, or things like this. But for the value proposition, there was still enough money left.
So if you’re working on all that process, with all that investment, and you’re doing it right, your prioritizing everything, then you should measure the right stuff. Because higher conversion rate does not automatically mean that you earn more money. The closer your conversion goal is to your business goal, the better are your results.
So, summary. It’s about growth, it’s about effectiveness, you need to prioritize your hypothesis, and you should work on motivating users. And at the end measure the right stuff, so you’re not wrong. That’s it from me as the great David Ogilvy said, “If it doesn’t sell, it isn’t creative.” So, thank you very much.