In the early stages of his career, Claudiu became obsessed with a thought that still shapes how he sees things today: how to make web projects that actually impact people's lives?
He established his first company at the age of 18 and his main service was creating websites for clients. Around that time, only a few companies had websites set up.
When he got to work on the website for the biggest company in their city, Claudiu got very excited. But when he presented the finished website to the client, the CEO was in such a hurry that the meeting became merely transactional: Claudiu was only asked about the cost, he got paid, and that was it.
"He didn't want to see the website, he didn't care about it, and that frustrated me. I would have been happier if he would see the website and be excited and not pay me, rather than pay me for it and not look at the website."
This frustration sparked a question: how would you know if a website is good or bad?
"Slowly, I got to realize that it's about the audience. How do you make something that has an impact in the audience's life? Because you make a website for somebody to visit. What's their goal and how do you get them to that goal?"
Because he wanted to pursue this question, Claudiu decided to drop out of college to focus on it. At the time, there was no solution addressing the topic and he took it upon himself to learn about it, eventually becoming a usability expert for another company.
As a usability expert, he got very deep into understanding how people use websites. But as he went deeper, he also got more frustrated because study participants would usually take a long time to learn how to use a website or an application:
"It would take months to find something valuable that you can make a decision based on. And that's when I discovered data."
Since then, Claudiu has been on the data-led road because he realized that data gets you faster to insight than anything else.
"There was nothing and nowhere to learn. There was no Google Analytics when I decided that data was the way to go. I learned how to write tracking codes, analyze data, and build reports. This allowed me to see the inside of everything and give me a different perspective"
But despite having more access to data and analytics today, Claudiu thinks that only a few are able to make data-led decisions:
"Everybody has data, everybody has access to visualization of the data, yet why don't they make more or better decisions based on the data? And that's when I realized that the secret is in how you think about data and the questions you ask from the data.
It's not you not being able to get answers from the data. Most people don't know what questions to ask from the data."
The problem of how to ask the right questions led to the creation of Claudiu's SaaS, InnerTrends, which is a prebuilt product analytics solution that works as a collection of existing questions that they've seen across the companies that they've worked with.
The role email plays in the product
"Depends on what the product is. There are products where email is the center. There are products out there where email is the product. Most of what people interact with is the email, so they build a product inside the email."
It's an uncommon case when everything happens inside an email (as a product). A lot of companies use email as a marketing tactic in cases like recovery to reduce churn or onboarding.
Unfortunately, companies have yet to develop a good strategy for recovery:
"Often, companies don't have a strong strategy around recovery and a lot of them send emails when it's already too late. The impact of those emails is very small: if you're going to send an email to somebody that already made a decision to stop using your product, it doesn't really matter what email you're going to send them. The chances for your email to change their mind are minimal."
This is why it's best to send those emails earlier when they haven't made the final decision:
"If you send emails when that decision has yet to be made, your email might have a very big impact. If you send an email to help them see a potential problem and how to avoid it, your email will have a huge impact."
InnerTrends has seen companies with great strategies recover up to 20% of their onboarding drop off with email. Meanwhile, most companies are only able to recover 2-3% of their users.
"That also makes a lot of sense because when you send an email to a hundred or a thousand people you already know, your open rate is around 20-40%. And then out of those that open the email, there is a huge dropoff in the number of people that actually interact with the email."
Which is why you shouldn't expect your email to change your business. Instead, think of email as a tool that can help you make your numbers better.
Avoiding bias when measuring the impact of an email
Even if you're looking at data, there's still a chance that you might interpret it with a bias. So how exactly do you remove the bias when measuring the impact of an email?
Claudiu shares his experience working with a product marketer trying to improve their new onboarding campaign that had 50 emails to cover all scenarios. They looked at two things to compare the impact of their emails:
- People who opened the emails and converted
- People who didn't open the emails but still converted
After analyzing the data, they found that regardless if people opened the email or not, people still converted, which means, the new onboarding campaign didn't really increase conversion:
"I showed her the report, which was pretty simple: you've sent around a thousand emails. Around 300 people opened them and you got 40 sales. 700 didn't open them and you got 70 sales. So that's around 10% for each group which means that no matter if people open or didn't open the emails, the conversion is the same."
This then led the marketer and her team to look at the granular data and they found that some campaigns had negative impact on users:
"She realized that the campaigns that were having a negative impact were very aggressive. They were trying to get people to upgrade even before they finished the onboarding process."
And from there, the team redesigned all their emails that eventually got them a higher and statistically significant result.
Doing experiments with email
When doing experiments with your emails, it's important to be familiar with statistics so you'll know if your results are statistically significant or not. You also have to observe a minimum time duration and sample size when doing it:
"It's important to have a minimum period of time, probably around two weeks. For the minimum sample size, I would say around the hundreds. Like 300 or 500 signups or people that received emails and then get statistically significant results."
It's also important to write the story you want to tell before doing the analysis:
"Let's say I go to the mall and I split the audience. Some people go through one door and other people go through the other door. And I bet you that I can find a statistical significance in the results that one door will lead people to eat more of something than the other door, which has no correlation between them.
That's why you need to write the story first. What are we analyzing? What's the context? What do we want to achieve, and do we have statistically significant results in achieving that?"
Writing the story first will get you a conclusion faster compared to looking at all the data and variables to see if there's any statistical significance anywhere.
Tracking the impact of email
Sending an email can cause two consequences: the immediate behavior impact and the indirect impact that comes later in the journey. How exactly do we track these?
For the immediate behavior impact, since it's short term, it's pretty straightforward: a subscriber opens the email, they do something, and you can track a certain order of events.
On the other hand, the long-term impact is a bit more complex. Claudiu says it's best to have a goal in mind when sending an email:
"It's very important that when you analyze the impact of an email, the first question would be about the impact of the email on what we intended. If I'm sending emails to get people to upgrade, how many upgraded after interacting with that email and after how much time? How many upgraded immediately in the next hour or so? How many upgraded on the same day or in the next three days, or in the next seven days?
If you analyze the distribution of how long it takes from the moment of sending that email to the moment of an upgrade, you will understand how you can attribute those upgrades to the email."
While it takes a lot of work upfront, looking at the immediate impact of the email is important for understanding its effectiveness:
"It's a little bit more work because you'll probably need to download analytics data to put all those events together and do that time measurement, but that's how you look at the impact of email on immediate terms."
It's very rare that you'll see a huge impact from emails immediately. However, they might have impacts down the road:
"This is the second part of the story: you send emails and they re-engage people into the product. By re-engaging them into the product, you actually increase the chances for them to upgrade to paid if that's the goal."
While they might not have a direct impact, the re-engagement increases the chances for users to achieve the goal you've set for them:
"That's why when we look at emails, we don't care about the purpose of the email. Like we don't care about why you sent that email and what you intended for this analysis.
What we look at is what are all the emails that people receive from the moment of sign-up to the moment of the upgrade. They might be marketing newsletters, product updates, everything, anything in that area, personalized or not personalized."
By looking at these emails, you can see that they have indirectly influenced the user's decision:
"They don't get people upgraded right away, but if you stop doing them, you'll decrease the engagement of your product and thus the chances of people to upgrade."
Final advice
Do start from the goal and move backward from there.
"Start from the place where you can have the biggest impact and the immediate impact. You can then slowly move to areas where you can have an indirect impact, and just increase engagement."
Don't limit yourself to only the email and don't start from the perspective of the email when measuring email ROI.
"Always start from the perspective of the user. You want to analyze the impact of email on users. The subjects are users, not emails. You're analyzing users and how they are impacted by the emails and how their behavior is changed by the emails."
Don't ever think that the welcome email will make a big difference.
"You can even stop sending it and nobody will feel it. You could start trying to be creative or funny about it. The only good purpose we've seen for the welcome email is when users refer back to it when they forget their username."
Thanks for listening! If you found the episode useful, please spread the word on Twitter mentioning @userlist, or leave us a review on iTunes.
