App Store Optimization - Earnin

How app store A/B testing and its results led to a brand refresh.

When 2019 | Skills Testing Plan, A/B testing, Conversion Rate Study, Marketing materials

The Challenge

Earnin, like any other fast-growing startup in the valley, wanted exponential growth in its marketing numbers. There was talk was shifting narratives and changing the way they spoke to people about their offerings. The challenge was to make these changes while increasing install rates, tip rates, and retention rates.



My Role

Amidst these high level objectives, I decided to concentrate on a small piece of the pie, i.e. App Store Optimization. The reasons why I took that up were the following:

  • I was curious to see whether my brand vision for Earnin would be relatable to the users.
  • Google Play Store provides an extremely easy and fast platform for running A/B tests.
  • Numbers don’t lie.
  • Earnin’s Play Store and App Store looked different!



App Store Optimization

Before we could shift the narrative, I led running bi-weekly A/B tests on the Play Store to test the different styles in which we could speak and look to the consumers.



Before ASO, After ASO

Before
After



Testing Plan
I drafted a testing plan so all the stakeholders could follow it and see the progress and learnings from the tests. The plan included:

  • Concepts divided into buckets based on what I wanted to learn.
  • A diagram explaining testing across buckets.
  • A timeline.
  • Learning and next steps.

This plan was constantly evolving based on the testing results.



Test, Learn, Iterate
Since A/B tests don’t tell us much about the users’ behaviour, it was important to make small incremental changes one step at a time.


  • Round 1

    Round 1 Test: Small layout changes that I considered more pleasing to the eye.

    Option 1A
    Option 1B

    Round 1 Result: There was a small increase in install rates, big enough for stakeholders to approve of its push on play store and app store.

    Round 1 Learnings: While making the push from Round 1 testing to App Store, we realized that the Apple rejects screenshots that don’t have the CTA button clearly seen on the mobile screen. Thus, I decided to continue testing layouts where the CTA button was visible before pushing it on either stores.



  • Round 2

    Round 2 Test: Test two distinctive layout styles.

    Option 2A
    Option 2B

    Round 2 Result: There was no significant change in numbers due these options, which meant we could go with either and it wouldn’t matter. Thus, I decided to nix A and test my concepts against B.



  • Round 3

    Round 3 Test: Each product offering has is own screenshot VS all products listed on one screen.

    Option 3A
    Option 3B

    Round 3 Result: A outperformed B by a significant margin so I pushed it on Play Store as well as App Store.

    Round 3 Learnings: Pushing on app store requires a good amount of collaboration with product engineers in the company. I had to follow their timeline of releasing a build on app store, write down the new features they were releasing under the what’s new section, and test their build on my phone to ensure the phone screens I had included in the designs were exactly the same as what they had built in the app.



  • Round 4

    Round 4 Test: Test different conceptual images because, we were using these same images across all our communication and I wanted to see an initial reaction towards other images.

    Option 4A
    Option 4B
    Option 4C

    Round 4 Result: B and C didn’t do as well as A, which meant there was equity in the images we were using. They were recognizable and more relatable.



  • Round 5

    Round 5 Test: Test copywriting styles.

    Option 5A
    Option 5B
    Option 5C

    Round 5 Result:Even though I was favoring the two color headlines in C, B clearly outperformed both the other options. Its performance did better than anything that was ever achieved at Earnin through app store testing. The install rates were 80% better than the whole of last quarter thus, it was decided that the project has reached its peak result and pumping more into it wouldn’t be the best use.

    Round 5 Learnings:The results I achieved through testing, learning, and iterating was extremely rewarding. As designers, we are trained to trust our instincts. And while that is an integral part of being a designer, clubbing those instincts with numbers is an objective and irrefutable way of measuring success.





The effects of this project

While I was racing through this project, I was applying the learnings from it not only in app store designs, but also the other projects I was responsible for like static ads, instagram posts, and email designs.



Simultaneously, I was worked on creating a set of brand guidelines that could be used internally as well as by external partners. Check out the brand manual here.