Academia Leaderboards

Spearhead the creation of a brand new product offering that allows users to compare their work and the work of their peers in an interactive leaderboard experience.

Background

The "Mentions" product on Academia which provides users with detailed analytics about who is citing their work and mentioning their name in is Academia’s single largest product and accounts for about a third of all premium subscriptions. The size and robust longevity of this business reflects the fact that authors care deeply about the impact of their scholarship.

An authors work is often a key aspect of their personal identity as a scholar. Authors want to understand the value of their work and they regularly upgrade for Mentions because it meets some of that need. Academia identified an opportunity to meet this need at a deeper emotional level by creating a rankings leaderboard product

Length

Role

Deliverables

2+ Months (ongoing)

Lead Product Designer

A/B Test Plan, Marketing Materials, UX Flow, High Fidelity Designs, Interactive Prototype

Tools

Figma, Periscope

Vision

"A million leaderboards, each waiting for you to discover where you stand.” There can be a leaderboard for any sub-sub-sub-field. There can be a leaderboard for researchers who have published in Nature. There can be a leaderboard for authors who first published in 2016. There can be a leaderboard for all authors affiliated with UC Berkeley. Academia can generate and populate a leaderboard among any set of authors for which we can identify a common characteristic.

Risks

We knew that the risks of building this product would be significant. Because ranking is such an emotionally charged concept, we expected users to pick apart all of our decisions and motivations.

Our plan relied on the team's ability to start de-risking the harder parts of creating a rankings product, validating user interest and willingness to pay, identifying whether the concept has product market fit and understanding overall user needs. The team identified some ideas for lean testing, including a fake door test and we can also launch some intermediate products along the way.

Process

This project consisted of several major steps: Conceptualization and sizing, risk assessment, approach planning, research and analysis, lean test planning, lean testing design and MVP product design.

After we identified the vision, potential positive impact to the business and risks associated with the vision I started the process of putting together a competitive and comparative landscape analysis readout. I wanted to make sure the team had a detailed understanding of the best practices and trends associated with leaderboards as well as the failures and successes of our competitors attempts at entering this space. My full audit readout is below.

Testing

For our A/B testing phase I worked closely with the product manager for the team to identify some different product angles that we felt would have an impact on our user base, conceptualize a testing strategy and design the multiple test variants. We then tracked the 8 different variants against each other using periscope dashboards to determine which angle and messaging had the most impact and drove the most traffic. An example of one of our 8 A/B test variants is below.

The user was sent an email with 8 copy variations that we wanted to test such as "See how your Citation Rank compares to that of other authors on Academia." and "[full name] is moving up! See how your ranking compares to that of other authors on Academia.". Clicking the CTA in the email would then take them to a variant specific marketing page with different value props. After entering their credit card information they were taken to a rejection page that made clear that this was a test and their card was not charged. These tests helped us to identify the strongest copy and value propositions as well as the level of willingness to pay within our user base and the canibilization rate from other products.

Final Design

Results

This is an active project and results are constantly becoming available but so far results show a statistically significant lift of 16% traffic over our control group as well as an increase in premium upgrades coming from the navbar by users included in the test. It appears to be a success for both the business and user's goals as well as an excellent addition to our product suite and premium business. We are now running a phase 2 lean test where we try to zero in on a winning variant out of the highest performing buckets. We have also moved forward with designing and building an MVP of the product.