Case Study: LendingTree

"Given the fixed nature of our cost per unit, every percent gain in creative performance drops to the bottom line. So yes, RevJet [Innervate] is in the position to have a material impact on LendingTree’s financial performance."

- Josh Eldridge, Senior Director of Marketing

LendingTree Turbo Charges Creative Optimization on Innervate

An Industry Leader Always Looking to Improve

When LendingTree found Innervate, they were already one of the web’s most successful lead generation engines. By taking a very methodical approach to creative experimentation, LendingTree was able to deliver quality leads to their business partners while optimizing for revenue. Because of their business model, a multi-faceted optimization strategy was already critical. Initially they were only looking for an analytics system of record to provide them with a base for greater accountability, but found so much more.

Josh Eldridge, Senior Director of Marketing shared his thoughts on how LendingTree approaches experimentation: “For at least the last five years, we have put our ideas to the test by using tried and true scientific methods. Our testing methodology isolates variables and measures confidence intervals to ensure the statistical validity of a conclusion. This rigor has been applied to all media, creative and site testing, yielding clear winners and losers. This enabled the Display team to scale spend with ROI certainty. We built this organization upon thousands of experiments which were mostly conducted manually.” Before Innervate, LendingTree conducted creative experiments within their ad server without any optimization automation. Experimentation required lots of oversight to ensure validity.

GAINING SPEED & POWER

Manual testing, while effective, lacked the speed and power the LendingTree team demanded. Limited by the number of experiments they could run, the team needed a great deal of resources to run one experiment correctly. Each test required an initial brainstorming session with the creative team to decide what creatives should be tested, sign-off from legal and brand compliance, ad execution, production and coding, rigorous discipline to ensure valid comparisons, and follow-up work and sessions to learn and disseminate results.

Eldridge knew that manual methods had their limits. “We embarked on what became a two year journey to talk to and evaluate many DCO platforms. The goals were: 1. Greater team control over creative assets 2. Shorter testing cycles 3. De-averaging of segmentations and 4. Greater ROI. We choose RevJet [Innervate] as the platform which could help us realize them.”

A NEW CREATIVE OUTLOOK

The LendingTree team also knows that losers, or sub-optimal creatives that did not outperform the control, can provide valuable insights. However, these insights shouldn’t cost you any more than they have to. With Innervate, the system automatically takes down underperforming ads with statistical confidence. Creative teams don’t agonize over what might have been, and a greater percentage of media campaigns run with the most profitable creative.

Now that the LendingTree team is autonomous, its creativity has exploded. Suddenly these class-leading direct response advertisers are able to experiment at the volume they had envisioned.

Speaking about a recent completed experiment, LendingTree’s Michael Beery said, “Its promise was to better align our CTA (Calculate Payment) with the experience of the user. Our hypothesis was twofold: allowing users to see their monthly payments within the ad, rather than just the rates, would drive higher click-rates; and framing the calculation as their personalized calculation would invoke the endowment effect and increase conversion rates.” The experiment was very successful, driving 66% lift.

WHY IS BLUE WINNING?

The team would spend significant time and energy analyzing why one creative beat another, creating theories that weren’t always correct or actionable. When the LendingTree team first began working with Innervate, they started by backtesting older creative that was designed before engaging with Innervate. Eldridge recalls an example where a creative with a blue rate slider and call to action beat a green rate table after only two days.

The team would ask, “Why is blue winning? Is it because it’s a calm color? Is it that it stands out on certain sites more than others? That it’s associated with certain connotations?” This would lead to new hypotheses, which would then be controlled for and tested. Rather than debate why an ad creative wins, with Innervate, the team can simply move to the next stage of an experiment to test new variants. Everything that was previously holding them back from deeper testing is now automated for speed.

FINDING WINNERS FASTER

When starting to work on Innervate, the LendingTree team established two broad goals. The first was to increase the score of a creative against the control ad. The second was to grow the number of total tests they conduct. As compared to the 15 tests they were doing each week while in the first stages of interaction with Innervate, and even fewer before then, Eldridge said he “expects that number to double or triple through continued use of the operating system. Our goal is to find winners faster, and we’ll find losers faster as well. Overall, I look to RevJet [Innervate] for more test learnings sooner and wins by a greater degree.”

One of the areas they identified as important was to look at audiences at a much more granular level. “There are a lot of struggles that RevJet [Innervate] is going to help us solve, but I think the biggest one is granularity,” Eldridge said. “RevJet [Innervate] will allow us to target creative to browser, operating system, device type, weather patterns, day of week, publisher metrics, and then expanding to include registration data with gender and age and so on.”

During their first months using Innervate, Eldridge said LendingTree did not dive as deep into the granularity as they could have, but that they’d had “huge wins anyway. I can only imagine where we’re going to be in a year from now, after starting fresh with a whole new approach.

CONTINUALLY DIVING DEEPER

While the experience on Innervate has been great and profitable so far, Eldridge is eager to push it further. “I think RevJet [Innervate] is deep and really forward thinking. You’re thinking about the problems I encounter, so I know you’re helping me solve them.” Eldridge continued, “I feel like RevJet [Innervate] is really investing in the relationship and wants to make it a win. There’s a partnership with RevJet [Innervate] where we’ve talked about ideas in the past, and I can see us pursuing new creative adventures." 

“RevJet [Innervate] has been just great. I can’t say enough good things. I like how after every meeting we get a comprehensive recap. In addition, the RevJet [Innervate] team has come into the office multiple times. They’ve probably trained us three different times on the same thing and there’s no attitude. There’s just, ‘Absolutely, let’s get that done’.”

“We have increased visibility of our testing at the most granular levels for all to see. We have uncovered many new creative wins. We have empowered these team members with the ability to iterate creative autonomously. We are starting to de-average results down to deeper and deeper levels, with the ultimate goal of showing the right ad to the right user at the right time. This will enhance the user experience and provide maximum value to LendingTree.

As the relationship grows, and continual experimentation builds one experiment upon another, LendingTree continues to drive a material impact to LendingTree’s financial performance. Every percentage gained in creative performance drops to the bottom line, which is why LendingTree continues to be a premiere direct response marketing organization.