Digital Advertising

Should I Run Incrementality Tests to Validate a Retargeting Campaign?

Illustrating incrementality tests

Retargeting is one of the most powerful marketing tools in any brand’s arsenal, but building an effective retargeting campaign is as much an art as it is a science. To maximize the effect of their retargeting campaign, clients naturally want data that proves a retargeting provider is worth their salt. To do this, many companies rely on incrementality testing. 

This makes sense, as incrementality testing is considered the gold standard for good reason. However, they are not a catch-all solution. They aren’t cheap to run, and without a dedicated analytics team to interpret the results, companies can end up spending money on a test they can’t interpret. 

Let’s take a deep dive into incrementality testing, see how it works, and what you need to run one. 


  • Incrementality testing is just one way to qualify the performance of your retargeting providers.
  • In order to maximize the value of an incrementality test, you need a dedicated analytics team.
  • If you have a limited budget, a low average basket value, or a low number of transactions, an incrementality test may not be able to give you enough data to draw meaningful conclusions.
  • Incrementality testing is just one tool you can use to validate your multiple retargeting strategy.

Table of Contents

What is the purpose of an incrementality test?

The goal of an incrementality test is to determine what portion of conversions are a result of the retargeting campaign specifically, rather than from your general marketing efforts. The difference between the level of conversions between two groups during the test is called incremental lift and enables you to understand what impact your retargeting provider has on your general marketing results. 

How do incrementality tests work? 

There are a number of different ways to run an incrementality test, but they all share the same basic approach.

First, the retargeting provider and client define the target group. These users are then split into two groups:

  1. Test Group – These users are delivered ads using the retargeting provider’s methodology. 
  2. Control Group – These users are not delivered ads using standard retargeting activities, and are used as a comparison. 

Before conducting a test, you need to keep in mind the statistical difference between these groups. Specifically, this means that you need to eliminate the risk that the test results were obtained due to random chance. To do this, we use a P-value, which represents the probability that the results were obtained due to randomness, and variance, which tracks the randomness of the result itself. 

Incrementality test setup process

Over the course of the campaign, it is possible to determine the difference between the test and control group, you can see some of the results below:

Incrementality tests split

These results are useful because they give you a more accurate picture of your campaigns than a single method, such as attribution, is able to do in isolation. 

What do I need to consider before starting an incrementality test? 

On the surface, incrementality testing seems like a simple tool to validate your retargeting campaign spend. However, like anything worth doing, an incrementality test requires companies to invest significant time and resources in order to maximize their value. 

Before starting an incrementality test, you should consult this checklist to ensure that your business can conduct a test that will produce reliable results.  

1. Do you have an analytics team in place? 

While your retargeting provider can give you some insights about your incrementality test, it is important to be able to verify the results internally. To do this, you need an analytics team in place with the resources to analyze them. 

The team will examine the granular conversion data, and validate partner data against your internal data to confirm that it is correct. Additionally, this team will need to verify that the audience split used is statistically valid. These steps help to ensure that you’re working with valid data that can be used to derive correct insights 

2. Is your campaign large enough for an incrementality test? 

Data is good, but more data is always better. If your campaign is small, or you have a limited number of transactions, an incrementality test may not provide enough data for your analytics team to draw meaningful insights. 

In these cases, you would be better placed using another tool to validate your retargeting strategy.

3. Is your campaign stable? 

It can be tempting to run an incrementality test from day one, but doing so can distort the results. This can either be due to technical issues with the campaign itself or because the campaign has not scaled enough to produce statistically significant results. 

Generally, we would recommend that you run a normal retargeting campaign for a limited period of time, typically at least two to three months, in order to run your incrementality test with real and stable data. 

Assuming that you meet all these criteria, then you can begin looking at what kind of incrementality test you want to run. If not, you can skip straight to our incrementality test alternatives chapter here

Data in Fashion and Sports [REPORT]
The fashion industry is in a state of rapid change... but then that’s nothing new. Top fashion brands have always sprinted from one season to the next. The real challenge is not just keeping pace with change but staying ahead of it.

How does RTB House conduct incrementality tests?

RTB House has a few ways that they can run incrementality tests, however, we use two main methods

1. The intent-to-treat method

The first is the intent-to-treat method. With this approach, we split users into 2 groups: control and test. Over the course of the incrementality test, users in the test group will be served ads, while users in the control group will not. Data is then collected from both groups, and compared to each other. 

This method is the simplest way to run an incrementality test, and the only way we can run an incrementality test in a multiple retargeting campaign. However, it also means that many users won’t be served ads during the campaign, leading to potential opportunity loss. Additionally, this approach has lower accuracy due to potential noise coming from unreachable users in the test group, and it is not possible to provide precise audience identification. 

2. RTB House Ghost Ads

The second, and recommended, approach uses a bit of RTB House proprietary technology: Ghost Ads. As with an intent-to-treat test, we split users into a control and test group of around 50/50; however, this time the control group is exposed to the same adverts but with a limited frequency, typically of around 1 impression every 7 days. 

This approach includes a 7-day lookback window after each impression, which helps to reduce the dilution of the results. This approach is advantageous because it provides more accurate results than an intent-to-treat test, while still connecting with the control group, reducing potential opportunity loss. 

However, there is one limitation. RTB House Ghost Ads are proprietary technology, which means that they are incompatible with other providers’ retargeting solutions, and wouldn’t provide like-for-like data. This means that they are only applicable for checking the incrementality of RTB House in your overall targeting mix, and not to validate a multiple retargeting campaign. 

Incrementality testing methods

No matter which kind of incrementality test you chose to use, RTB House will use Deep Learning in order to improve the results of the campaign. Deep Learning is an advanced form of machine learning capable of understanding complicated unstructured data sets and providing meaningful insights from them. 

What are the alternatives to incrementality tests? 

There are not many alternatives that provide the same granularity of data as an incrementality test. In general, we discourage clients from relying on approaches like time series analysis or comparative market analysis, because they are highly susceptible to outside factors like seasonality. 

The only real alternative is to rely on unbiased attribution, for example, Google Analytics which shows the results for all advertising sources using the same methodology. While not as granular as an incrementality test, it will enable your team to make an apples-to-apples comparison. These analytics solutions also have the advantage of being a proven method, used for many years in online marketing. 

A test-and-learn mindset is key 

Whether you use an incrementality test, or one of the many alternatives, the key is in how you use this data. Adopting a test-and-learn mindset will ensure that you are always looking for ways to improve your marketing campaigns, regardless of what methodology you are using to do so. 

With all that said, we think it’s time we lead by example and showcased some of our own data-led retargeting success stories. RTB House takes a proactive approach to retargeting, with a heavy emphasis on using our proprietary Deep Learning technology in order to maximize the benefits of our retargeting campaigns. Our own in-depth internal testing has ensured that we’re able to deliver significant incremental improvements. 

One of the best examples of multiple retargeting success can be seen in this case study, detailing how we became Huuuge Games’ most successful retargeting provider after just three months, exceeding ROAS goals by 3-4 times. 

Another example is AllBeauty, where the RTB House team was able to help the brand make the most of a spike in online interest through an extensive retargeting campaign. Over the course of the campaign we helped boost the client’s year-on-year revenue from retargeting campaigns by 65%, and the number of conversions by 45%. 

You can find all of our retargeting success stories here. If you’d like to learn more about how RTB House can help your brand kickstart a retargeting campaign, or reap the benefits of a multiple retargeting campaign, talk to our team today.

Into the future of online ads?
Subscribe to receive email notifications about our latest blog posts
By subscribing you agree that RTB House S.A. (data controller) will process your personal data to send the newsletter and – if you agree by checking the box below – to contact you to discuss potential cooperation. You may revoke your consent by "Unsubscribing" or at For more information, see Privacy Policy.