Cloudflare Workers AB Testing

Overview

We currently use VWO as our A/B testing tool. We primarily use it for multivariate testing but for CMS we would need to test split URL testing.

Split URL Testing is a website optimization process to test different variations of your website hosted on multiple URLs. This test takes two or more variants of a web page and divides your website traffic between them. It helps identify the best-performing variation based on your conversion goals.

For A/B testing it’s standard practice to have a control group that you can then use to compare against one or more treatment groups.

It is worth mentioning that static sites are bit more complicated to A/B test as we will need to create a different version of our site.

What is a Cloudflare Worker?

Cloudflare workers operate very similarly to service workers in that they act as a proxy between a user’s request and a server’s response. The worker can read the request and programmatically respond using whatever logic you give it.

Can Cloudflare Workers facilitate split testing?

Yes. We can write a Cloudflare Worker that receives a user’s request cookie header (or generate a new one if value is null) and returns a site that corresponds to their test user group id.

Once each version of our site has been built and deployed (however many versions we choose to test) we create a Cloudflare Worker that can serve different versions of our site to different users based on their corresponding experiment group stored in the clients request cookie header.

example:

const NAME = 'myExampleWorkersABTest';

async function abTestingWithPassthrough(req) {
  const url = new URL(req.url);
  // Enable Passthrough to allow direct access to control and test routes.
  if (url.pathname.startsWith('/control') || url.pathname.startsWith('/test')) return fetch(req);
  // Determine which group this requester is in.
  const cookie = req.headers.get('cookie');
  if (cookie && cookie.includes(`${NAME}=control`)) {
    url.pathname = '/control' + url.pathname;
  } else if (cookie && cookie.includes(`${NAME}=test`)) {
    url.pathname = '/test' + url.pathname;
  } else {
    // If there is no cookie, this is a new client. Choose a group and set the cookie.
    const group = Math.random() < 0.5 ? 'test' : 'control'; // 50/50 split
    if (group === 'control') {
      url.pathname = '/control' + url.pathname;
    } else {
      url.pathname = '/test' + url.pathname;
    }
    // Reconstruct response to avoid immutability
    let res = await fetch(url);
    res = new Response(res.body, res);
    // Set cookie to enable persistent A/B sessions.
    res.headers.append('Set-Cookie', `${NAME}=${group}; path=/`);
    return res;
  }
  return fetch(url);
}

addEventListener('fetch', e => {
  e.respondWith(abTestingWithPassthrough(e.request));
});

Alternative example:

async function handleRequest(request) {
  const cookie = request.headers.get('cookie') || '';

  // Read the existing `xid` cookie value or randomly generate a new one.
  const xid = cookie.match(/(?:^|;) *xid=(\.\d+) *(?:;|$)/) ?
      RegExp.$1 : `${Math.random()}`.slice(1, 5);

  // Update the URL to include the experiment subdirectory, if applicable.
  const requestURL = new URL(request.url);
  const resourceURL = new URL([
    requestURL.origin,
    getExperimentPath(xid),
    requestURL.pathname,
    requestURL.search,
  ].join(''));

  // Fetch the response from either Cloudflare's cache or the origin server.
  const response = await fetch(resourceURL, request);

  // Add the `xid` cookie to the response so future requests from
  // this user will remain in the current experiment group.
  const clone = new Response(response.body, response);
  clone.headers.set('Set-Cookie', [
    'xid=' + xid,
    'Path=/',
    'Max-Age=31536000',
    'SameSite=Strict',
    'HttpOnly',
    'Secure',
  ].join('; '));
}

How can we analyse the data?

Cloudflare anlyitics. Alternatively, the performance data can be sent to Google Analytics.

example:

import {getFCP} from 'web-vitals';

getFCP((metric) => {
  gtag('event', 'FCP', {
    value: metric.value,
    experiment: self.__EXPERIMENT,
  });
});

Cloudflare Workers vs Vercel

Both solutions leverage edge technology, they are essentially the same product. Performance between the two varies from study to study however most say Cloudfare Workers have a faster build time. Other than there is no nobatlbe difference between the two.

Using Cloudflare Workers opens up the possibility of leveraging Cloudflare analytics which has been reported to track much more traffic than Google Analytics for the following reasons.

“Google Analytics and other web-based analytics programs use JavaScript on the web browser to track visitors. As a result, Google Analytics doesn’t record threats, bots, and automated crawlers because those requests typically do not trigger JavaScript. Also, these services do not track visitors who disable JavaScript on their browser or who leave a page before it fully loads.”[3]

For some organizations this is great feature but for others who just want to track how real life visitors are interacting with the site, the extra data provided by Cloudflare analytics isnt needed.

Resouces

1.A/B testing with same-URL direct access · Cloudflare Workers docs 2.Performant A/B Testing with Cloudflare Workers 3.About Cloudflare Analytics