All tags:

A way to implement conditional donut caching or server side content targeting using varnish

Created:2017-10-29

Even problems this article is trying to solve is related to Adobe AEM Personalization, but the technique it uses is AEM agnostic.

About Varnish

Varnish has been widely used nowadays as a caching HTTP reverse proxy. I use Varnish to act as second level Load Balancer behind F5, traffic controller during release, cookie manager, redirection manager and first layer cache server.

About Adobe Content Targeting

AEM Personalization uses client side javascript to change page parts according to configurations defined by authors. A classic example for this is, to maxmize your campaign's outcome, you probably want your campaign page's staging component show different attraction photos according to visitor's age, geo-location and devices they are using. When you need a proper report to record and compare diffferent content's result, you need to use Adobe Target as Targeting Engine -- you need to pay money for using this service for sure. Otherwise when you just need to change the content and don't need report (for example, you just want to show different phone numbers based on visitors' geo-location), you will choose AEM as Targeting Engine to save money and energy.

Like all other client side content targeting mechenisms, AEM as Targeting Engine has all the issues it should:

  • page rendering either gets blocked or it causes flashes when changing targeted content.
  • targeted component is shown asynchronously so javascript inside the component also executes asynchronously which might cause bugs that are hard to locate and fix.

AEM as Targeting Engine also has issues it shouldn't have:

  • complex and distributed configurations in author.
  • oversized campaign size and client side javascript file as time goes by.
  • uses <noscript> tag to hide default experience which causes problem when default experience also outputs <noscript> tag
  • hard coded timeout (2 seconds as I remember)
  • shows duplicated experiences when network is unstable.

I spent some time trying to implement content targeting in server side to solve all the problems listed above. The goal is, the targeted component gets rendered on page just like other non-targeted components.

Use Varnish to implement server side content targeting

Varnish supports this Edge Side Includes to allow us to implement donut caching easily. But to implement the goal mentioned above, this donut caching needs to support showing different esi:include according to conditions predefined. I'd like to call this common requirement as conditional donut caching. A simple example would be, for given html page below:

<html>
    <body>
        <esi:include src="/c1.html"/>
        <esi:include src="/c2.html"/>
    </body>
</html>

with c1.html:

<div>GB</div>

and c2.html:

<div>OTHERS</div>

The page shows c1.html content for visitors from gb, and shows c2.html content for others.

To achieve a common solution, I assume this condition is known when parent page is generated, so the conditions defined can bewritten into esi:include urls. Then the parent page would look like:

<html>
    <body>
        <esi:include src="/c1.html?positive-match&showMeWhenVisitorIsFrom=gb"/>
        <esi:include src="/c2.html?negative-match&showMeWhenVisitorIsFrom=gb"/>
    </body>
</html>

In varnish code, it uses "positive-match" and "negative-match" to know whether current request is doing normal page request, or doing requests for esi:include. When it knows it's processing esi:include requests, it also req_top to access parent page's client context information -- req_top.http.X-Visitor-Geo-Location in this case. It then finishes the request flow so real c1/c2 html page will be returned, or returns empty 200 page, according to condition match result and negative/positive proces it is in:

sub vcl_deliver {
    if (req.http.X-Recv-Flow == "conditional-esi") {
        set req.http.X-Esi-Condition = regsub(req.http.X-Original-Query, ".*&showMeWhenVisitorIsFrom=([^&]+).*", "\1");
        set req.http.X-Esi-Context = req_top.http.X-Visitor-Geo-Location;

        if(req.http.X-Original-Query ~ "^positive-match&"){
            if(req.http.X-Esi-Condition != req.http.X-Esi-Context){
                return (synth(200, "positive: context doesn't match condition, ignoring...."));
            }
        }
        else{
            if(req.http.X-Esi-Condition == req.http.X-Esi-Context){
                return (synth(200, "negative: context matches condition, ignoring...."));
            }
        }
    }
}

The code above makes request to c1.html show empty content when visitor is not from GB. It also makes request to c2.html show empty content when visitor is from GB. Both c1.html and c2.html are cached based on other customized caching mechanism and they are used only when conditions match. Now the simplest version of conditional donut caching is implemented. It makes server side content targeting become possible.

Conclusion

By expand example shown above, readers can easily append more segmentation factors to c1.html and c2.html. Apart from c1 and c2, you can also add more donut pages to be shown based on positive/negative conditions of your own. When this all finishes, server side content targeting of your own needs is completed.

Some known restrictions should be highlighted here though:

  • Parent page shows different content according to all segmentations factors, in theory, this means parent page's server side hash generation and client side caching mechanism should take this into consideration. Otherwise the same visitor might see old cached page when he/she visits the same page with different conditions in url.
  • Regular expression cannot be used when trying to judge whether the context matches the condition since both of them are dynamic. Hopefully one day Varnish starts to support dynamic expression in regular expression. Email me when you find it's finally supported please.
This article's tags: