Blog

User behavior tracking using ETags

Tracking in e-commerce apps is essential to increment ROI in ads campaigns and also to have information to increment sales numbers. At Ensolvers, for one of our customers we had to implement a very complex tracking system using a plethora of methods like cookies, pixels, fingerprinting and ETags.

These approaches vary slightly between each other, some of the most common are

  • Cookies being created when visiting a certain page and then used to get that information and know where the user had been
  • Transparent pixels, that, when rendered, send all the information available at that point and allow the platform to univocally identify a person
  • Fingerprinting, using a browser resources and libraries to get a simple “fingerprint” (supposed to be unique to each browser/PC), generating a hash from it, and then use it to identify that browser on subsequent requests.
  • ETags, (the ones in which we are focusing on here) where a couple of headers are used to save a hash on the communication between the client and the backend, allowing the backend to know if the requests are coming from the same client.

What are ETags and why they are used?

ETag (aka entity tag) is an HTTP response header used as an identifier for a specific version of a resource; i.e it tells the browser if a resource was changed, and if that’s the case, ignore the cache and get the new content. This allows to save bandwidth, avoiding to send the full response and making caching more efficient.

ETags mechanism is implemented as a header exchange – depicted in the picture below. It starts with the client asking for a resource (1), then the server responding with the resource and adding an ETag header (hash-like, depending on implementation) representing the version of the resource provided (2). All the following calls from the client will include that generated header so the server can compare and match the value to see if the resource is the same or was changed (3-5). If the resource is still the same, the server will reply (4) with a 304 NOT MODIFIED status and no further information included; otherwise it will send (6) the new resource data with a 200 OK status and the new ETag value, which will be updated on the client side.

As you can see, except for the hash value, no extra information from the client is involved, making the use of ETags not only an excellent tracking option, but a modern solution to comply with privacy policies.

The first question the reader may ask is: if we already have cookies for this, why are ETags required? One of the most important reasons, in particular in the e-commerce area, is that the use of cookies is being slowly restricted starting with the recent GDPR regulations in the EU making some of our tracking methods non-compliant with this document.

As the problem started to rise on the tracking end, we decided to combine the classical sales tracking methods with ETags as a back, to individually attribute a session to each potential buyer (usually called “lead” in e-commerce jargon).

Implementation + code

Our custom solution is not as straightforward as it sounds on the previous section. We had made some tweaks to adapt the ETag approach to our old version and combine them to get even better results. Some of these mechanisms include:

  • Cookies: classic cookies usage when allowed
  • ETags: replacing the cookies with the ETag exchange using the ETag hash as a session
  • Cookies-ETag hybrid: both mechanisms used, not only storing cookies on the client side, but also using the ETags header.
  • Fingerprint: implemented to use as explained above; didn’t work as expected due to new privacy policies

The next code snippet shows, in a Java-like pseudocode, how a response is built when the ETag header is present.

As you can see, we don’t modify the ETag hash ever; that’s because once we create an ETag for a specific browser, we don’t want to change it so we keep tracking the next requests as the same “source”. Where we do create and include the new sessionId and the header is when the request doesn’t have the “If-None-Match” header, meaning that the “communication” is new and no tracking was started. Also, we can see that, besides the ETag, we are sharing no personal or private information from the user, since we are only interested in tracking the results of the campaign to improve it in the future.

Caveats and Results

Adding this mechanism and combining it with some others that we described above allowed us to improve tracking results considerably for our customer. However, with ETags there are two cases considered as new communication that imposes some limitations on this technique. The first one arises when a specific browser performs the very first communication with the server – so, as with cookies, we have no way of associate this request with previous ones. The other one is strictly related to the functionality to which ETags were meant to: caching; i.e. when the client clears the browser cache or request an explicit resource reload, flushing away every resource stored for the site and requesting them again, as a fresh communication.

Conclusion

According to our experience, the ETags + cookies approach:

  • Is definitely a good choice if you want to track clients in an efficient and simple way.
  • Allows a server to recognize the same client in each subsequent request without using cookies or any other non-compliant privacy method.
  • No personal data extracted from the client making the communication anonymous.
  • It is easy to implement and add to every existing app without the trouble of adding new libraries or frameworks, just a couple of headers.

In general, if you want a simple tracking method to know where different requests are coming from and uniquely identify them, this is one approach that you should consider.

2020-01-28 12:08:39 Using Docker for CI/CD and Productive Environments: Part 2 – Services and Tasks

In our previous Tech Note, we described how binaries for multi-module applications can be created in a Docker environment, ready to be deployed. In this post we are going to discuss the underlying deployment infrastructure that can be used for staging and productive environments. To ensure app uptime and scalability, we choose a stack that adjusts resources dynamically according to the required throughput, but also would adapt to new requirements when our app grows. To automatically provision and configure the resources of our Stack, we used CloudFormation.

Step 1: Building a Stack

Our CloudFormation Stack

A CloudFormation Stack is a bundle of AWS elements that are created based on a template – written in general as a JSON or YAML file. Following the same logic that a generic container image could work for building every module in our deployments, we designed our Stacks for staging and production environments following the same principle: the main structure (the image) is shared between every Stack and the content (the concrete binaries to run and their configuration), differ. This way, we can reuse the underlying image across all our projects.

For provisioning containers, we use ECS and, in particular, we use the Fargate compute engine. With Fargate, AWS provides a way of provision container execution without having to define underlying infrastructure like EC2 machines. The engineering team just have to choose which container image to run, the parameters that will be passed to it, and that’s it

ECS works mainly at three levels, Cluster, Service and Task Definitions. We can summarize the ECS Cluster as nothing more than a collection of Services and Tasks (running containers) grouping. As a design decision (not a limitation) we decided to create one Cluster by Stack, one Service by Cluster and one Task Definition by Service.

A Task Definition is the name given to the element that provides the required configuration to provision containers in ECS. A Task Definition also describes underlying resources like the amount of vCPU cores, and how much memory will be provided to the container. The Task Definition has all the end-state configuration for deploying and running it and, if Docker has the internal configuration for our app (for instance, a custom command or/and entry-point), the Task Definition will define those as a part of its configuration.

An ECS Service is a manager for instances of Task Definitions. The Service is in charge of spawning the amount of desired containers and, if any instance fails, those that requires to be re-provisioned will be launched again. The health of the containers are checked all the time through health checks – for instance, pinging them in HTTP port every 15 seconds. A Service takes care also of creating and destroying containers according to auto-scaling policies (if provided) and also to attaching them to a load balancer. A Service has a strategy for its deployment, that will be used to launch each task, as many times as necessary, until one manages to be healthy. There are two strategies for the services: DAEMON, that only let you have one task for each service, and REPLICA that was designed to distribute the traffic through tasks.

The Elastic Load Balancer, as expected, distributes loads between two or more nodes (in this case, Tasks) according to one or more policies. But at the same time, the ELB will check if every Task is healthy and will inform the Service if one or more are not passing the basic health checks. If that is the case, the Service will start the draining and killing process and will take care of relaunching a new Task if required.

All the previous systems are secured through a Security Group. In a nutshell, a Security Group can be seen as a firewall that controls access from/to IPs or others Security Groups. By default, a Security Group let any IP/TCP or UDP packet out from its system, and denies every access to it. If needed, we can edit the inbound rules to allow traffic, always remembering to protect the private parts of our apps with them – for example, if our backend is private, we can restrict the access to only other systems in a valid Security Group, making public the inbound only in the frontend.

For logging purposes, it is recommended to use integrated CloudWatch Logs support, which collects all the information sent to the standard output from the Tasks. Logs can be easily searched using CloudWatch Logs Insights – see our post devoted to that.

To anyone interested in a real example of a CloudFormation Template, a simplified version can be found here. With this structure, all our stacks are actually implemented in the same way, where we only need to edit the parameters when we create a stack, not the resources themselves.

Step 2: Loading a Task Definition Overwriter

To implement an automated deployment as a part of our CI/CD processes, theoretically we must need to update the original Stack every time that we want to deploy a new version of our app. However, we chose a simpler approach.

At this point it is important to mention that an ECS Service is configured with a specific Task Definition that will be used to instantiate their containers. When the Task Definition is updated or changed, the Service takes care of deploying new containers and draining old ones, ensuring zero downtime.

Since the structure of a Task Definition can be imported/exported using a JSON schema, the approach that we decided was to save a template for it in S3, replace the required placeholders that change on every deploy and then update the Task Definition via the AWS CLI. A simplified example of our JSON used for Task Definition rewriting is listed below.


Conclusions

In this second part of this Tech Note series, we saw how to define the infrastructure that will be running our app and the underlying foundation that it implements. With the concept of Stacks, we can provision all the infrastructure that we need for new modules only providing the required variables in the template, without creating it from scratch each time that a new module is integrated in the project.

Compartir el post

Share on facebook
Share on twitter
Share on linkedin
Share on pinterest
Share on print
Share on email