Heatgen

Ever wondered how exactly e-commerce and other large scale websites get to know what products or sections on their website are popular and in-demand? How can this analysis be done in an instant say 10 seconds or less?

Many people think that AI is yet another solution for this well, this can be true but moreover they use real-time analysis methods like heatmaps in order to know which regions on their website are "hot" meaning higher interaction rate.

According to the article titled "What Are Heatmaps? How They Work and Ways You Can Use Them" by Heap and "Visualize your website’s performance with heatmaps" by webflow

• "Attention heatmaps combine many types of interactive data like the number of clicks, their patterns, pages visited, and so on, to provide an aggregated perspective of a visitor's activity"​ (Webflow)​.

• "Mouse-tracking heatmaps monitor a cursor's overall movement on a webpage. They show where visitors hover or hesitate before clicking or scrolling to highlight how users interact with the site"​ (Webflow)​.

• "Heatmaps visually represent user interactions, they highlight specific behaviors based on various clicks, scrolls, and tracking trends"​ (Webflow)​.

• "Heatmaps are effective at giving you a snapshot of user engagement across your site and mobile app"​ (Heap)​.

Thus, heatmaps are indeed powerful for gettin valuable insights for your website along with much more potential now that Artifitial Intelligence is at our disposal.

Our Aim

We aim to create a free and open source tool that website owners of any scale can use in order to not only view their heatmaps in real-time but also get real-time analysis at their disposal so that they can improvise and further plan their marketing.

Description

The project comprises of mainly 4 parts-

1. Our SDK which the host website will use (currently developed in react).

2. A flask server dedicated for generating statistical analysis.

3. A Node JS server that is responsible for managing all hosts running the SDK and communicating with the flask server and dashboard to manage our users (the website owners).

4. A Dashboard built using Next JS to show real-time heatmaps and analysis to the owner of the website and for the frontend of our project.

Our Journey

We started by doing some research on heatmap based analysis tools on the web and found plenty of them but none of them were a "FOSS" project meaning that they were paid tools and this had us thinking of what exactly they have in common and can we actually build a free and open source tool that can just give you heatmaps in real time and analytics so that,

• The manager of the website just have to use an sdk and use our dashboard to see the heatmap changing live and see their analytics like- visitors, interactions and more.

We started with this idea and thought it would be a good utilization of our skills mainly, full stack development, data analysis, research etc. So, as a team, we decided to build a tool that shares unique features which none of the other paid tools offer you. They are simply, to see a heatmap of your website changing in real time and get a cool dashboard for your on the spot analytics.

Problems we conquered

1. We used html canvas element along with js to build a heatmap by recording user's cursor movements and clicks which is the standard way of tracking cursors and slowly dived deeper into the details of how to exactly show a heatmap that is responsive.

2. We came across a very old package that just fit our needs but it was no longer maintained and we had to change a lot of its code just to make it configurable in our react environment.

3. We found a solution for a responsive heatmap so that any user can conveniently see the heatmap and interact with it. We also offered two heatmaps- one for desktop and the other for mobile. This way, we keep the users classes separate and can track them independently as mobiles are more common ways to use websites.

4. Loads and loads of data used to come earlier that exceeded the limit of payloads in http requests and thus, we also optimized the ways to collect data to make sure the heatmap is not compromised and the servers can easily handle the computations with no problem of wait time.

5. We used to think that the tech stack we decided was an overkill but as we implemented our idea into real life, we changed our minds and came up with an architecture carefully designed to handle such use cases so that we can add more features and make it scalable as well.

Future steps

We will add an in-memory database for even faster computation and handle even bigger loads as well as a global database for storing historical data. This will ensure that the "real time" feature is still there and constantly blooming.

Since we offer a mobile heatmap for mobile users of the host website, we will also offer separate analytics for it.

Allow dashboard users to download analytics. We will also convert our current react-sdk into a modular npm package that can run across any framework and lets the host website connect with our backend. This will be like plug N play only and managers can sign up with our tool and download our package from npm and run it in their websites.

We can have data of heatmaps, so we can apply multiple linear regression or any other ML model that can predict the next heatmap which can hold potential for upcoming resource intensive websites which generally have large traffic.