How to tame your Big Data dashboards

Great reporting is the Holy Grail of digital marketing retainers. Whether you are reporting on Paid Media, SEO or CRO tests, your data should be illuminating, educational and as fresh as it can be. It should empower our team to optimise marketing performance, and clients to make good business decisions. But it’s true for everyone that by the time the agency-client reporting call comes around, it’s already out of date. We’re not the only ones who think this is a bit lame, are we?

We need to find a way to empower clients with this information. Clearly, empowerment is not going to come from a traditional marketing report. What we were looking for was a way to show test data (to compare all our metrics against the previous month, previous year, and against our forecast) that clients can see in real time.

Traditional reporting is a boring has-been

• Data on traditional reporting is static and always out of date
• Collating data from multiple streams can be time-consuming

We made a decision to use realtime data, explored this, and chose Google Data Studio as the best route.

Your data is powerful. Use it. Unlock the power of your data with interactive dashboards and engaging reports that inspire smarter business decisions. It’s easy and free – Google Data Studio.

There are other options (like ReportDash, DashThis, Swydo, Klipfolio, Cyfe…) but Data Studio is within data structure we already use and pulls data from all sources. And, y’know, it’s Google. Plus, the aesthetic is in your own hands. Fantastic!

We hooked it up to Analytics, Facebook Ads, Google Ads, MS Ads and pulled the data into client dashboards with a ‘fetch’ programme (let’s call this #1). Initially it worked. We started to roll it out and scale up, very excited that our clients were going to get something innovative and clever.

The amount of data was eating us alive

Quite quickly, it became clear the dashboards couldn’t cope with it. Everything slowed down. The more data we tried to bring in, the worse it got. Then we started hitting the daily API targets, and everything just about stopped altogether – Google was blocking us daily because the scale of the data we were trying to manage was massive.

At the root of this is how Google Data Studio works. Whenever you load up a dashboard, it is reactive and requests fresh data. Because we were being ambitious and asking for data from multiple feeds, we got timeouts.

We also discovered that we weren’t able to get the full benefit of the dashboards. There was better conservation of data by displaying the previous full day’s worth of data. So, the reactive pulling wasn’t required.

Also, some conversion data was laggy in some streams and would only be updated overnight.

We’d invested a lot of time but we had to admit, this wasn’t working.

Our sophisticated dashboard evolution

Convinced it could still be done, we did a deeper dive and came up with a plan. SPOILER: it works!

What we are now doing is one big synch from platforms into the data warehouse every morning. Put another way, we’re not pulling multiple queries regularly throughout the day – just once.

To support this, we’re keeping the data from all streams we’re observing is in a data warehouse – in our case, Google BigQuery, which has a very sophisticated setup.

We use fetch tool #2, an API that queries Google BigQuery and brings the stream information into the dashboard once a day. We chose a new fetch tool that:

• Has lots of integrations
• We can scale
• Has good support
• Can cope with Big Data query variables (it doesn’t crash the dashboard)

To put it into realtime context:

• Fetch tool #1’s dashboard loaded Client A’s website data in 71 seconds
• Fetch tool #2 + BigQuery loaded it in 0.8 seconds

That’s a 98.9% decrease.

I believe this kind of dynamic, customisable dashboard is the future of reporting, making digital marketing teams more efficient and puts really useful information at clients’ fingertips. Try it – you won’t want to ever go back.

If you want a marketing agency that uses data like this, here we are.

Photo by Michal Mancewicz on Unsplash