Navigating Brand Advertising Analytics with George London

Very happy to be here. Thanks for inviting me on.

Yeah. I’ve had a bit of a nontraditional career journey in technology. I actually did not even study computer science. I studied philosophy at Yale. Then as I was about to graduate, realized it’s a little bit hard to get a well-paying job in philosophy. I needed to get a real-person job and ended up getting a job doing investment research at Bridgewater Associates, which is the world’s largest hedge fund. That was a really fascinating role. It taught me all about macroeconomics and investing and financial industry. I also started about 15 days before Lehman Brothers blew up and the global financial crisis started. So it was a really interesting front row seat to some pretty massive turmoil in the financial industry. But luckily, Bridgewater had managed to see the crisis coming, and so they were well positioned to ride it out. I got to do some really interesting analysis there. In particular, I worked on a cool project trying to integrate data streams about government financial spending from things like the Stimulus Act, which were spending literally hundreds of billions of dollars trying to help turn the crisis around. And that was both a really interesting modeling problem and an interesting data management problem because I was looking for data from all these different sources and government statistics agencies and things like that and trying to put that together into a coherent picture that would show how the spending was impacting growth and economic health, both in the US and in other countries.

That really gave me a taste for how impactful data can be on effective decision making because Bridgewater was making very high stakes decisions all the time. They were investing billions of dollars, betting that either growth was going to go up or it was going to go down. It was very important to get the answer. After a few years of doing that, I realized that I was more interested in the data problems than in the financial problem, specifically, and left to do my own startup that was trying to help initially make it easier for people to find data for use in financial modeling. That went through a few different pivots and iterations, as many startups do. But it eventually became a platform for helping people find metadata about music and get more connected with musical culture. And then on top of that, built an app that recommended electronic music that was both personalized to your individual taste, but also to your activity at a given time. If you’re looking for quiet, steady music or looking for high energy dance music, you would find the music that was right for you. That was also a fascinating startup experience, but it taught me the critical importance of distribution and consumer apps, which is something that one should very much not take for granted.

And also the very substantial legal and economic challenges of working in the music industry. I decided to move on from that. But through that process, I had built some cool recommendation systems and learned even more about data management and the programming skills required to do that effectively. So that set me up really well to join Upwave, which is where I’ve been for the last eight years. Upwave, I started out helping them do a statistical analysis of survey data because Upwave’s core business is helping advertisers, particularly brand advertisers, plan, measure, and optimize brand advertising campaigns. There’s two major types of digital advertising. There’s what’s called direct response. We’re trying to get somebody to click on a link and buy something right away, which is what things like Google AdWords typically are. They’re trying to drive very short-term conversions. But then it’s everything else that tends to have longer-term objectives and it doesn’t typically drive a click at all. A canonical example would be, say, a car commercial, which is not trying to get you to jump off your couch and buy a car today, but instead it’s trying to make sure the next time in the market for a car, you remember the Mercedes or Toyota exists, and hopefully you consider one.

And so Because nobody can click on a car commercial, it can be very challenging to actually figure out the cause and effect of how the money you’re spending on your brand campaigns is translating through to actual business outcomes. And so Upwave helps close that gap by tracking exposure to ad campaigns and then targeting surveys to people who’ve seen ads and have not seen ads. And by carefully comparing people exposed to ads to people who have not been exposed, and using a process called causal inference to remove any pre-existent testing effects of targeting, you can actually calculate a correct incremental unbiased estimate of the lift that’s driven by these ad campaigns. And you can say, these car commercials for Toyota actually made people 10% more likely to consider buying a Toyota. The key to being able to do all of this is being able to actually record and track and organize and analyze both ad impression data and survey data. And so it’s the data management and Python-based programming skills I learned from my previous roles were really helpful for building that analytical engine that continues to power Upwave statistical analysis today. At Upwave, I started writing code, but then got it into management over a few years, worked my way up in management, and then became CTO a couple of years ago.

Yeah. So Upwave does many different things to help make brand advertisers most effective. The most commonly recognized thing that we offer is what’s called brand lift, which is just telling advertisers how much their ad campaigns drove increases in core KPIs, like awareness and consideration. Brand lift is not a new thing. It’s been done for decades, but it’s traditionally been done as a consulting and research service with very human involvement, with humans handwriting surveys, setting them up, setting up tracking, doing all the analysis in Excel or other desktop programs, these kinds of things, and then carefully putting together a PowerPoint presentation at the end and giving a big grand reveal of the PowerPoint. It’s certainly better for advertisers to know how their campaign did than to not know. But the problem with all this is, one, it’s a very expensive, very labor-intensive consulting engagement, and two, you typically don’t get the result until after the campaign is over. Maybe you get a mid campaign report. If it’s a three or six month campaign, typically, you don’t get real clear answers until the campaign is already done. And so that means that by the time you get analysis of performance, it’s too late to do anything to the campaign.

If the campaign, almost no campaign comes out the gate, perfect. Some of them are highly effective but still have substantial room to improve. Some campaigns are just not that effective for one reason or another. And if you know that early in the campaign, then you can actually understand what’s going wrong and you can do something to improve of it. And you can make sure that the money that you spend on the campaign actually ends up being well spent. But if you don’t have the results until the end, it’s too late to do that. And so Upwave’s core thesis is that we could turn this brand with a process from a consulting thing into a software thing. And we could basically completely re-automate from end-to-end the process of commissioning brand with, setting it up, executing it, doing the analysis, and doing the reporting. And it’s been a journey over the years to get all the way to automation because the advertising ecosystem is complex. But we’ve gotten really very close to having everything fully automated to the point where we can now be a button in certain demand-side platforms where somebody who’s buying an ad campaign can just click the Upway button and say, Okay, everything happens on directly behind the scenes, and suddenly they get actual granular actionable reporting on how their campaigns are performing and optimization recommendations that can tell them how to make that campaign better in real-time.

And then even to put a cherry on top, the advent of generative AI in the last year has meant that we can even do reporting that is comparable in quality to what used to take human consultants days or weeks of time to do. And we actually will even give you a complete PowerPoint presentation available at any time during your campaign that summarizes how it’s going. Upwave really pretty much offers everything that all the competitors offered at similar or better quality, but also with these really the key advantages of being both much more cost-efficient and much more timely and much more actionable.

Yeah, so there’s a lot going on under the hood. The interface that our users who are mostly either advertisers or large publishers interact with is a pretty familiar SaaS Web dashboard, not that dissimilar to other SaaS tools in its appearance. It’s a basic client server, React front-end, JVM ecosystem back-end. But where the real magic is happening is all of the analytics that’s happening under the hood. We have to ingest advertising data. Ad campaigns They’re very large. So we’re ingesting tens to hundreds of billions of events per month, tracking all these different campaigns we’re ingesting, which means we need to have a large, complex, scalable data pipeline for ingesting all this advertising data. Then we also need to go out and collect the survey responses on our own. We have our own in-house survey collection mechanism. We work with a number of external partners who integrate what we call our survey wall, which is basically like a paywall, except instead of having to pay money to read an article or play a game, people take a survey to unlock some premium content, and then that lets us collect survey responses. And then those survey responses have to go through some complex statistical analysis.

So we have a pretty bespoke internally created, we call our analysis pipeline, that’s doing some pretty heavy duty statistics using Python and the various Python data and machine learning toolkits. Then, as I mentioned recently, we’ve added generative AI capabilities. We’re using OpenAI, GPT-4 to integrate all this data that we’re collecting and produce dynamic human readable reporting on top of that.

I’d say there’s two major things. One that I touched on earlier, which is timeliness. The campaigns we measure typically run between a month and a year, and so have large amounts of money spent on them. And so being able to have results promptly enough that you can actually make decisions and make changes early enough in the campaign to make a big difference is a perennial challenge. And so that’s one of the reasons people come to Upway, again, is because we give them data very early in a campaign and do as much as we can to make it actionable early, to let people have as much clarity and confidence as they can in the analytics that we’re doing. And so tell them as clearly as possible which tactics are working best for them, whether that’s their advertising on 100 different websites, and five of those are really driving performance, and 10 of those are driving performance down. We can tell them clearly and early which websites they should emphasize in order to get the best results. And the second challenge, I would say, is just complexity. Advertising is complex. The ad tech ecosystem is complex. Modern national brand campaigns may have media plans with hundreds or even thousands of line items on them.

They’re advertising on hundreds of different websites, many different channels. They might have digital banner ads and connected TV and streaming audio all on the same campaign. And so being able to accurately track the exposures across all those channels and calculate performance and compare performance in an apples to apples fashion is a challenge. And so Upwave has put a lot of work into making it as smooth and ergonomic as possible for people to just get set up with all the tracking they need to do, have that happen seamlessly, and then have all the reporting come out in a way that is clear and understandable. And then so the genero-AI innovations we’ve done lately have also been really helpful for boiling down the very large data set that can be produced into something that is human-readable and understandable and that highlights to people both what is most impactfully driving performance on a campaign and also where the biggest opportunities are to actually make changes. Advertisers can’t realistically change everything on a media plan, even if it might be optimal to do so being able to point them in the right direction towards a small enough set of things that are going to be high impact that they can actually focus their energy there and make real changes.

It’s a challenge they have, but it’s one that Upwave is really focused on helping solve.

Yeah. So we use the pretty common suite of analytical tools, things like Google Analytics, other SaaS tools in that category that help us understand our users engagement and activity in our platform. And we look for points of frustration, we look for common user flows, and find patterns that we can put more emphasis on making smoother or more accessible or more intuitive for people. We also talk to our customers frequently. We have a support team and a support at upwave. Com alias that does an amazing job of taking questions from customers and really getting to the bottom of what they’re trying to do and both helping them solve the problem in the short term. But then also our product organization talks on a very regular collaborative basis with our support and success organizations to understand what our customers really want. And then we try to be as customer-obsessed as we can in designing our roadmap to make sure that the things we’re building are the features and capabilities that our customers are really going to get the most value out of.

Yeah, I’d say I have to give credit to cloud services like AWS that they really have changed the game in terms of helping startups take more scalable approaches. You still have to architect your systems in a good and scalable way, which we have put a lot of effort into doing. But we found that the data platforms we built because they were designed to be parallelizable and horizontally scalable, as we get more data, pretty much all we have to do is just press a metaphorical thought in AWS to add more compute power to our clusters. We’re able to scale pretty smoothly from a technical perspective, I’d say that one of the bigger challenges we’ve had has been going back to the complexity of media measurement and just the… There’s so many different places you can advertise nowadays. There’s so many different channels, there’s so many different websites, so many different approaches between going back, even just a streaming audio. There’s so many different podcasting platforms and websites that do streaming and people streaming on different devices and these kinds of things. And Upwave wants to help our customers through advertisers, measure their campaigns everywhere they’re running.

Many of our customers are also publishers who may run very streaming apps that have many different types of ads, whether they’re post-read ads directly in the podcast or they’re dynamically inserted ads. Just being able to keep on top of this, the industry is constantly introducing new channels and new ad types and new formats, that thing. Just being able to keep up with making sure that we can track all these different types of exposures and provide the granular reporting that our customers need, it keeps us on our toes. No, makes sense.

Yeah, so there’s two things that I would really highlight. One, I will just call digital convergence. People in the industry have been talking for a long time about convergent TV, which just means the fact that TV used to be broadcast. Originally, it was radio signals. I can from my house see the Suttro Tower in San Francisco, which is the big radio tower of the city that broadcast TV signals to the Bay Area. And we then moved on to cable, which was still a broadcast mechanism. But over the last decade, we’ve been moving to connected TV, which is this TV coming through the Internet. And TV is still a medium where people spend a huge amount of time and where a huge amount of advertising is delivered. And the The fact that TV has become digital has really changed to the way that TV advertising is created, bought, and measured. But the convergence of TV is not the only thing that’s converging. It’s really that all types of media are converging to become digital. I’ve mentioned streaming audio a couple of times. Audio also used to just be radio, which was broadcast and much harder to measure, much harder to address.

But now people are listening to a tremendous amount of audio that’s delivered through the internet, through apps, and that thing. Print magazines still exist, but they are going away, to an extent, and being replaced by people reading just about everything through screens. When virtually all media consumption is digital, that means that Any type of advertising that’s done through those media can, in theory, be done using the same common set of advertising tools that have been developed over the last couple of decades. And suddenly, radio ads are not just something that are recorded once and broadcast to a million people. Instead, each individual person listening to a podcast can get ads that are, in some sense, particularly made appropriate and interesting for them. And the opportunities for measurement also become much greater. And the easier something is to measure, the easier it is to optimize. And so that means that the plausible performance of advertising just really goes up. But it also means that people who maybe are more used to traditional ways of buying advertising have pressure to adapt to this more modern landscape and to actually start thinking through a performance lens. In many ways, brand advertisers for a long time have been insulated from needing to think too much about performance because performance on linear TV or radio has been very challenging to measure in the past, but now it’s easy.

All you have to do is work with a company like Upwave, and suddenly you can start actually collectively understanding understanding your campaign across all its channels and actually start optimizing and taking a real performance mindset. And that’s a huge opportunity, but it’s also a mindset shift. The second major trend I would highlight is AI, which I think is just going to be, has already been transformed the industry, but has so much further to go on transforming things. There is just, as a bit of a broken record, but there’s a lot of complexity in advertising. There’s a lot of time that’s spent on just trying to make different systems work together effectively to execute a modern ad campaign. And that has traditionally required a lot of expertise, a lot of trial and error, a lot of frustration from a lot of people. And AI, even in its current generation, is already able to act as a connector between many different systems or a translator. So even things like taking the reporting on a brand Live campaign, which was sometimes thousands of different numbers and required quite a bit of human interpretation analytics, and being able to just pass that as well-engineered prompts into GPT-4 and get back a clear accessible summary is so time-saving, and we’re working on things that are going to help people take those recommendations and hopefully make them more directly actionable in other tools.

This is just what the current generation, but AI is getting better over time and getting smarter every year. And as it gets smarter, the scope of things that are going to be possible to have it help with will just increase. And I think people are going to find that the process of executing advertising becomes a lot easier over time as people are able to rely more on AI to help them out with the day-to-day. But it goes even farther than that if you start looking a little bit farther into the impact of AI the impact that AI can potentially have because AI can do things like help with what’s called Advertising Creative, which is the, say, actual TV commercials. Just a few weeks ago on X, formerly known as Twitter, saw a commercial from Toys R Us that was produced entirely using Sora, which is OpenAI’s video model. And it’s a little unnatural. It doesn’t quite feel all the way like a commercial. If you just look the progress that video models have made over the last year and take this Toys R Us commercial and extrapolate it a year into the future or two years or five years, it’s really not that hard to see how these things are going to become much better.

And I soon potentially become indistinguishable from things that were produced by humans or even better than things that were produced by humans in the not-too-distant future. As it becomes easier to connect data on performance to things like the actual creative that you’re putting together and how you’re delivering that creative through the media, the opportunities for doing really effective advertising using AI are going to become pretty impressive over time. But that also has its own challenges, which is the As there’s going to be more and more AI-empowered advertising over time, it’s going to be something of a flood of this type of advertising, which consumers may have negative reactions to. You may start to see the consumers try to distance themselves more from advertising by doing things like using AI to do product research on their own or using AI as what you call a buyer’s agent. For example, somebody shopping for a car may no longer go to Edmunds or to the car manufacturer’s website, but may instead start their search by going to an AI and saying, This is my family arrangement. These are my needs. What car is best suited for me? And if the AI is just boiling the information down and making a recommendation for them directly, then there’s fewer opportunities for the car companies to influence decision making through advertising.

So it’s going to be this really interesting race between advertising getting better, but then people relying less on advertising to become informed about their buying research process.

Exciting times.

Yeah. So culture is something I think is really very important to organizational success. But the key thing about culture is that Culture is what you do. It’s not what you say. And you can put together a culture PowerPoint presentation and talk to your company at an off-site about these are our company values and these are the things we believe. But then if you go home the next day and you as a leader act in a way that is inconsistent with those values, then people will immediately understand the real culture of this company is what the leaders are doing on a day-to-day basis, no matter what they say the values are. If you want to be collaborative or you want to have a collaborative culture, then you as a leader need to collaborate with the people on your team. You need to listen to them. You need to treat them respectfully. You need to be transparent with them. You need to reward collaboration when you see it. You need to give people positive feedback when they’re acting collaboratively. You need to give people negative feedback when they’re acting not collaboratively. You need to actually every day show up and create that environment through your actions.

I do genuinely believe that engineering is a team sport and that no matter how, quote, unquote, brilliant an individual engineer might be, if he or she he is working off in a silo or he or she is being destructive or aggressive towards the rest of the team, pretty much no amount of individual brilliance can overcome the negative impact that has on the organization overall, because tech platforms are large and complex. They need multiple people working together. And you, as the manager or leader, really need to create the environment where people can be brilliant by how they work together and how they bring their efforts to get together in a seamless way to produce success for the business. And that just comes from hundreds of daily actions and not from any rhetoric.

Yeah. So there’s two things, both of which I’ve touched on a bit earlier. One, the flashier, one is the generative AI. We call them our AI Campaign Insights Reports that we rolled out earlier this year. I think those are just really impactful and useful for customers. We’re already seeing heavy adoption for them and seeing customers tell us that they’ve saved them a lot of time and have both to help them understand what’s going on, help them communicate with their end stakeholders and so on. It’s a real practical production application of generative AI using GPT-4 to do something that would have been literally impossible to do before it. It’s really cool to actually see AI in action and directly adding values to customers. This is not just a flashy demo. This is an actual production application that people are getting real value out of. I’m very proud that we’ve been able to build something like that. The second thing I’ll say is just the extent to which we have been able to rein in the complexity of doing measurement and automate it. It’s just years and years of effort and analysis to understand all the different places when humans have needed to be involved in this process, why they needed to be involved, the complexity they were dealing with, and then progressively building more and more parts of our system to automate that all end-to-end.

You wish that there was just one thing that if you built a system that did this one thing, then everything would just magically be automated. But in practice, that’s not how it works. It’s just there’s There’s dozens and dozens of different touch points that we’ve had to handle one at a time. But through long-term patience analysis and engineering, we’ve been able to get to just a really smooth and automated solutions that I think is transformative for the industry.

Sure thing.

I think probably I have a temptation to spend a little bit too much time on individual tasks, trying to make them all super high quality. I was calling myself a perfectionist. It’s a little bit cliché, but I think I do have that tendency. When I was an IAC, individual contributor, I really put a lot of emphasis on trying to make sure that all the code I wrote was really good and effective. When I was a line manager, making sure that everyone really got the full attention and support that they needed and all the communications were perfectly crafted. As CTO, there’s just so many things that I have to do on a daily basis and so many balls to juggle that the only way to keep all the balls in the air is to be okay with doing a good enough job on certain things, even if it’s less than the ideal I would have liked, because it’s just more important to keep the balls in the air than it is to do a great job of every individual thing. Becoming CTO has been one of the harder transition points for me.

Yeah, I would say we have to do finance, tax accounting stuff and carefully categorize and summarize all of our projects to make sure that they’re properly reported for tax purposes. That’s a tedious one.

Astronaut. Okay.

Might have been looking you up on LinkedIn to understand what I was getting myself into.

I am very interested in economics in general because I think it combines philosophy’s orientation around how through understanding the world around us and thinking things through deeply with a practicality that focuses on trying to understand the real world day to day and a bigger picture. I think it’s really interesting to think about the interplay between economics and culture and advertising and how society evolves over time and how that’s influenced by technology. It all comes together under that general economic lens.

Was great speaking. Thanks for your time as well.