Resources
Jul 8, 2024 |
58 min
|
419 views

Peter Spangler, Marketing Analytics Leader | Data Science Hangout

We were recently joined by Peter Spangler, Former Head of Marketing Analytics at Lyft, Nextdoor, Toast to chat about marketing measurement and impact including metric selection, incrementality, attribution and experiment design.

Speaker bio: Peter Spangler is an analytics professional with 10+ years in the data science space with a heavy GTM focus that helped scale products at Lyft, Nextdoor and Toast by leveraging open-source tools to deliver business solutions driven by causal inference, propensity models, MMM (media mix models) and experimentation.

________________________

ā–ŗ Subscribe to Our Channel Here: https://bit.ly/2TzgcOu

Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software

The Hangout is a gathering place for the whole data science community to chat about data science leadership and questions you're all facing that happens every Thursday at 12 ET.

To join future data science hangouts, add to your calendar here: https://pos.it/dsh

We'd love to have you join us in the conversation live!

Thanks for hanging out with us! šŸ’›

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hi, everybody. Welcome to the Data Science Hangout. Nice to see you all again this week. Thank you to Randy for covering for me last week when I was out. I'm Rachel Dempsey. I lead Customer Marketing at Posit. Posit's the open source data science company building tools for the individual, team, and enterprise. Thank you so much for spending time with us today. If you're new to the Hangout, this is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others facing similar things as you. And so we get together here every Thursday at the same time, same place. So if you happen to be watching this as a recording in the future, and you want to join us live, there'll be details to add it to your calendar below.

And I know that people really enjoy connecting with other attendees here. I see you all saying hello in the chat already. But if you are interested in connecting with others, I want to encourage you to say hello in the chat and introduce yourself, maybe include your role or your base, something you do for fun, if you're going to Posit conference in August too. But we're all dedicated to keeping this the friendly and welcoming space that you all have made it. We love hearing from you, no matter your gears of experience, titles, industry, or languages that you work in. I made a note to remember to say this, but if you are hiring, feel free to share your roles in the chat as well. It's great to see all the open roles there to be able to share them with everyone here. It's also 100% okay if you just want to listen in for the day, or maybe you prefer to hang out mostly in the Zoom chat. But if you do want to jump in and ask questions or provide your own perspective, you can raise your hand on Zoom, and I will call on you. And you can put questions in the Zoom chat. Feel free to put a little like asterisk next to it if you want me to read it. Or I can call on you to introduce yourself and add some context. I forgot to make the slide a link for today where you can ask questions anonymously. So feel free to like DM me or something in Zoom if it's something you really want to ask anonymously. But with all that, thank you again for joining us today. I'm so excited to be joined by my co-host Peter Spangler, who has led marketing analytics at Lyft, Nextdoor, and Toast. And Peter, I'd love to have you kick us off with introducing yourself and sharing a little bit about your background.

Yeah, absolutely. My background primarily has been working in technology companies. I got started working in an Alibaba startup, which is primarily focused on bringing their marketplace to the United States. And so I got my first taste of marketing analytics working in that environment. I kind of moved on through different SaaS companies like Citrix and Toast. And then I built a marketing science team really from the ground up at Nextdoor, which is just a very localized social media platform where we're really focused on growth and customer engagement, as well as marketing to SMBs. And so a lot of my time was really trying to understand, is our marketing effective? Is it efficient? And what are the kinds of metrics and methodologies that we can use to tell that story back to the business and do it in a way that's compelling? And so a lot of what I work on now are projects that are really mapping business solutions to a scientific approach.

And I forgot to add, what is something you like to do for fun too?

Yeah, absolutely. I recently just started playing golf again. I played golf a lot when I was a kid. I played on a team when I was in school. And I recently just started doing that again. It's finally summertime here, and so it's been really enjoyable.

Awesome. I was just in Scotland, so I went to St. Andrews.

I love Scotland.

And I got to cross over the famous bridge.

It's a beautiful, beautiful golf course.

Getting into data science

Well, to get us started here, I get to ask you a few questions while we wait for them to come in from everybody. But I was curious, what actually got you into the world of data initially?

So I had a really, I would say probably ordinary experience in college. I was studying statistics and political science. And at that time, political science had become primarily a quantitative field focused on understanding the relationships between voting behaviors, election results. And a lot of that mapped to the types of inferential statistics and causational models that we have today. I think the best example is when President Obama was running, that was really the first time that you saw a lot more investment in digital marketing, essentially, as a way of influencing voting behavior.

And so when I started, I was using the tools that universities use, which are essentially paid statistical tools like SPSS. Once I left that academic environment and was going into my career, I realized I'm not going to have access to SPSS. IBM had purchased that product by that time, and it was on the order of $4,000 or $5,000 for a license. And so I discovered R. And R and RStudio just gave me essentially a key to open up my career, a key to a bigger world, and a way of helping to accomplish the goals I was trying to accomplish in my own career while having a community to support me.

R and RStudio just gave me essentially a key to open up my career, a key to a bigger world, and a way of helping to accomplish the goals I was trying to accomplish in my own career while having a community to support me.

And so that's also led to myself and a colleague starting up a data consulting business of our own called Paradigm Data Group. And a lot of what we do is spend time giving support to communities that don't have a lot of attention. So, for example, in my own local community, we built the Shiny app for our university, and that Shiny app delivers essentially inferential data about how we are delivering food to the needy in our community and where we can do better and how many people we're serving and our progress over time. And last year, we helped to train the L.A. County Public Health Department in RStudio and the languages that we use in addition to that as well as the packages. And so there's a lot of opportunity, I think, and a lot of growth opportunity for organizations that want to get into open source. And there are loads of people that are willing to help, and I think that's one of the things that has been such a value for me over time.

Building a marketing analytics team from scratch

So if somebody is here, like, listening on the Hangout, and is that a company that doesn't have a marketing analytics team, but they're interested in building out their first use case for it, what would you recommend when getting started?

I would recommend, I think, if I was to go into any technology or non-technology company and there was not a team or there was a beginning team, the first thing I would do is have a meeting with every executive or the most senior leaders I could at that organization and ask them the questions they're trying to answer. I think that one of the ways that you can be derailed or be prevented from being successful in any data role, especially in a marketing analytics role, is to not capture those core questions. So the first thing I would do is go in and ask, you know, what are the core questions you're trying to answer? I would then understand what the infrastructure we have available is. Are we capturing data that's able to measure the things we need, the metrics we need in order to answer those questions? If not, what are the lightest weight tools that we can spin up? And then I would start answering the lowest hanging fruit first. So I think a lot of it is about being able to build yourself a reproducible analytical framework and build the communication first. In my experience, building trust is the quickest way to be successful. And that really means doing good work, but it also means communicating your work and being someone that is engaging, being someone who's available. And so I would really start out with those first three steps. What are the questions we're trying to answer? How are we going to be able to answer them given the technology and infrastructure we have in our data? And then what is the lightest question I can answer first that gets me started building trust?

Training teams on R

Thank you. I'm seeing a lot of love for marketing analytics in the chat and people who are working in marketing analytics. I see, Sol, you had asked a question a little bit earlier. Do you want to jump in?

Yeah, I can. Can you guys hear me? Sorry.

I can, but you're, okay, I was like you were invisible for a second.

Yeah, thanks for hopping on, Peter. Yeah, my question was a little bit related to your SPSS experience and I guess now with R. Do you have experience with training people who aren't familiar with R on R and would love to hear about how some of that's gone, some learning, some takeaways? Because I think a lot of teams have tools like SPSS or other things, and then the benefits of R are pretty obvious to me. And so sometimes I try to understand how to bridge that gap. Thank you.

Yeah, absolutely. So that is part of what I do when I'm not at work is Paradigm Data Group trains, like I said, the example I was giving is training folks at the LA County Public Health Department. And the example I think with them is very good because it's a timeline, right? The LA County Public Health Department is the largest of its kind and probably in the world, at least in America. And as COVID started ramping up during the pandemic in 2020, there started to be a greater and greater need for reproducibility. Analysis that could be put into analytical frameworks that were built in code and where the results of those could be shared across domains and across other tools. And that is an example I think of exactly what you're suggesting, Sol, is how can we do this in code? And you're right, when you have a really large org and they're not sensitive to costs, you might have something like SPSS. I would say that that's probably more marginal these days rather than the greatest density of data users. But in that case, it really starts from doing what kind of what I suggested before, which is let's find out what your team wants to answer. Let's find out a problem that we can solve together. And then let's use the parts of the language. And intentionally, I always start with a tidy verse because it's like how we construct sentences, right? It's a verb, subjects, and objects. And that can get you off to a very fast start. And I think it also has the benefit of building trust, right? You're delivering something to a stakeholder that answers a question. And it gets you started in being able to work one-on-one with someone who maybe has more experience and is more comfortable in the language.

Gotcha. Or do you have any funny, I guess, snags along the way? I imagine a lot of teams also use SQL, SQL Server. What's your experience in getting teams to connect to these databases? I've had some pushback on IT, security, sensitivity of data, access. But I try to evangelize R, but those seem to be pretty big obstacles for me.

I think the best way to handle... So I've been writing SQL, I guess, for maybe 15 years. I haven't ever... And that's going to be an essential skill in any technology company, right? Because every database is cloud database these days. So you're accessing it through something like Snowflake or Databricks. And in that case, you're going to be able to connect through either the IDE on your desktop or through the platform tools that Posit offers. So Posit offers tools where if you would like to, you could simply write SQL inside of a browser window, right? That browser window is essentially is when we're connected to an instance in the cloud. And then you're pulling down data through the connection that you build inside of the IDE. It's relatively easy. And it's an example of essentially usually two lines of code. There's a passkey and there's a library that allows you to connect to those databases.

I would suggest that if it's a security question, that you really look at the resources available through Posit. This is something that is very, very common, but I can tell you it's easily overcome. When I worked at Lyft, for example, we were advocating to use Posit's platform tools and it was not a challenge. I think you just have to be able to demonstrate the security and what's under the hood of Posit. If the company like Lyft is comfortable with their data being passed through this connection, I can't imagine anybody else wouldn't be happy.

No, super helpful. Thank you so much.

Sure.

Prioritizing requests and influencing strategy

Thank you. I see Arthi asked a question in the chat. I'll read it out. But it's being a marketing analyst, I know how fast paced every day is. How do you help your team prioritize requests and have you faced situations where you recommend strategy changes, but the business doesn't want to implement them?

Absolutely. Let's start with how do you manage the day to day? So the way that I work primarily is through prioritizing our workflows, meaning what we're, which projects we're going to execute on and resource to based on size and sensitivity. So for example, I have a very simple framework of where I measure the size of a project. What is its contribution to the organization? How are we defining that contribution? And it is a sort of arithmetic way of doing it. It's quantitative, but it also needs to ladder up to one of the strategic goals of the organization. So choose an example. At Toast, for example, you know, one of the strategic

goals of Toast, like all SaaS companies, is acquiring new customers. So if we have a project that is focused on that, and there will be many projects focused on new customer acquisition, how do we compare those different projects to each other? Meaning what is the size potential of the contribution of each one of those projects? Is this going to get us, you know, 100 new customers, 1,000? So I would first focus on prioritizing based upon the influence, the effect, the impact of the project itself. And make sure that you're working with your stakeholders.

If you're reporting into a director or a manager or something, make sure that you, you know, spend time and take the opportunity with that manager to sit down and say, hey, you know, I'd like to be able to more effectively manage my work. Maybe I feel overloaded. I don't have time to do the quality of work that I would like to. And here's a framework in which I'm thinking about using with my partners to size projects. And then internally as a data team, you can discuss those and prioritize them yourself.

Changing strategy through analysis

When it comes to changing strategy, that typically comes from an analysis, meaning some data product, some data science product you've created.

I can give you an example. Oftentimes in marketing, what we really care about is are we getting close to our goal or plan for the organization? Meaning like if we say we're going to deliver a thousand customers this month, are we delivering, are we pacing to that? If you're not pacing to that, which will occur from time to time, you need to understand why. And so you might create an analysis that surfaces that maybe a channel is underperforming. Maybe something has changed in your acquisition funnel. And you need to understand how this change is contributing, say, negatively to your goal.

If that's the case, then you need to be able to first identify the problem, define it, to be able to express how it is related to the same thing, to one of the strategic goals of the organization, and communicate that to your team. And once you communicate that to your team, make recommendations on how you think you could attack answering this question. What are the types of analytical tools you could use to understand what the problem is? And then if you can define that problem, if you can define the influences of that problem, sort of the root cause of that problem, then you can make recommendations to strategic changes.

Marketing has a bit of a challenge compared to most product analytic roles, where the marketing funnels themselves can change on a regular basis depending upon tactics that change. And so oftentimes you'll see volatility or variations that are out of the norm, and you need to be able to work with your marketing partners to understand how they may be contributing to those changes, and be able to tell that story back to the business.

Personal background

Okay, maybe a quick, fun question, but what did you want to be when you were a little kid?

It's funny, when I was a kid, my parents spent a lot of time traveling with us. I have a twin brother, and we spent a lot of time traveling, and my dad was a geologist, so we spent a lot of time in national parks. And I think when I was a kid, there was nothing better than being outside. And I think even now, I think about, like, how could I do my job with more travel involved? But I didn't really have a specific career path or anything in mind. I was mostly just very curious, and I knew that I was interested in kind of understanding how people work.

And I think that's why political science was so interesting to me. It was sort of getting to how people make choices. And a lot of what I do today is trying to understand how people are choosing one thing rather than another. What are those different types of signals that people's behavior gives that we can use to better understand what might happen in the future?

Decision frameworks and ROI

Definitely. I love that. J.U., I see you had a question in the chat on decision frameworks. Do you want to jump in?

Hey, everyone. My name is James Yu. I support GIS and its data science and analytics initiatives. I had a question for decision frameworks. You being in the marketing science industry and having been there for such a long time, I understand that key stakeholders have to make immediate decisions based on ROI and different key performance metrics. I was wondering what type of frameworks you utilize. And I mean, is there different methodologies? I know that typically a lot of marketers have this approach of maybe testing everything that they want to question, right?

But the thing is, let's say, for instance, in a finite space, you don't have the perfect data. You don't have the ability to identify the sampling that has evolved based on the customer cohorts. How do you make your decision frameworks? I mean, are we making some sort of exploratory analysis that is based on probabilistic attribution of these customers? Or what are we doing? And what is the new shift? I mean, there's a lot of research that has been going on in the customer centricity framework. I mean, I'm hearing that a lot of people are moving away from that shift now. So what have you been doing in industry and what type of decisions do stakeholders make from that specific product set? I know it's a specific use case. You may not have the answers, but feel free to deviate it from it.

Let's talk about that. So you mentioned the term ROI, right? So that's a that's going to be a common point of talk. I would say a point of conversation generally with any finance partner. So one of all marketing partners, no matter where you are, is going to be a finance team because that's who holds the budget for marketing. And the job of finance, like the job of, you know, any good finance teams to be skeptical. You know, if a marketer says, like, I need a thousand dollars to acquire a customer, they're going to say, why do you need a thousand dollars? And you should have observational data, right? You should take an empirical approach to this. What is the empirical evidence that suggests you need a thousand dollars to acquire a customer?

So it does start with every like every other analytical exercise. It does start with EDA. It starts with exploratory analysis where you're providing, you know, descriptive statistics on the performance of your marketing.

Let me let me use an example. So, for example, if we want to evaluate the return on investment, how much you're returning in revenue compared to what you're spending. That can happen in both a probabilistic way. So, for example, like it can happen in an incrementality study. Or it can happen in a naive way, which is, for example, you spend a thousand dollars on a Facebook ad and you get eleven hundred dollars back. And so your return is essentially a hundred dollars, the 10 percent marginal return on your spending.

So I think that depending upon where you are with the finance team, depending on what your organization is like, there will be more skeptical or less skeptical. I think one of the one of the ways marketers often work is they create exactly that example I gave. So create a CPA, a cost per acquisition number. It's based upon a set of assumptions and that cost per acquisition number is largely arbitrary. It's something that they usually think that they can achieve, but it's not based upon efficiency, meaning how low could it possibly be?

And that's when you start having to have more causal frameworks, meaning incrementality studies. And those come in a variety of flavors, whether it's a randomized controlled experiment, whether it's a geomapping, geomatching experiment. But at the end of all those experiments, there should be some metric that you can validate internally on your own data.

I think that the one of the challenges for all marketing teams is being able to make those causal statements, meaning that if we spend another hundred thousand dollars, a million dollars, ten million dollars, we will know that this marginal difference is incremental, meaning it would not we would not get this revenue if we didn't spend this money.

I think that the one of the challenges for all marketing teams is being able to make those causal statements, meaning that if we spend another hundred thousand dollars, a million dollars, ten million dollars, we will know that this marginal difference is incremental, meaning it would not we would not get this revenue if we didn't spend this money.

Those are the most challenging questions to answer, and I think that is where oftentimes marketing teams and finance teams can get disconnected, is that there's a lot of value for the marketers or the marketing analytics folks should be spending time communicating these concepts. Evangelize for measurement, but spend a lot of time on communicating the concepts of causality and how you're going to define success.

Recommender algorithms in marketing

I think this might be Sunday. You said you have a question that's similar to the current one, so I'm going to jump to yours if you want to go next.

Thanks, Rachel. Yeah, Sunday I work with Premier Incorporated Senior Research Analyst. I'm just interested in knowing, do recommender algorithms play any role in your work? I know very often people think of them, those in like context of like, you know, shopping sales and that kind of thing. But, you know, for your work as a marketer, you know, analysts in that space, do this, how much do this play a role in, you know, your work or any other algorithms probably that you use?

When you say algorithms, do you mean algorithms that we use to measure marketing performance or the algorithms of the platforms that are delivering the ads?

It's neither this or that, you know, when I think about, you know, whatever recommender systems, especially I'm interested in those, do they play any role at all, at any level?

Yes, absolutely. So I'll give you an example. At Nextdoor, you know, one of the things that we're always trying to accomplish is increasing engagement on the platform, meaning getting people to log in, to connect with each other, to communicate with each other. So delivering value to the members of the platform. And in our case, one of the objectives we had was creating more connections between users. So, I mean, we want to see people connecting with each other so that they can communicate. And in that case, we developed our own algorithm, which essentially was a distance measure, to think of like a cosine distance or something like that. We developed an algorithm based upon a distance measurement and then features about users themselves in order to recommend who they should connect to.

And that, so that plays a very critical role in also how we deliver ads, right? We want to make sure that the ads themselves are relevant to the person receiving them. And we know certain things about users, we have features about them, and we can measure similarity to the success of an ad with somebody like that. So, for example, if an ad is successful with me, and there's someone who has a very similar profile to me based upon some metric that we can create from an algorithm, like a distance metric, we might serve that other person the same ad.

We also did that, something similar on our advertising platform. So, Nextdoor advertises small businesses. And what we want to see is that people are clicking on those ads and engaging with the businesses on Nextdoor. And so we would explore different models, different algorithms in order to rank those ads. And it could be even something as simple as an algorithm that ranks the density of ads, meaning like how many ads do you see while you're scrolling? And ads have positions inside of platforms like Facebook and Instagram and Nextdoor, and you have to be selective about how do you, how dense do you want those placements to be?

So you're right, it matters. Those algorithms are important for re-recommending connections or relationships, but those relationships could be a number of things. It could be a relationship from you to a company or you to another user.

Measuring experiment outcomes

Thanks. Just a real quick one is, do you, how do you then measure, you know, your improvements afterwards?

Right, right. Great question. So you need to have a metric. All experiments really require two things from the outset, at least, which is you need a metric that's sensitive enough to change during the observational period. So you need to know, you know, how sensitive is this metric? Then you also need a metric that you know represents getting closer to your goal. So, for example, if we wanted to understand if we were increasing engagement on the platform, and we say engagement is defined as the percentage of days in the month that you're on Nextdoor, we can't wait a month to run that experiment.

Right? So we need some metric that correlates with that. And we know one of the things that correlates with that is your connections, the number of connections you have. And so we chose connections as our experiment metric, meaning the objective function of that experiment. And then we would be able to identify, you know, if we ran this experiment on, you know, X number of users, we would expect hypothetically based on effect size, say 10% increase or 15% marginal increase, this many new connections. And we'd be able to then have a power analysis to determine, you know, a sensitivity analysis to determine if that was going to be sufficient to be confident on that result. So it's about making sure that you're selecting the right thing to measure, which can sink an experiment, right? Because if you say, oh, we need six months to run this, you know, you're probably going to get some pushback.

And then you're going to have to have a metric that ensures it's correlated with something up funnel from the final objective that you're interested in. And in marketing, similar to those ad placements in marketing, what we're interested in oftentimes is revenue.

But, you know, a precursor to revenue is people clicking on ads. Are we getting more clicks per session, for example, per user on ads? And so you have to think of it sort of as a sequel, I think all things as a sequence. And what you want to do is work upstream in that sequence, where you can define the relationship from those down to those down funnel events.

Thank you. So, I guess, thinking about the data science hangout space, like, how would you recommend they go about measuring like positive impact of the space?

Of the hangout itself?

Yeah.

Yeah, absolutely.

I'm just saying this, I'm just thinking about that. It's hard to measure like the great vibes in the community and everybody hanging out here.

I would first think about, like, what is the objective of this, right? There are many, right? But many of the objectives are part of the product itself. Like, Posit wants to create a community for all of us to support each other. That means reaching out to each other, too, right? Like, when you log into the open forums, they're going to be people that are not just going to give you, you know, sort of what I'm doing is like a narrative of a framework or something.

But really give you help to create a solution. And so I think that part of the value of this hangout is definitely the level of engagement, right? And I would think of engagement as something that you can measure over time. How many people are joining this hangout?

And this is the way we would measure this on any social platform, right? How many people are commenting? How many people are reacting to comments? What is the average number of reactions per comment? What is the density of those comments? Meaning, what is the percentage of people in the hangout that deliver a comment?

How is, like, what is the curve line sort of look like of the number of attendees over time? Like, is it growing in that that line is very steep or is it starting to flatten and it's a little bit sort of reaching its apex?

I would also start thinking about, can you extract learnings from the actual content itself, right? Like, what are the subjects? If we were to plot the distribution of attendees against, and we were to look at that over time, and we, and that's a scatterplot, right? Like, what is that over time?

If we labeled those points in the scatterplot, what is the topic of that hangout, right? So then we can see sort of what is the linearity, what is the sort of effect size of the topic itself on the amount of engagement.

So I think there's loads of things that you can do. I mean, we have everybody's email address, right? Because we emailed everybody to join this hangout. What are the domains of people that are joining the hangout?

Yeah, I didn't mean to ask you a free consultancy type of question. But I'm just saying, I'm using as example, because we all are very familiar with the hangout, of course. But yeah, thank you. I see Alan asked a question in the chat a little bit earlier. Alan, want to jump in?

Yeah, hi, Peter. I'm curious, coming from prior roles and experience into working with LA County, if there were things that directly carried over, or if there were, like, assumptions you ran into that broke there, like, does success look different when you're working with an agency like that versus in other places? And what carried over cleanly? What assumptions broke? Just, you know, how that experience differed from what you've been doing previously?

Working with LA County vs. commercial companies

So I would say working with LA County, since it was more of a getting started consultancy, we have a framework for getting started. And we like to think of it as sort of the same way that you would have a boot camp, which is like just getting a lay of the landscape, getting visibility on the language, and getting to deliver something to them as an example that they want to answer within the same session, for example.

Or the same group of sessions.

And I think what I'm doing in companies is different, which is, it's not so much about, I don't spend a lot of time talking about methodologies, or how I get to a conclusion.

I spend a lot more time talking about the nexus, the connection between the data science product itself, and the recommendations I'm making. So I spend more time on the inferencing part.

And I spend with, you know, and commercial work with somebody like LA County, I spend more time on how do we get there. And data visualization, I would say, is probably the most powerful part of that.

For organizations like county governments that aren't going to have big expensive paid tools, like they're not going to have Looker or something like that, when there's many, many thousands of people that would need a seat for that product, right?

And not everybody there, like you have in a technology company, not everybody there is going to write SQL. And so Looker becomes sort of not useful, because if you want to build those views, under those views are big queries that are essentially aggregating data together for a specific visualization purpose.

So I think that it is making sure that you're answering a question that they're interested in, getting them started inside of the IDE, and then being able to leverage what are the most valuable parts of the programming language for a new user.

Thanks, I appreciate hearing that.

I see, I think Jared had a question earlier, I'll read it, but it's do you use a customer data platform? And maybe if you could tell us what that means as well.

I'm sorry, you're asking the person asking the question.

I'm asking for Jared, because I saw a little star next to it. But do you use a customer data platform, a CDP?

Yeah, absolutely. So most marketing works are going to use a CDP.

Can you let us know just what that is?

Yeah, it's essentially a way of connecting users to marketing, for the most simple explanation.

For example, I think at Toast, like many places, when you engage in a piece of marketing content, based upon the ability to track, say it's a piece of content or an ad, something else, a website that you might visit that belongs to that domain, you'll be put into a campaign.

And the campaign is specific to that engagement that you've had in there and their marketing.

And the goal of that is essentially to move somebody through a marketing model.

If you're selling a product that's expensive, like if you're selling an enterprise level SaaS product, that's going to be very different than somebody who's selling a product on Instagram, right?

Because those are very transactional immediately, right? You see that ad for this or that clothing item or something, and that starts to move the transaction along right away, you probably complete the transaction within a day.

And so when you're thinking about what marketing should accomplish, it should accomplish a few things in that funnel, which is creating awareness for your product, delivering content around how you're solving a problem for a customer more than anything else.

And then telling your story about how other users are getting value from that product, and then eventually connecting you with someone who could tell you more about how this might work in your specific environment.

I think the best SaaS companies tell that story often, and they tell it in as many places as they can.

I think the best SaaS companies tell that story often, and they tell it in as many places as they can.

If I think about the case of Toast, a lot of the growth that Toast has seen is inside of domains that were not part of their core customer base, but where their solution was relevant.

And those are things like food trucks and bars and restaurants.

And by paying attention to the types of customers you're acquiring and what channels you're acquiring them in, what they're consuming inside of that funnel will give you signals on the types of work you can do inside of a CDP.

It's interesting. I was thinking back to COVID, when Toast was helping out a lot of restaurants who were suddenly doing a lot of takeout.

And one of those restaurants was my mom's bar in Connecticut, and just helping her get stuff done.

But I imagine there was some big analysis done after the fact to see what was the impact of that, giving away some of your services during COVID.

Right. I think that one of the things that I think is really important to think about is, whether you're an enterprise customer or you're an individual business owner.

Zoom is an example of that.

Right. I think that one of the things that, especially one of the things SaaS companies do is to get you started inside of the product.

Whether you're an enterprise customer or you're an individual business owner.

Zoom is an example of that.

It wasn't that long ago, I worked at Citrix in 2016.

You know, GoToMeeting was an Oracle's product, but really sort of the only competitors we had.

And now nobody knows who GoToMeeting is.

In just a couple of short years, Zoom invested in a product and went after verticalists, right? They went after very specific verticals of who would be successful with their tool.

And they told that story really, really well.

And now Zoom owns that market, essentially.

So I think you're right. It's like getting someone inside the tool.

And you can start measuring the contribution of that, right? You can, for example, the way that we usually get inside of a tool is through a trial, through a free trial.

And you can start measuring the conversion rates out of that trial for different types of customers.

And maybe even different types of locations to understand how much value you're extracting.

Alan, I see you just put a question in. Want to go next?

What is a CDP and how does it differ from a CRM?

Yeah, thanks. I think this is sort of exploring, like, what does a CDP do a little bit more?

Because I've been in this conversation a bunch, and people say, we've got to have a CDP.

And they describe it, and it feels a lot like it's a CRM plus other stuff.

And it feels like it's sort of this incantation that they say.

And it's hard to actually define, like, functionally, what's it additively doing?

So if you've got a – it turns into what feels like a sort of, like, well, this one goes to 11 sort of argument for me sometimes.

And, like, I still don't understand exactly what you're telling me you need.

And I'm not in the seat to, like, drive getting that thing anymore.

But I'm often, like, hanging along with these conversations.

So, like, functionally, Peter, is there, like, a way that you can concisely differentiate those things?

Or is it my perception right that, like, these are sort of in a big bucket together, and it's hard to differentiate?

I think that the best way of thinking, conceiving of a CDP is what you're trying to accomplish.

I sort of kind of go back to what we talked about initially of how would you do an ROI analysis.

There – you first need to find what it is that you're trying to accomplish and who your customer is and what kind of marketing you're doing.

So, for example, if I'm a product marketer, a lot of what I'm going to be doing is non-transactional, right?

It's creating awareness. It's telling the story of my product.

It might even be including, like, having events and things like that where you're talking about the product to respective customers.

But I would spend time, based on your type of marketing, looking at those different tools, looking to see what the tool most accomplishes what you're trying to accomplish.

So, yeah, so that makes sense. That makes CDP feel more like it is a process or an activity versus a platform, which is often what it's shaped as in the conversations that I'm in.

Is that the right way to think of it?

It is an aggregation of activities and the right tools for the job versus a thing that you get to do all that stuff?

So, okay, let's talk about a specific one. Let's make it very specific.

So, I mean, Salesforce is a CDP.

Okay.

Okay.

Salesforce is going to aggregate, like you said, all of the events for every one of your contacts.

Yep.

And a contact will just be an email, essentially just an email address.

And that's going to be able to map to some customer journey, right, given a sufficient analytical resources.

So, Salesforce is a CDP.

I mean, but there are other CDPs that don't do the same thing.

So, for example, a Twilio is a CDP.

Twilio is the anonymized technology that communicates between you and your driver at Lyft or Uber.

It's making possible for the product to function and even the marketing to function, right, because that Twilio tool is also going to be what's used to send you text messages about discounts and offers and things like that.

But it's not Salesforce in the sense that when you're going to look and see what are the most common types of content are being consumed

or if somebody converted into a product, that's Salesforce. So, it really depends on what you're trying to accomplish. Both, one thing they both have in common, right, they are platforms. They integrate into your data. And by integrating your data, I mean like they integrate into the actual events firing in your product.

So, a question I. Oh, sorry. I think I was lacking in my Internet connection for a second. A question I wanted to ask you, like thinking about all the different companies that you've worked at, what is one of the most surprising things you've learned about user behavior from marketing analytics?

I guess it doesn't have to be like the most surprising, but something that was surprising. Yeah. I think that one of the things I've learned that I was struck by early on was that people understand value based upon something they can map to themselves. Based upon something they can map to themselves.

So, for example, when I worked at Lyft, we had this hypothesis that there would be, we'd be able to identify the likelihood of acquiring a driver based upon the features of a driver that map to another driver on our platform. So, meaning another way of saying that is people that are like each other will behave similarly. Right. And I think as someone who is, you know, living in the cult or where we live today, I think a lot of how we think about people and everyone is different. Right. That's something we talk about a lot. Right. Whether it's in medicine or whether it's as consumers, everybody is different. But we know from, you know, he talked to any statistician, everyone is different, but they're also very similar.

And so one of the things we found was, is that, you know, depending upon where someone lived, the neighborhood that they lived in, like something very nuanced, depending upon if they knew someone who was also driving for Lyft. That made them wildly more likely to join our platform and become a driver. And so I think that like really the insight there is that try to think about how people can be similar as much as they are different. And that people understand value based upon something that they can relate to.

And that people understand value based upon something that they can relate to.

So for example, if you think about asking someone to do something in your marketing funnel, meaning like say make a purchase, if you present them with something that is very, appears to be very low value, like if you were buying Zoom and it was 99 cents, that's probably going to harm your conversion rate because it doesn't look like it has very much value. Anything that's a dollar doesn't have any value. Start valuing your products the way that people would perceive that value. And especially when you tell that story, tell the story about how people extract value from it so that when you present them with a value on purchasing, that it makes sense. It's something that they can relate to in themselves.

Geo analysis and marketing measurement

I guess I need to read or listen to the Freakonomics podcast as Curtis is suggesting here in the chat. Jared asks a question. I will read it, but I might need some help with these acronyms. Oh, you spelled it out. Okay. Do you use defined marketing areas or core based statistical areas? Can you be more clear about that specifically, like provide an example? Jared, do you want to jump in?

Sorry, a little noisy here, so I should stay muted. Yeah, so a lot of marketing, you have households. And the way you break up households could be by geography, could be by CVSAs, micro metropolitan areas. But you might have a city that sits on the state boundary where you have very different activity just by the factor in two different states. Market defined areas are nil since what people watch on TV or radio and defined areas like that. I would recommend for anyone who's not part of it, especially if you're thinking about doing any type of geo analysis, I would recommend joining the Census Bureau's live channel. It is full of loads of resources, many of them in R, because most of the geo work is done with libraries in R. And there are loads of folks on there. There's an R channel within the Census Bureau's live channel that is very active. And there are many statisticians to help accomplish people's geo analysis goals.

So I'd start with that as a resource. Yes, Jared, to your point, yes, I do use, I oftentimes break those down like cities or households into census tracts. Census tracts, because it creates homogeneity, right? So you want to make sure when you're doing any kind of analysis that you understand the variation within the units that you're looking at and then within the subjects. So I think of census tracts as also being, or even DMAs, as being a really great opportunity to run studies. So, for example, you can look at the package that was created by Kim Larson, who's one of the, I think he was the first head of data science for Citrix. He's a friend of mine, created a package called Market Matching. That's a way of using those geo locations in order to test hypotheses about any number of things, including your marketing.

And so if you have a household, for example, and you're trying to find other households that are similar, you could match them on census tract, for example. But yes, that's definitely something interesting. One of the examples I would give with why that is so important, those geo locations, is when you're doing paid marketing. So, for example, on Facebook, the way those algorithms work is that they're going to capture the cheapest click. And one of the most influential variables in that model is where's the greatest amount of capital, meaning just where's the greatest amount of money. And so your marketing will primarily go into the 20 or 30 largest cities in the country. And if you're trying to understand over time how the saturation of customers in those cities could be influencing the success of your marketing, meaning how much does it cost to acquire a customer, how valuable is your average customer, you have to start being able to break down your acquisition based upon those locations. So having mapping inside of your own domain data, meaning your own database, of where a customer is, is very important.

Sales forecasting with sparse data

I know we have about five minutes. I know. I don't want to miss Jacob's question. Yeah, Jacob, do you want to jump in and ask that? Yeah, sure. Yeah. Thanks for taking the question. I know we don't have a lot of time left. Yeah. So, Peter, just basically, you know, as the question says, have you ever had to, you know, do any sales forecasting with, you know, sparse data?

Yeah, absolutely. The reason why I was asking is because I was assigned a marketing project from our consumer brands department where we were pulling Shopify Klaviyo data to forecast sales at a zip code level. And basically all I had was about, you know, six to nine months of customer sales data. So it was kind of tough to use a traditional RFM approach. And in terms of any linear modeling, the only thing I could use was publicly available census data. But that led to a lot of, you know, bad models. So I was just curious to know if you've ever had anything similar happen to you. And if so, what kind of approaches did you use to solve those problems?

Yes. So there's two types of. And if you want to email me, I'll put my email in the chat here. Yeah, sure. But I will start with this. There are two types of forecasts, basically, when it comes to sort of marketing work, which is. Sorry. Which is a forecast based upon inputs. So, for example, like if I don't know what kind of data you have, but you might have things like impressions or website traffic, things like that. That is something you could use to build a forecast from. I don't think I would start with doing it at specific zip code. I think I would start with doing it at the aggregate level. And then if you have the same data for every the same amount of data for every single zip code, you could test it on some zip codes to see how close you get to the actual values.

Another way I would do this, and I think I probably have an example I can share with you offline, is thinking about doing this with a structural, a structural model, meaning a time series model where you're just fitting a line. Right, you're fitting a predicted line to the actual data. Matt Dancho has some really good resources on this that are free and publicly available. He's our guy out there as far as like publishing work. I would also check out. You could try using profit, which is Facebook's forecast model, because it takes does take inputs. And there's a great getting started in our. I think the time series one if you're just if you don't need explanation of why you're getting you're you're getting the results you're getting meaning like you don't need to know. Do we need more impressions do we need more website visits. And you don't need to know why but you just need to know if you're right. I would think about using a time search.

Yeah. Yeah, the way I was trying to approach it was looking at population density and racial breakdown, but as I said before that didn't really didn't really work too well but yeah probably don't have a huge correlation to, especially zip code level they're probably not really strong predictors. That's, and do you have to know why you're, you're forecasted sales are performing the way they do. No, they were just they were just trying to figure out which areas would would be potentially would be a good candidate for spending more budget in the marketing. Okay, okay.

Okay, if that's the question. If that's the question. I don't know if that's what the way I don't know if that's the way I would do it. I would think about okay if you're looking to see where it's actually it's like where should we be growing. Basically, yeah, yeah, yeah. Um, let's talk offline shoot me an email. I actually tried to tackle something really similar to this. Talk to you about that. Yeah, thanks. I appreciate it. Thank you.

I know Josh had asked, you mentioned the Census Bureau swap channel. Do you know how people can get the invite to that or where to find it. Is it. Josh has shared a link you just wanted to confirm if that was the right one. Let me put it again. Yeah, look, the one I found was just when I googled, but it looks like you need to be invited to it. It's not just open. So, I'm not sure if I found the right one if that one's meant just for government employees or. No, no, I'm looking at my Slack channel of my US Census Bureau. I think that's it. You can also. Yeah, I would try that 1st.

Okay, I think we have covered most everything. Sorry if there were any that I missed, but thank you so much, everybody for all the great questions and thank you so much, Peter for joining us today. This has been great. Awesome. Thank you guys.