image: thumbnail.jpg
Bryan Butler | R in Marketing - Survey Design | RStudio
Presented by Bryan Butler
In marketing, a common way to gather data and insights is through a survey. It is often thought that surveys do not provide enough data for machine learning. In this talk, we’ll go through some survey design best practices for applications of machine learning. Then we’ll cover a use case where we use survey responses to segment customers with unsupervised machine learning, and then perform classification with supervised machine learning.
Speaker Bio:
Bryan Butler is VP of Business Insights & Analytics at Eastern Bank
Transcript#
This transcript was generated automatically and may contain errors.
This is only going to be about a 30-minute talk, so we should have plenty of time at the end for any kind of question, discussion, or whatever we want everybody else wants to talk to after all of this. And this is a project that I've worked on over the years, and it keeps coming back. So that's one of the reasons why I decided to bring this up with a marketing group.
I've been in and out of marketing groups for the past few years, and also worked in marketing research. And so a little bit about me, I am a head manager of analytics data science at a small bank in Boston. I started out in chemistry and then got an MBA, did a lot of work in the insurance industry doing some pretty exotic stuff, got heavily involved in Monte Carlo and pricing, and then got involved in quant banking consulting. So I think, as you can see, like every other data scientist, I just went the exact straight route out of college into data science.
Overview of the project and goals
So what we're going to talk about here is what this project is all about, and I'm going to try to give it to you in a way so that you can take this to your next marketing project. Talk a little bit about what good survey design is, or what I think it is. There's an interesting bit of intellectual property that the ability to work on called customer quotient, similar in a way to NPS, but also very different. We'll look at some parameters to constraints, then getting involved in the machine learning part of how do you actually do machine learning with survey data.
So the role of this was, how do you take a survey, because that's what a lot of people in marketing research do, and develop a segmentation engine out of it, so that as people complete your survey, you can put them in the right segment for whatever, and your segments can be based on whatever. In this case, this was a sort of a behavioral psychographic segmentation, which means it wasn't so much, you bought product A, B, or you live in this part of the world, or wherever. This is more about how you feel and how you behave. And so it gives you a little different approach to sort of the different dimensions of marketing.
And it was looking to reveal, this is, again, this is part of a much larger project, how people view companies. And the other way to think about it is, at the end of the day, it rates how customer-centric a company is. And that's how the questions were sort of phrased.
Good survey design
So these are some of the things that I've learned about what makes a good survey. And these are just my, these are my takeaways from being involved in surveys and trying to apply machine learning and, you know, how do you get it, how do you ensure you get a lot of respondents, et cetera. So the first thing is, I want a goal of, like, what is the survey design to answer? And then you're going to want to have some supporting questions. So I was thinking structurally, that's how we want to start it. And the goal, you shouldn't have more than three things you're trying to answer. I usually say one to two is good, just in case, you know, you pick up some other ancillary information. And these goal statements are binaries, yes, no's.
My first experience was dealing with a survey and people are like, well, how likely are you to purchase from us again? And so you end up with these neutral, you know, everybody's in the neutral range, and it doesn't really help you out. Or what do you do when you get a handful at the positive and a handful in the negative and a handful in the neutral? Do you drop them? Do you include them with something else? So this sort of resolves that issue. It's a simple yes, no. With that, the supporting questions, you want as few as you can get away with, but still get the information that you want. So, you know, I like to think maybe you're between five and 10 questions. If you start getting over 10 questions, you're just getting too much, you're asking for too much information, and you'll find that people will drop out over time.
The flip side around it is in the marketing research world, you do have the ability to pay people with, pick your incentive, Amazon gift card, PayPal, whatever else, they will stay in for longer, knowing that if they finish it, they'll get some money at the end. And I've done a couple of these, I've done many of both, and there doesn't seem to be a difference in engagement as far as like, you get different answers if you pay people.
And then here's a key thing. I like my supporting questions. So we got our one binary, like now instead of how likely are you to buy from me again, is I have bought from you again, or I will buy from you again, yes or no? And then we're going to ask a bunch of questions that support it. And I think everybody's seen these, strongly agree, agree, neutral, et cetera. I do not like that scale. One, you don't get as much variability in the modeling phase, whereas one to 10 gives you plenty of variability. And I think if you go over more than 10, people don't know what the numbers mean.
And just as a little aside, I worked in the wine industry for a while, and you know, they have their 100 point wine scale. But the funny thing is 99 point, whatever percentage of the wines are ranked between 80 and 100. So what they really have is a 20 point scale. And so I don't know, they disguise it as a 100 point scale versus 20 point scale. Same thing here. 10 points will give you enough variability.
And then the other thing is, you know, people always say, oh, we can ask a question a different way to make sure that we don't have people gaming the system. Maybe if you get this giant survey that are like a personality test or something like that, but that's a whole different animal here. If you're surveying your customers, they're not going to, they're not going to do the duplicates. And if they start seeing the duplicates, it'd be like, this is a waste of time. And it's also a waste of your time, too. Like every question matters.
And I've seen some other indices out there where they have like, they try to decompose like a net promoter score, which into three indices, and it's kind of like, the three indices are so incredibly correlated that you just wasted two questions asking about the other two indices when they're like 90% correlated. So it's kind of like, you really have one index. In the market world, I know anybody who's worked in marketing has heard of net promoter score. It's kind of a hybrid because it does ask on a scale of one to 10. But you can also, but at the end of the day, you can make people binaries out of it. In fact, there is a, you know, people that score you nine and 10 are your promoters, six to eight are your neutrals, and five and below are your detractors. And the score only works with promoters and detractors. So it's a very unique scale. But when you do NPS, you should really ask, you know, get into the whys.
And then a good rule of thumb is to try to get about 1000 responses. Because you can do some really good modeling. I know everybody thinks, this is a world of big data, I need 10,000, I need 100,000, I need a million. No, you don't. You need a well designed survey. And if you get close to 1000 responses, you will get a reasonable amount of information.
You need a well designed survey. And if you get close to 1000 responses, you will get a reasonable amount of information.
What's interesting about surveys, is that if your questions are distinctive enough, then even if you get smaller segments, you will be able to predict them. So for instance, you know, if segment one is questions one to three, and you know, segment two is maybe a question three and question four, but then segment three is this oddball question, question five, even if it comes out small, and you get into this highly imbalanced data scenario, the fact that question five is so unique, it will help even with the prediction in an imbalanced situation, because it's not random.
Customer quotient
So a little bit about customer quotient. This is developed by a colleague of mine at C Space in Boston, Dr. Manel Austin. And it talks and it builds on this cost of this sort of how customer centric as your company, and they ask questions from from the customer's perspective, and you build up the pyramid from, you know, relevance, you're just in the game at that point. Then we have a customer series of questions around customer experience, how well you do, then the better you are, you get into openness, and that begins to really unlock the doors. Are you willing to have an honest and open two way dialogue with your customers?
Empathy is really what changes the game. So when I've seen companies in the segments, companies in the segments, it's not till you do reasonably well at the openness, and get into empathy that you begin to really like, we've grouped different companies and segments and everything else. And you see, when you start rating up on the empathy scores, you begin to run away, and become extremely well. And the last one is what we call emotional validation. Those are the people out there that are selling your product on their recommendations.
What we used to talk about is you have two people, you have three people having a conversation, someone says, I need to get a new smartphone. And the first person says, get an iPhone phone, and the other person really doesn't say anything. So your iPhone is your sort of emotional validation person. And they'll give you five reasons why you should get an iPhone. Or on the flip side, they'll give you five reasons why you should get an Android or whatever it is, but they are selling your product. And it's called emotional validation. Those are your ultimate customers, because they are doing your marketing. And we find that people that score higher up there actually have to spend less in marketing and advertisement over the long term, and they get some better scores. That's just a little bit of this intellectual property area in the space that I worked on here.
Mapping to traditional marketing frameworks
For those with traditional marketing backgrounds, where you talk about the four P's and the three C's and all those. So the four P's, product price, place and promotion. The three C's are customer, company and competition. And what these do is sort of map various questions that help with the CQ scoring into these different traditional marketing buckets. So that when we talked about pitching this to different companies, if I had a traditional marketer, I could say, yeah, this is your product piece, this is your price piece. And you can see price, it's all from that customer point of view. The company appreciates my loyalty. And that's literally how the question worked. It's an action word, or I don't feel ripped off. And you can see in most of these questions, and these are sort of paraphrases of it, but we've made it action oriented, and it's from the customer standpoint, not I. Versus if you think about a lot of surveys is the company is asking, do you like us? Do we do a good job at this and things like that? Whereas this whole framework really reverses it around.
Project parameters and constraints
So let's talk about this project that we had. Client specification number one had to be built in Excel. And that happens. I wish I could just do it in Markdown or something easy like that or a web app. But at the same time, it's workable. What it does do is it changes the model processing a little bit. I'm not going to be able to run XGBoost or any of that. But at the same time, I may not need to. And there's a good way that I get around this down the road. Our sample size was 900, not 1,000. But that's fine. And it's like, okay, we're probably only going to get about three or four segments out of this.
And part of this came out of is that a major company had a project with a major consulting company. And they said, well, you have eight distinctive segments. And it was like, okay, really? So we're going to test out whether we really have eight distinctive behaviors by using this survey. And the initial step, and this is one of the things that, like I said, this one did not have the dependent variable. And that's why I added it in early on as a good survey has a binary outcome. So we had to create one or create some sort of segmentation outcome. The nice thing is survey questions on a 1 to 10 scale, easy to work with.
And if you think about it, from the machine learning world, as we go down the road, we're not going to have to center scale or do any of that with the data. It's already prepackaged on a 1 to 10 scale. And we're good. Whether we do any kind of, and we're going to do unsupervised learning or supervised, you are not going to have to rescale all the data, which makes it nice. And when you do some of these things, and these end ones are my discussion with them, what's good enough actors? Are there penalties for false positives or false negatives? Like if we put the person in the wrong segment, how costly is it to your business? And should we add penalties if they are costly in one way or another?
And that would change a little bit of the modeling too. We would actually add a cost matrix into it down the road if we had to. But in this case, there weren't too many big deals. And they said, you know what? You give me 80% of the way, you give me 80% of the right answers, I'll be happy. I was like, okay, 85, that's what we're shooting for. Hopefully we can do better. That's one of the things for me is I always like to be in the 90s.
But sometimes it doesn't always work out that way.
So here's a little roadmap here. And you can use this over and over, like any kind of survey work you do. I've used this format for a lot of NPS modeling that I've done for various companies over time.
Design a nice survey. So we ask for NPS people, we ask that question, but then we ask them questions about customer service, we ask them questions about pricing, other aspects of the business to see what you're trying to do is figure out what drives that answer, what drives that yes, no. So then we're going to do some clustering, I like hierarchical clustering, it's easy for visualization, the scale is easy. So we're going to separate our respondents into some different segments and clusters. And then we're going to create a dummy variable for each one. So you're in segment one or you're not, you're in segment two, you're not, you're in segment three, you're not. I think a lot of people would try to go straight to sort of a multinomial model, hey, we're just going to predict what segment you're in right out of with one equation. Those require a lot of data. So yeah, if you're in one of these big data worlds, and you've got 10,000s or 50,000, you could probably pull it off with 1000 simple binaries is going to be the way to go. So here's my little my little magic here.
Elastic net regression for dimensionality reduction
What I first do is use the elastic net regression, the glimnet model, a glimnet package by Trevor Hassey, Rob Tertranian. It was designed for genomics, it really will help you reduce dimensionality, and it's extremely fast. And to give you a little bit more information on this, um, the people that are familiar with ridge, or lasso regression, this is the hybrid. So a ridge will attempt to squeeze down all your coefficients to make them very small. And the lasso will do is if there are multiple variables that are correlated, the lasso will pick one and leave the other ones out. Now the problem with lasso regression is you don't know why it made the decision. Is it, you know, a function of the gradient descent, or a function of the order of the data, you just don't know. And they talk about it in their paper about that, whereas what the elastic net does is, I have three variables that are correlated. If they're statistically valid, we leave them all, we leave all three in, if they're not statistically valid, we pull them all out. And so that's why, how it gets to your dimensionality reduction. So then this last step is, all right, we got to go into Excel. Now you probably could do an elastic net, maybe not, in Excel, but you can easily do a logistic regression. And so what we're going to do is build out our template. So when we get batches of scores, we just run them through our worksheet with logistic regression in there, and then it's going to use a voting method, and it'll choose which segment you're in.
Clustering: the elbow plot and hierarchical clustering
So the first thing we did is, we run this, the elbow, famous elbow plot used for clustering. And it's called within the sum of the squares residuals. And so what we got is number of clusters, they told me they had eight segments, so I went for up to eight clusters, or at least their consulting company did. And what we're looking for is the bend in the plot. And you can see, you get the elbow at about three or four, at which point then it all sort of flattens out. So if I had a lot more data, I probably would shoot for four clusters.
Given that it was only 900 respondents, three would probably do the job. You can see that the sum of the squares goes down, obviously, the most between one, two, and three, between three and four, it's very little enough that tails off. So that's why we're looking at that inflection point. So we plot them all out visually, we get three groups.
And this isn't, you know, you can see in the hierarchical clustering, that you get the divisions, almost fairly cleanly between the blue block and the green block, and the green block and the red block. And the other way to interpret this is, the people in the blue block are very different from the people in the red segment. And there's some gradations across the middle. I use this plot a lot when I do text analytics, because it's really fun then. Because what you're seeing is, when you do this with text, is these people are saying one thing, they're saying something similar to these guys. But now it's getting different, and then you get something really weird at the other end. So I like this plot from a visual standpoint. And the other axis is just the distance metric.
And as I mentioned, because all of our questions were a one to 10 scale, I didn't have to rescale anything to be able to do this clustering. And yep, this is small, and it may be difficult to predict, but it doesn't automatically mean that. Because as I said, if there are certain questions that are answered by a handful of people in the group, that makes them marketably different, then you'll still be pretty good at predicting whether or not people are in there. So when I overlay my cluster analysis across their eight segments, it was kind of interesting. I was like, well, you may have eight segments, but you have three distinct behaviors within each segment. And what I mean is that there is this grouping here, we'll call number one for now, behavior one, that seems to cut across all of them with an exception of this segment. And we also have behavior two. There's people in between, but then this behavior number five stands out. And it appears that this convenience store resellers segment is actually kind of small in general. But looking at those numbers, it's pretty clear that we have, you know, you could describe it two different ways. You can say, we have eight segments with three behaviors. Or you could say, I actually have 24 segments. So granted that I have this social couple, there are actually three forms of social couples.
Now, one of the interesting things about looking at some of these segments is, what about people that would be crossing over some of them? Like, how did the company specifically put them in each segment? And can you move? Or will you move over time? Like how long do you stay a new mom? And, you know, how long do you stay a social couple? Some of these others are a little bit more static, but we got a nice clean picture here. So it's good. Our elbow plot told us we had three or four segments. And most likely three, our hierarchical clustering came up with about three segments, and we can see it here. So the size thing is, so we have three segments, three equations.
Model performance and accuracy
So here's a little bit more on the elastic net. The reason why I use it is it's going to give me my upper bound of performance, or accuracy in this case. Because my goal is to get down to a logistic regression with a couple of questions. But if my elastic net comes in only at 80% accurate on the rough, before we even do any kind of dimensionality reduction, then my logistic regression isn't even going to cross my 80% threshold. So that's going to be problematic down the road. And the way it works is, you know, you think about your logistic regression. It's, you're in segment one or not, you're in segment two or not, and the probability you're in segment three or not. And for each person, you get three scores and the highest score wins. So in this case, the person, this, the data point came back, the probability in segment three was 90%. It beats 55, it beats 21. Therefore, they get assigned a segment three.
Versus I think, when you think about your traditional way of just using a single logistic regression, you generally assign your probability at 50%. And it's greater than 50, it's in that segment. Here, we're using a little, an ensembling type approach, and we're going to, we're going to vote basically. Standard split training testing. So, you know, 900 data points gets me 250 testing and, you know, 650 training. It's not great. It's not bad.
How do we do? So here's the first segment. Segment number one. The lasting net regression. Near perfect. Very little false positives, false negatives. Good split here. So it's not so unbalanced that basically it's an easy guess. So this is nice. 99.26. Really good. And so you see down here, when we did these, I'm not going to sort of show you the same plot three different times.
And another key piece to look at is this, this no information rate, 0.61. That is essentially your data split. If the no information rate in this was 90 or 0.90 or something or higher than that, which when you get an imbalanced data sets, then that, you know, that 95, 99, well, it's a good lift, but it's not as impressive versus we're at 60 on a random guess. And, you know, we're near 100% accurate. And you can see we got 94% on this segment too, and 80 in this small segment. That's, that's not surprising. We knew it was going to be a small segment. We knew it was going to be a little more challenging, but it's still, for the client standpoint, they're quite happy with what they were seeing initially. This is a nice little variable importance plot with the elastic net model. And you've got your questions ranked order of importance. And when the, when the plots out this way, that means they put you in the segment and you'll see in another plot, they go the other way. They're saying, when you answer this question, it kicks you out of the segment. And now there's others, more sophisticated approaches out there. There's LIME. That's probably the most common now that people are using locally interpretable modeling. But this, one of the things that tells me now is I can begin to remove some of the questions from this equation, right? What I'm trying to do is make us simplify this as much as possible. So I'm going to say for segment one, we're just going to use a couple of questions. So that not only have three variables for this equation, make it perfectly simple for it, especially since we don't have a lot of data. And so what did model one, what did question one, segment one say? Well, they appreciate my loyalty. I feel proud, sense of belonging, and they believe the company employees use their own products and service.
So this is, we think, highly correlated to an emotional validation segment. And so these are your really good customers. You want to take care of them. You want to know who these people are. In the world of computer vision, I was at a conference back when we used to go to those things in a live person. And someone said, you should run your image recognition about your customers when they come in your bank because they match the face and the balance or something like that. And you can come up and say, wow, this person's like a really big customer. You should do something to take care of them. Interesting portrayal, but someone would say, yeah, but everybody's an important customer. Just another way to think about how do you take care of your loyal customers and how do you show them their loyalty? And that's a question 16 is an important one. And I know when we were doing this, we were presenting something like this to some internal people. My marketing research person used the concept of Amazon pantry. She's like, she said, they get it wrong all the time, but they make it right all the time. So yeah, maybe I order six things and five out of the six are right. And one of them's wrong, but she said, they always make it right. And I think that's that part of that. How do you get, even if you make a mistake, how do you get people coming back to you? And it's all about fixing your errors as a marketer and as a company.
How do you get, even if you make a mistake, how do you get people coming back to you? And it's all about fixing your errors as a marketer and as a company.
Simplifying to logistic regression
So now when I reduce it down, this is that what I said is we're going to go to a simple logistic regression with just a couple of questions. I lose a little bit accuracy. So rather than almost a perfect with only one off in here, we've actually began to make a little bit more false negatives or false positives here as we think they're in segment one where they're really not, but we're still at 94%. So we're good. And we're not, again, we got our no information rate. So it's, we've got good lift. We now have a happy logistic regression model for segment one. So segment two, as I said, looking at the variable importance, these, when you answer these questions a certain way, they actually pull you out of the segment. So you need them as a way to separate out segment one from segment two.
And if we ran this through Lime, you would actually see a red and green, or if you use the traditional colors, you'll see red and green scoring for these questions that they're important because they will either put you in or push you out of the segment.
And these are a different set of questions we see. So our 16s and 21s, as you can see, are up here, not as important. And here we're imbalanced. And it's a mixed accuracy, right? So yeah, we think we're really good, but most of these people are getting pushed out in the other segment. And this is the logistic regression model for making, and that's where I said the known information rate is 90. We're getting a little bit more lift, not much. And we're making equal errors, false positive, false negatives. But we definitely have these questions that are oriented around customer service. And that's essentially what these four questions point to.
And then segment three, our small one, and this is why it's hard. You got some of the questions that kick you out of the segment, and some of the questions that kick you in the segment, and there's a little bit of both. So you think about it this way, in segment one, these guys were high scorers. So now in segment one, in segment three, you're getting low scorers on here, but now you're getting a different scoring across here. So these questions are pushing you out of the segment. And then in segment two, you're getting a little bit of those questions are pushing you in them into the segment. And the nice thing is, we do okay at a small segment. No information, no information rate is 52%, almost purely random coin toss. And we're able to beat that by almost 30 points, 78. So again, I mentioned it was like, love it at 80%, love to hit 85. But at the same time, given that it's a coin toss, and the data set, then at 80%, we're doing okay. And obviously, if I use a far more sophisticated model, we could probably do a little bit better. But it goes back to the constraints, the customer wanted it in Excel, they wanted to be able to batch load their own data, and just kind of see the levers. And I think that's the other part of the importance of logistic regression. And same thing with Elastic Net, it's just a logistic regression with a regularization function, is that a business person can say, yeah, if I score better with this question, I'm actually going to increase my loyalty scores or, you know, that that's partial differential part where you say, and this is what I do when I do when I analyze surveys, I said, Look, if you increase your score for this question by one point, the probability that they are one of your promoters or, you know, frequent buyers, and everything goes up by 5%. And if you increase by two, it goes up. So it's a very quantitative approach to being able to say, how much do I need to move the lever?
quantitative approach to being able to say, how much do I need to move the lever?
And I'll say in the NPS world, and I was doing this in my last company in healthcare, it was, it's a lot easier to make a person go from a detractor to a neutral, than it is to take a neutral person to make them a promoter. And so as you think about ranking your your actions that you're going to take, and so you're in, in this case, your dependent variables, they're promoter or they're not, and you rank the questions of importance, it'll tell you what's driving your answers. You should focus on ones that are the low hanging fruit, but they're also easy to get them out of the detractor hole. And that's why actually, we did that in a couple places. And it worked, our executives actually had in their bonus plan, that their NPS had to go up by four or five points. And this actually created a bit of a an action plan for them.
Summary and results
So what did we do here? So we started out with our elastic net. We got the question, we got the survey down from 17 to nine questions. That's good. Obviously, the elastic net is very good. I looked at and I didn't show these here, but we looked at CV, ROC curves, confusion matrix, I showed you confusion matrices, just so we could get the few extra stats out of there. We have a model for each segment, we use our voting approach. Overall, the overall accuracy was over 85%. And they said that was the client said it was acceptable. And they didn't have any, we didn't have to come up with penalties for misclassification. Making a false positive or false negative on a segment is from a market standpoint, it's not the end of the world. Maybe if you're spending lots and lots of money on something, it may become more important. But in this case, it was really looking to understand what's going on within their customer segments.
And that's it. So this, so if we look at it, you can take a survey, make a binary question out of it, apply clustering, maybe like KNN better than hierarchical or something like that. Get your segment labels and then apply some machine learning against it. And if you get about 1000 respondents, as you've seen here, even with 900, it'd be reasonably accurate. And you can do this to all any survey. And this is just I do this on a regular basis, over and over different surveys, try to tease out some really good quantitative aspects of it.
Q&A
Awesome. Thank you so much, Brian. I see there's a few questions that are starting to come in right now on Slido. But just want to let people know if you want to unmute yourself or raise your hand on on zoom, you could ask questions live to Lee, I see you had asked the question earlier in the zoom chat to Brian, I believe it said, do you reverse code? Or any other item types? Would you want to add any other context to that question? Can you hear me? Okay. Yep, I can. So earlier, when you were talking about item types, you talked about Likert scales and 10 point scales and all that. I'm wondering if you use any reverse coding or Thurstone type items to make sure people just aren't repeatedly filling out the same value over and over when you do your assessments.
I don't like to do that. We used to have a large, I used to have large conversations with people, not so much reverse coding, but it comes straight liners, which I think is what you're getting at, you know, somebody, they give everything a five or everything, you know, whatever it is, I look at it. I don't want to throw it out because especially with a Likert scale. It's a, you don't know, that could be a very valid data point. 10 point scale, you tend to see a lot less straight, what they would call straight lining. But it's always part of the EDA. And just because we sat on that conversation all the time.
Thanks, Brian. I can read some of the anonymous questions. But again, if you want to use Slido too, you can put your name in there. And you can ask it live as well. But the first one was, how are the surveys conducted and collected? I've noticed YouTube does surveys in between videos. How did you do these?
This survey was done through a marketing research platform. Our same, actually, most of them were done that way. So when I, so I said, this one is through a marketing research platform, I don't know who you are. And you just plug it away. And then we get it out on the back end, I think, like SurveyMonkey and something like that. I've done this exact same analysis with data from SurveyMonkey also. That was on InsureTech industry. We did about 2000 respondents from all from different groups. And some of them are just, a lot of them are generally web based, where you've had a transaction, just like you say on YouTube, you know, hey, how did you find this video, you know, give us a rating one to 10 or sometimes, you know, you'll get a note. At the end of this call, you'll be asked to take a survey. And my health care company that I work for had it that way too. So which was interesting, because now you're punching in numbers on a phone. You can still do one to 10. It's just, it's a little different than click in a box, and you don't get comments as much. So then you, you have to ask good questions, detailed questions.
I was curious, if, if you don't know who is answering the survey, and then you're going and making changes through the business, how do you measure, like, what actually made the change in the survey, or if it was like those actual customers or?
Most of them, most of these types are directed at your, at your customers. So you, so you know, your customers. And in fact, I did one recently at our bank. It was a survey. And so we had all their information. And it was actually a rerun of a service. So they ran a couple pre-pandemic, post-pandemic, wanted to see what was going on. But it was highly skewed by the age of the respondent. And, and, you know, you just have to point that out. Like, you know, I'd actually show a slide to our presidents and say, look, this is our real customer base. This is what they look like. This is what our respondent group looks like by age and demographics. They are different. They're not really speaking for our 100% base. But we will, like one of them was to pull the pull the older segment out and then compare the different segments. But yeah, so you generally try, well, you want to know if they're customers or not. And you want to know So you generally try, well, you want to know if they're customers or not. So that's, that's always an intro question. Or, you know, obviously, if it's coming through your own platform, then you know whether they're a customer or not. But you if you can get demographic data on people, highly recommended to do it if they're, if they're willing to, you know, ask just a couple questions at the end. Because it can be important. Just like I said, with our, ours was a kind of a technology based survey. So age matters.
Thank you. A few other anonymous questions in Slido. And one is what distance metric are using? Are you using for the hierarchical clustering? I use good old Euclidean distance. So think about a matrix where we have 10, 10 columns, and we're just going to, you know, calculate square the distance between along all those questions. So it's very, very standard, very clean. And because everybody was a one to 10, so I didn't have to like, scale something, scale them all down. Like, for instance, if, if you're looking at web, web data, and you've got bounce rate and page views, you can't do hierarchical clustering with it, because page use is going to be in the thousands and bounce rates going to be as a percentage. So you'll have to convert those so that they're in apples to apples type metric. These are all zero, zero to 10. Don't really have to do too much for that.
Thank you. Lucinda, I see you asked a great question on Slido. Is it okay if I pass the mic over to you to ask that? Yeah, sure. That'd be good. So I saw that you, hi, Brian, how you doing, first of all? And I saw that you were using logistical regression for these, which I think is a linear based type model. But I was wondering if you've ever tried, and I know not for this one, because you can't do random forest in Excel. But can, have you used random forest or other ones that are that deal with nonlinear relationships? Well, and if you have, like what's, have you found that there's a, find other, find out other things? I have, not for this kind of stuff, but I have. And what I, what I was doing a lot of work in the insurance industry, if you talk to actuaries, that's all they want to do is use GLM models of some form, linear. And so I showed them a standard data set of, if you think about age and income, right, as the older you get, you were supposed to have more income, but you often see it is like a, a U or a nonlinear, almost like a bathtub where you've got people that are very young that have a high income, then it goes down and then it goes back up. And there's so many, we see that a lot of times in the insurance industry, age of a house, you know? And so the short of it is the naive Bayes actually does a nice job at capturing that. And that was the plot I used to hold up to these people. I said, here's a problem with your generalized linear model all the time. And you're going to miss this every time. Random forest is a good one too. It's a little harder to explain. And I think that that's where it starts to get challenging. In what sense is it harder to explain? Because you see, you have, you have the, the feature
importance. That's how I use it. I use feature importance as my list of things to pay attention to. Yeah, it is. It is. So, so I, I like it that way, but I also try to get into this cause then they, cause they'll ask me, well, how much do I got to increase my score by?
And that was, that's what, that's when the advantage of using a linear model, if you can, you can always, but, and I was able to say, look, you get the score by about five points. You'll be much better off or, or you only need to actually with a lot of this stuff, it's like, you'll see a difference between like a six and a half and a seven. It's like, you only need to get your average up a half a point to get more people over the line. And that's where it becomes really useful is that sort of marginal calculation. But if it's just like, what do I need to fix? Then I think that I agree with you a hundred percent, like, especially in a more accurate, like you'll obviously probably be more accurate with random forest model. And then you can, then, then you can just say, Hey, this is my problem area. It's pretty clear. And that's what you focus on. And then that's what you have, you know, team members and subject experts there to do. Okay. Thank you very much. Huge fan of all that you share with us. Thanks, Brian. Thanks, Lucinda. A few other anonymous ones. One was how big should the sample be to get good results? My rule of thumb is a thousand. And the reason why is because I've seen some, I remember calculating some stat for logistic regression. And it's like, you need a minimum for one variable at a hundred, maybe 200. So you think about, well, if I'm going to have three variables, I'm going to be in the five, 600 range. I'm going to start splitting. Well, you know, you gotta be in the thousand range to, to get a, get some decent results.
And it's come through so many times, like this, this survey had 900. I did NPS with 1200, a thousand, 1400. And I've been lucky to do surveys with 2000. And the nice thing is it doesn't take tens of thousands, I think is a good answer, but being in a thousand, sometimes it just may take you more time though. And that was a conversation I would have with leadership is, you know, we're going to have to run this several months till you're going to get an answer that's usable. And you just have to understand that this is the, this is the rate of answers we're getting and we need X amount. And as long as you explain that upfront, I think most people are reasonable to work with on it. I wonder how that differs by like industry or, I mean, the types of surveys that people are doing and curious to like hear from everybody here as well, if you want to chime in or, or share info in the chat too. I would say the more questions that you have, the more data you're going to need.
Survey design clarifications
Someone had asked, could you clarify the survey design? Are we talking about three binary main questions and several one to 10 scale supporting questions? No. So the survey was just seven, I think it was 17, 17, one to 10 questions. We didn't have, we didn't have a binary. Over time, I actually got them to introduce a binary into it. And so that was why we had to do the unsupervised learning is to say, given these 17 questions, can I find some groupings in which case we could find three groupings that are made up by how people respond to different questions, different questions. We can see that one was customer loyalty, one was customer service, and I think the other one was emotional validation. So, or value, sorry, value, loyalty, and service.
And that was, but yeah, like I always tell people, if you're doing a generic survey, you want to know something about them, please have a binary question. Even if I had a good experience with this company, yes, no, like, because then you can actually use, again, those questions, those supporting questions are your variables to whether or not they say yes or no. And that's why I want to keep it that way.
That's really helpful. When making the model, is there a difference in setting the response variable as an ordered factor or non-ordered factor? In this case, it doesn't matter because it's just a binary. It's the one or zero. And for both the elastic net package, and I think I used caret for this too back then, it just has to be a factor, that's it. So, your dependent has to be a factor and that's it, and it runs cleanly. This was all done in R, so I can't remember if Python requires that or not. I think it can handle a one zero as a standard numeric.
Community resources and R for Data Science Slack
A few other questions, but I did, in case people have to jump at the top of the hour, I did also want to let everyone know that we've made a chat marketing channel on the R for Data Science Slack community. And so, I think it's a great place to go to connect with each other too, or if you have follow-up questions as well. So, I just want to point that out before I forget to do so, but I'll put the link to join in the chat. And then the channel is just chat-marketing.
But back to questions. One was, for bank satisfaction scores, is there a trend by community between historic issues such as predatory lending practices and current satisfaction rates? And if anyone has to add any other context, feel free to jump in too. I'll say I don't understand fully that question, but I will say I read a JP Morgan report out, when did it come out? December of 2020. So, at the end, in December 2020, and it basically showed that smaller regional banks had a way, had a way, much higher satisfaction than the national banks. And part of it was due to the pandemic in that when they did the whole CARES program and the loans, I don't know if anybody remembers all that, when the payment protection plan loans were coming out so that you could keep your employees, most of the big banks didn't, you know, didn't really step up for their customers, whereas all the regional players really jumped in and did a lot of loans. I mean, my bank is quite small, and we processed almost 10,000 loans between the two programs. And I think it was a total of almost $1.5 billion worth for, you know, small local businesses to keep their employees on the payrolls. And a lot of more like small restaurants and small, you know, just really small business getting 25,000 or 50,000 or whatever. And then the SBA once would forgive it, and the whole program, but that was the whole CARES program. So that's the only thing I've seen. So satisfaction scores are generally better for smaller banks. I don't have any insight into the predatory lending part of that question.
Career experience with survey design
Thanks, Brian. Someone else had asked, does your experience with survey design come from any specific techs or purely from implicit knowledge and career experience? Mostly from career experience. So I started out working in a marketing research firm that, I mean, we would issue thousands of surveys a year. That was how they operated. And what I initially found out is that they were just leaving so much on the table because of that. You know, you get a lot of, in an old school marketing was, what was top box, you know, percentage, top box, bottom box. But I was like, that's not, you can do better. We can do better. And so with this original data set that the CQ stuff that underlines, underlined a lot of this, that created the survey, and I was lucky. I mean, we had a hundred thousand responses to work from. This specific data set alone, after we developed the CQ, was 900, but, and then they kept rerunning it. So it was growing. But it's just, I think it's part of it. I've been in this marketing type role. So I had a marketing research role, did the quant shop for them, created one from scratch. And then I found that this approach, I've taken out of three other places and done the same thing, mostly around NPS modeling. And it worked. Like we did the logistic regressions for an insurance company, and it said, you know, here's problem A, B, C, and that's what they did. They worked on problem A, and then we took the survey a quarter later, and problem A was now no longer problem A. It had been cured, and actually the NPS score went up three and a half points. And of course, everybody's quite happy because, you know, you meet a major KPI for your unit performance. Everybody, everybody's happy then. So it's, part of it is, it just works.
Everybody, everybody's happy then. So it's, part of it is, it just works.
Recommended packages and resources
I know we just hit the top of the hour, but we have two more questions, if that's okay. Okay. Someone asked, do you have any go-to resource recommendations? On what? I'm assuming for survey design, but I'd also be curious to ask the audience here as well, if you have really helpful resources, or also maybe specific packages that are helpful in the marketing space. So, I have not found a good survey design text. In fact, we even had a class, I had a class on it, marketing research, and I disagreed with the instructor, so that didn't really go so well. But for packages, the Glenette package out there, there is a great paper by Rob Titranian and Trevor Hastie on it, and explains how it's different from original SO. And I think, for me, I think I've read that paper five times, just because I want to know an inside and out. This was all standard, like, this is simple code, really. That's why I didn't want to get into a lot of coding. I mean, caret, Glenette, and understanding the DendExtend package is extremely helpful. I've sort of mastered that now. As far as being able to do it for anything, like I said, I do it, I use that Dendogram for text analytics all the time. And it's been a lot of fun, actually. So, the DendExtend package shows you how to color any which way you want, size it all up, flip it horizontal, left, right. And I find it extremely useful in text analytics visualizations when you're comparing groups. Those are probably the three best ones there. I wish I had better marketing survey information, and I think part of it is there's just not that much out there, which is why I've sort of created my own page on what I think makes a good survey. And I still clash with people in CX all the time, and they disagree with me. And, you know, that's fine. That's what we're professionals about. But if I'm going to analyze a survey and I get my say, that's what I ask for. I ask for 1 to 10s, and I ask for binaries.
Survey length and completion rates
Thank you. One last question was, so I guess as you're designing the survey, then, how are surveys designed to ensure they are actually completed? For example, how many questions are too much before someone just gives up? We found between 5 and 10, and closer to 5 is better. So, in this case, we got from 17 to 9, so we got a 50% reduction. In longer surveys, you're going to have to pay people. Like, that's just the way it is. The insurance industry, one, we had to, and I can't remember how much they paid for it. But, yeah, after 5, people will get, especially for a simple, like, how did we do kind of thing, you've only got maybe three questions before they're going to disappear on you. So, that's why I said make them count. Don't make them overlap at all. You're not trying to spot the person that's trying to game the system. You really want to get in a short period of time, how much information can you get from people.
Awesome. Thank you so much, Brian, for sharing this presentation with us and just answering all those questions as well. I don't see any other that have gone unanswered, but I also just want to open it up to anybody that, unmute themselves or raise your hand if there's anything we've missed. Thank you so much, Brian, for sharing this with us. I don't see any other that have gone unanswered, but I also just want to open it up to anybody that, unmute themselves or raise your hand if there's anything we've missed.