image: thumbnail.jpg
Ralph Asher & Laura Darby Rose | R in Supply Chain Management | RStudio
3:24 - Start of meetup
3:24 - Intro to Supply Chain Design - Ralph Asher
33:38 - Forecasting Demand with R - Laura Darby Rose
Intro to Supply Chain Design - Ralph Asher
Abstract:
COVID-19 has moved supply chain management from the back office to front-page news. And along with it, the discipline of supply chain design – the strategic evaluation of deciding where to locate manufacturing sites, warehouses, and other supply chain facilities – has gone from a little-known niche to a C-suite priority.
In this talk, I will introduce the field of supply chain design to the R community. Drawing upon my decade of experience in supply chain design and R, I will give a short example of how to design a warehouse network to support future customer need. This example will be drawn directly from my experience in the corporate world and my consulting business.
Speaker Bio:
Ralph Asher is the founder of Data Driven Supply Chain LLC, a Minnesota-based consulting firm that uses data science and AI to evaluate, design, and optimize supply chains. (www.datadrivensupplychain.com) Prior to founding Data Driven Supply Chain, Ralph worked as an Operations Research Scientist in corporate supply chain functions for 8 years at Target, designing e-commerce supply chain networks, and at General Mills, designing warehousing networks. Ralph has used R for supply chain analytics for a decade and can be reached at ralph@datadrivensupplychain.com or via LinkedIn.
Forecasting Demand with R - Laura Darby Rose
Abstract:
Over the course of about a year, Mallinckrodt Pharmaceuticals’ Specialty Generics division replaced an expensive SaaS (Software as a Service) with an R script and alternate forecasting process. Using an RStudio Enterprise solution, they have found a more flexible and cost-effective way to forecast demand and analyze data with R. This discussion will detail the process of replacing a SaaS with R, as well as challenges and next steps in the project.
Speaker Bio:
Laura Darby Rose is Manager of Demand at Mallinckrodt Pharmaceuticals, where she is responsible for statistical forecasting, forecast visualization, and forecast accuracy measurement for the Specialty Generics division. She has a M.A. in Economics from the University of Missouri-St. Louis, and enjoys using R and SQL for time-series analysis, creating Shiny apps, and data wrangling/cleaning.
Are you a data scientist or data analyst working in supply chain management? Are you interested in joining a group of fellow practitioners and taking a leadership role in developing and promoting open source solutions in your field? Join us!
At this meetup, we also proposed the formation of a community working group focused on developing and popularizing open source solutions for data scientists and analysts working in supply chain management. We'd love to start by creating a home – initially a website – which hosts resources for supply chain data scientists. More to come. Form to be part of the working group: rstd.io/supply-chain-community-org
Transcript#
This transcript was generated automatically and may contain errors.
Welcome to the RStudio Enterprise Community Meetup. I'm Rachel Dempsey. I'll be your host for today's R in Supply Chain Meetup. I'm joined by my colleague and co-host for today, Curtis Kephart, if you want to say hello as well, Curtis.
And we're also joined by our two awesome speakers for today, Laura Darby Rose and Ralph Asher. If you just joined now, feel free to introduce yourselves through the chat window as well and say hello, maybe include where you're calling in from. But just to go through a brief agenda while we wait for a few others from the waiting room, we'll go through some introductions of the meetup, have two awesome presentations, Intro to Supply Chain Design with Ralph Asher and Forecasting Demand with R with Laura Rose.
We'd like to also use today's meetup to kick off a Supply Chain Working Group, too. And Curtis Kephart from our team will give us some more information on that and how to join if this is something that's interesting to you. And then lastly, we'll open it up for additional audience Q&A and just open discussion among everyone. I did mute everybody upon entry right now, but towards the end, we will unmute. But if you do have questions during any of the presentations, we do have a Slido link that you can use. So you can ask anonymously or include your name there as well. And we'll share that in the Zoom window right now. That will really help us to organize questions, too, and you can upvote people's questions.
And just a quick note that the recording will be shared up to the RStudio YouTube as well. So if you want to share it with anyone or go back and listen later, that will be shared up there. One other housekeeping note, if you want to turn on live transcription during the meeting, you can do so as well. And that's offered in the Zoom bar below. And Curtis, if you could go to the next slide. Awesome. For anyone who's joining for the first time today, I'd love to welcome you to this meetup group. This is a friendly and open meetup environment for teams to share the work that they're doing within their organizations, teach lessons learned, network with others, and really just allow us all to learn from each other. So thank you all for making this a welcoming community. I really want to create a space where everybody can participate and we can hear from everyone. I want to reiterate that we love to hear from everyone, no matter your level of experience in the field as well.
If you ever have suggestions or general feedback or want to speak at a future meetup, I'll share a few links in the chat window right after this as well. But with all that, I would love to introduce our first speaker today, Ralph Ascher. Ralph is the founder of Data-Driven Supply Chain. And prior to founding Data-Driven Supply Chain, Ralph worked as an operations research scientist in corporate supply chain functions at Target and General Mills. And so with that, I will turn it over to you, Ralph.
Introduction and background
All right. Thank you, Rachel, for the introduction. Hello to everyone who's on this webinar, wherever you may be. Just reading through the chat, sounds like we have a global presence already, and that's great. I really appreciate the opportunity to speak with you today. As mentioned, my name is Ralph Ascher, and today I'll be talking about supply chain network design. What is it, first off, and how can we use open source languages like R and Python to design supply chains? So I'll share my screen, and hopefully this will go off without a hitch.
All right, so a little about myself. Like many on this call, I have a math and science background. I have a bachelor's in physics and a master's in operations research. I started my career after college in the military. I was on active duty in the Marine Corps for several years, and I've been serving as a reserve Marine in a part-time capacity for about 10 years since then. After the military, I worked in supply chain design and simulation for eight years at General Motors in Target here in beautiful, cold Minneapolis, Minnesota. I left the corporate world last summer to found my own consulting firm. If you're interested in learning more about that, please reach out. In addition to running my company and serving in the Marine Reserve, I'm also an adjunct at the University of Minnesota Business School, where I teach statistics. I've been an R user for 10 years, since 2012, and I really found the language to be an ever-increasingly versatile tool for my work. I've also loved maps and math and programming since I was a little kid, so I'm kind of fortunate that I've found a career field that lets me merge these interests.
Supply chains and COVID-19 disruption
So supply chains, from the back office to the front page. The COVID-19 pandemic has really caused unprecedented disruption to global supply chains. Manufacturing shutdowns as well as labor shortages and a dramatic shift in consumer spending from services to goods have caused massive disruptions to the previously invisible flow of materials. This disruption can be seen in the container ships waiting off the coast of Southern California, awaiting a dock in Los Angeles and Long Beach. It can be seen in the reconsideration of reshoring American manufacturing capacity or the delays in building and repairing homes due to supply shortages. It can be seen in the inflation that you see at the store shelves and at the gas pump. And even Santa Claus can't fix these problems. And he has the most advanced logistics operation known to mankind, or Elfkind. Instead, the task of ensuring supply chains continue to function, so we literally have food on our plate and fuel in our cars. That falls to the legions of hardworking supply chain management professionals. And as a data scientist, I'm really quite proud to be able to use my quantitative and technical skills to help these efforts.
Supply chain analytics framework
So you may be asking yourself, how can analytics be used in the management of supply chains? Depending upon your background, this slide may be old news to you, but I find it's useful to discuss supply chain analytics in the framework of descriptive, predictive, and prescriptive analytics. Now, descriptive analytics, the biggest and most important part of the pyramid includes business intelligence and diagnostic analytics of the supply chain. How many of my trucks arrived on time? How productive were my manufacturing plants in my warehouses? How are these metrics trending? And are any of them worrisome? Predictive analytics is next. How do we predict what's going to happen next in our supply chain? That includes things like sales forecasting, as we'll be discussing next, as well as forecasting costs for your supply chain inputs and some types of supply chain simulation. Finally, the top of the pyramid, which by volume should occupy the least amount of time, is prescriptive analytics. What should I do with my supply chain? This is my wheelhouse, and it includes things like supply chain network design, discussed today, as well as inventory optimization and other types of supply chain simulation.
Now, a key point for this audience, especially, is that open source technologies are really changing the landscape in supply chain management and supply chain analytics. When you look at each of these capabilities, you can mentally pick out which R or Python packages you could use for them. This really is a sea change from when I started in the supply chain analytics field about a decade ago, when the toolkit consisted of Excel for data analysis, SPSS for statistics, and commercial visualization software like Tableau.
What is supply chain network design?
So, what is supply chain network design? Put plainly, it's the strategic analysis of the locations and activities of the various elements of a firm's supply chain. For example, if I'm a manufacturer, where should I locate warehouses for my raw materials, the stuff I use to make other stuff, or my semi-finished goods, which are items that are only partway through the manufacturing process? Where should I put my manufacturing facilities, my plants? Where should they go? Where should I put my distribution centers, my warehouses for finished goods? Where should they be located? And if I'm a retailer, how should I use my stores as a link in my supply chain? For example, many leading retailers, including my former employer, Target, use their stores as strategic assets for shipping e-commerce orders. Well, from where should we do that? That's a big question. That's a supply chain network design question. Supply chain network design seeks to answer these big questions, all with the intent to understand the impacts on profitability, cost, service, and other key metrics, and with the goal to minimize or maximize one or two key metrics, usually cost or profitability.
Today, I'll be focusing on a relatively simple example of supply chain network design, a distribution network with two echelons, the distribution centers, or warehouses, and customers that are serviced by those warehouses. In designing a supply chain, we ask ourselves, given a customer base of demand aggregated to a city level, one, where should we open distribution centers to satisfy that demand? And two, what customers should be serviced by or aligned to each distribution center? While most real life supply chain design problems are much, much more complex than this example, I hope it will be informative on how practitioners approach these kinds of problems and how we can use open source technology for them.
Optimization modeling
So how do we do this? How can we answer these questions? We can answer them with optimization modeling. Optimization modeling is a mathematical technique that allows a practitioner to optimize a set of interconnected decisions where the decisions are constrained by real world limitations. Optimization solvers take the decision options formulated as a mathematical model and return the set of decisions that optimizes, minimizes, or maximizes an equation that includes the decisions as inputs. This equation is formally known as an objective function. If anyone on here has an operations research background, I realize it's not the precise mathematical definition, but I find it to be useful in explaining the field. So in our case, the decisions we need to make are one, how many distribution centers should open and where should we locate them? And two, what customers should align to each distribution center? You can see that these decisions, they're interconnected because I can't align a customer to a distribution center in a certain city unless I also open a distribution center in that city. And the limitations on our decisions or our constraints, they're one, that all customer demand must be met. We may have a minimum or maximum number of distribution centers. We may really only realistically open distribution centers in certain locations. So we have to start with kind of a candidate set of cities. We may have a maximum volume per distribution center. That's also called the capacity. And for certain reasons, we may put a maximum distance between a customer and its servicing distribution center. And we'll see that later in an example.
Heartland Widgets scenario
So we'll shortly transition to the Shiny app I built for the webinar. But first I'll introduce this scenario. Now, because this is a simplified scenario, this is a simplified version of an app I would use for a regular project. You are a supply chain data scientist at Heartland Widgets Incorporated. Your company is the market leader in widget sales across eight state area of the American Midwest. The pandemic-induced shift in consumer demand from services to goods, as well as stay-at-home orders, has led to a boom in demand for your company's flagship product, the at-home widget. Based with record demand, the Director of Supply Chain Network Planning has asked you to, quote, reimagine the company's distribution network. You are to recommend the where to open Heartland's distribution centers, as well as the customer to distribution center alignment, to achieve a minimal cost. Heartland Widgets has customers in 117 Midwestern cities, spread from the Dakotas in the west, all the way to my home state of Indiana in the east. Each city's demand is proportional to its population. So, for example, Chicago, Illinois, which is a big city, you may have heard of it, has more demand than Fargo, North Dakota. Heartland is considering 20 cities for potential distribution center sites. Because each distribution center will cost $1 million to open, it may not be cost effective to open distribution centers in all 20 cities, but we're keeping our options open.
So, let's get a little bit more technical here. The decisions we need to make for Heartland Widgets can be represented by variables. When we ask, what customer should we align to which distribution center, this is equivalent to asking a set of yes-no decisions that must be made in concert. For each customer city I, should I service it by a distribution center in city J, yes or no? In making our decisions, we're limited to having exactly one yes per customer city. It has to be serviced by exactly one distribution center. Relatedly, we need to make decisions about how many distribution centers to open and where to locate them. This is a series of if-then decisions related to our alignment decisions. If at least one customer is aligned to a distribution center in, for example, Omaha, Nebraska, then we must open a distribution center in Omaha. The decision sets are interconnected. We cannot separate our decisions on where to locate our distribution centers from our decisions on how to align customers to distribution centers. Mathematical optimization and, more specifically, the techniques of linear and integer programming are ideal for making these kinds of interrelated decisions.
Mathematical optimization and, more specifically, the techniques of linear and integer programming are ideal for making these kinds of interrelated decisions.
When we use mathematical optimization, including for supply chain design, we're trying to minimize or maximize some equation that is the function of our decisions. In this example, Heartland Widgets wants to design the supply chain with the lowest total cost. The supply chain costs in this example are, one, the opening costs for distribution centers, which are priced at $1 million each, two, the handling costs of the distribution centers, which is one cent per unit per widget, and three, the transportation costs from distribution centers to customers, which are five cents per unit mile. Now, a unit mile is moving one unit one mile. So, for example, if you have two widgets, two units, and you move them 100 miles, that's two times 100, 200 unit miles. At five cents each, that's $10 to move two units 100 miles. You add up these three components, cost components, and that leads to the total supply chain costs that we're trying to minimize.
Now, one of the tenets of supply chain design is that these cost components can be inversely related to each other. For example, the more distribution centers you have, the closer you are, on average, to your customers. And if you're closer to your customers, on average, you're going to spend less on getting the product to them on that transportation cost because it's just closer. However, the more distribution centers you have, the more you spend to open them in the first place. So, there's an inverse relationship between the distribution center opening cost and the transportation cost to get the product to your customer. The more distribution centers you have, the more it costs to open them, and the less you spend on the transportation, inverse relationship. Optimization modeling helps us figure out these relationships and help and find the optimal solution that takes into account the relationships between the costs.
R packages and tools used
All right. So, because this is an R Meetup and I'm going to demo a Shiny app, I want to go over the packages I used. This is a Tidyverse-centric solution for Dataprep, but I also use DT, Data Table, and Vroom. Shiny Dashboard is used for visualization, and I use our hands-on table to allow for interactive
adjustment to input tables. Leaflet is used as a mapping solution, and the core of the optimization modeling rests with the OMPR package and the open-source Symfony optimization solver. As an aside, the CoinRR Foundation, the organization that maintains Symfony and other open-source optimization tools like Pulp and Pyomo for you Python users, many in financial help, please consider donating via that GitHub link.
All right. From here, I'm going to work from the app, but I have slides for backup in case of technical difficulties. I host my app on shinyapps.io. You'll see the address, but to avoid having technical issues, I'd ask that, you know, nobody try to load it while I'm giving the demo. All right. So, this is a pretty sparse app because it's a fairly simple example. And first up is uploading our data. So, we really only have two sets of input data, our customer information and our distribution center information. I've prepped the data. So, I'm going to use Vroom here. And these are small files, but even if they were large, they'd still upload pretty quickly because we're using Vroom. Then saying yes, it's going to be used.
Reviewing the input data
All right. So, now I'm going to inspect this upload data and talk a little bit through it. First, our customer data. So, our customers are, as mentioned, 117 Midwestern cities. And for each of these, we have city, name, state, latitude and longitude, their demand, which is proportional to their population. And then we have this column called maximum miles to distribution center. You'll see later on in an example how we would use this essentially if you want to enforce a maximum distance from a customer to the distribution center that services it, you would set this column to that distance limit. Right now, I don't care about that. So, I set this number so insanely high that it doesn't really matter.
Finally, we have this column called include. And this is a nice thing about using our hands-on table because you can actually dynamically update the table that you use later on in your app. So, everything has a checkmark here. If I unchecked any of these, they would not be used in the optimization model. So, I'm going to display the customer map. You see these 117 cities across the Midwest. And just as an example of how our hands-on table works, I wanted to get rid of Rapid City, South Dakota. You see it's down here. Now, it's gone. But I do care about our Rapid City, so I'm going to keep it back. All right. So, that's how we're using our hands-on table for the customer inputs.
Distribution centers, similarly, we have 20 candidate cities across the Midwest. And to put our distribution centers in, we have a latitude, our longitude, our capacity, which is the maximum volume that it can be – have aligned to it. This is effectively infinite compared to our customer demand because it's a very, very high number. Each of them costs $1 million to run, and our handling cost is $0.01 per unit, as mentioned earlier. And this checkmark allow essentially says, do I want to consider this location? If I unchecked any of these, that location would just be taken out of consideration right away.
Running the optimization
So, finally, we have our – some inputs here before we run the optimization. So, I mentioned earlier that you can put a constraint around your minimum and maximum number of distribution centers. And so, I'm going to start off by saying I'm going to have a minimum of one – I need a warehouse somewhere – and a maximum of 20. Because there's only 20 options here, having 20 as your maximum is kind of like saying I'm willing to take any of them. The transportation cost per five – per unit mile is $0.05, as mentioned. And these are some more technical things I won't get into.
All right. So, when I click Run Optimization, what's happening now is my app is taking the customer information, the distribution center information, and the maximum and minimum number of distribution centers, and forming an optimization model on the backside, on the Shiny app server. It then finds the optimal solution, the minimal cost, and returns the outputs. Luckily, it worked as fast in this demo as it has in my practice.
All right. So, let's look and see what those optimal solutions are. So, to service all of our customer demand for Heartland widgets, the optimal lowest cost solution is four distribution centers, one in Omaha, Nebraska, one here in Minneapolis, one in Chicago, and one in my old hometown, Indianapolis. You can see the cities that are serviced by each distribution center are represented by their color, you know, red for Omaha and blue for Indianapolis, green for Chicago, yellow for Minneapolis. And they're kind of geographically separated, as we would kind of expect, because you generally want to have your customers closer to your distribution center.
This table is an alignment table, which is just the tabular version of the map that we're seeing above. So, for example, Ames, Iowa, which is over here, is serviced by the distribution center in Omaha, Nebraska. The farthest distance from a customer to its servicing distribution center is 415 miles, and that's from Rapid City, South Dakota, over here, all the way to Omaha. Take note of that, because we'll actually come back to that in a little bit.
So, what is the optimal cost here? The total supply chain cost that can be the minimal supply total supply chain cost in this scenario is $7.5 million over here. That includes $4 million in distribution center opening costs, $1 million per distribution center and four of them, $13,000 for handling at the distribution centers, and about $3.5 million in transportation costs. So, our total $4 million plus $13,000 plus $3.5 million equals $7.5 million, and that is our absolute lowest total cost.
The total supply chain cost that can be the minimal supply total supply chain cost in this scenario is $7.5 million over here. So, our total $4 million plus $13,000 plus $3.5 million equals $7.5 million, and that is our absolute lowest total cost.
Understanding constraints
Now, I'm going to make a small detour before we get into another scenario to explain more what we mean by constraints in supply chain network design. In optimization modeling, constraints are mathematical limitations on what set of decisions you can make that correspond to real life limitations. The more constraints you have in your model and in real life, the worse your objective function becomes. So, in this case, the more the supply chain cost would be. Optimization modeling, including supply chain network design, is an exercise in understanding the trade-offs involved in making constraints more or less restrictive or tighter illusion, which is the term of art. In designing a supply chain, the tighter the constraints, the higher the total cost becomes. Supply chain design is partly an art around understanding the additional expenses associated with tighter constraints.
Next, we'll talk about what a couple of those constraints may look like. So, first, we see that the lowest total supply chain cost occurs when we have four distribution centers in these cities up here. So, what would happen if we actually force our model to have exactly three distribution centers? How will that change our cost? How will that affect what our network looks like? This is actually quite straightforward to see. All we need to do is go back to our minimum and maximum number of distribution center constraints. So, I'm going to change the minimum to three and the maximum to three. Because these are integer values by forcing your minimum and maximum to both be three, you're going to end up with three distribution centers.
Now click run optimization. Again, and this optimization model is just running in the background with the additional constraint that it needs three, exactly three distribution centers. Then I go back to my results, it'll refresh. And now we see that instead of four distribution centers, we have the required three. And it's the same thing, except now Indianapolis is gone. Sorry to my Hoosier friends. And so, question then, what does this do to our overall cost? Because as I mentioned, the tighter you make your constraints, the worse your objective function becomes in this, meaning the more you're going to pay on in total.
So, let's look at these outputs. Our DC opening costs are now $3 million because we only have three distribution centers and they're $1 million a piece. Our handling cost is the same because it's the same for all distribution centers and we have the same amount of volume. But our transportation cost, which was $3.5 million before is now $4.6 million. So, while our distribution center opening costs went down $1 million, our transportation cost went up $1,100,000. And so, our total cost went from $7.5 million to $7.6 million. So, we see that this one change increases the cost. Now, $100,000 is a lot of money to a normal person. In the consideration of a base of $7.5 million, it's not a big difference, but it is an increase. So, that's what we see when we just force exactly three distribution centers.
Distance constraints
Next, we'll see what happens when we enforce a maximum distance constraint between our distribution centers and our customers. This is often the case with food supply chains because food is perishable and you don't want your food outside of a climate controlled warehouse for too long before it gets to a store. And so, oftentimes when designing food supply chains, you would put in a distance constraint.
This is actually pretty easy to do with our answer on table and using this maximum miles to DC column. I'm going to say a maximum of 400 miles between all of my customers and my distribution centers. As you may remember, the distance between Rapid City, South Dakota and the distribution center in Omaha, Nebraska was 415 miles.
So, by adding this additional constraint, that alignment is not possible. It's not going to be allowed. So, we just change that column.
I'll give this a few more seconds. If it doesn't work as quickly as I thought, I'll just go to my slideshow.
Instead of waiting that, we'll just go back to the slideshow here. So, what would happen if we did force a 400-mile constraint? Is that we would force a 400-mile constraint.
Okay. So, instead of waiting that, we'll just go back to the slideshow here. So, what would happen if we did force a 400-mile constraint? Is that instead of, you know, we still have four distribution centers as the optimal setting, but instead of a warehouse in Omaha, Nebraska, we would have it in Sioux Falls, South Dakota.
Why? Because that is the distribution center that allows Sioux Falls, which is a – or, sorry, Rapid City, South Dakota over here, which is a very remote customer, to be within 400 miles.
And the impact on the cost structure is that instead of $3.5 million in transportation costs, we now have $4 million in transportation costs. So, by adding that additional constraint around 400 miles from your distribution center to your customers, your total supply chain cost rose from $7.5 million to $8 million.
So, by adding that additional constraint around 400 miles from your distribution center to your customers, your total supply chain cost rose from $7.5 million to $8 million.
Cost curve across network sizes
Let's say we actually wanted to see the overall total cost at different number of distribution centers. I went and ran this model, forcing it to have exactly one distribution center, exactly two distribution centers, three distribution centers, all the way to nine. I then recorded the cost components between the handling cost, the opening cost, and the transportation cost.
The handling cost in green is flat, no matter how many distribution centers you have, because it's the same volume, just split in different ways. This blue line is our distribution center opening cost, which is linear, because with the more distribution centers, you're spending another million dollars.
Your transportation cost in red goes down, because as I mentioned, the more distribution centers you have, the closer you are on average to your customers, so your transportation cost goes down. It goes down dramatically from one to two distribution centers, but then kind of flattens out, so there's not as much of a benefit.
Your total supply chain cost, which is this purple line, is kind of a parabola. The minimal cost occurs at four distribution centers, as we saw before, but between, for example, two and five, there's not a tremendous amount of difference.
Closing remarks
All right, well, that's most of the content, folks, so I really appreciate your time today. I just wanted to wrap up by saying and emphasizing that supply chain management really was a behind-the-scenes corporate function until the beginning of the COVID pandemic, but now it's front-page news. We all see it.
And analytics powered by open-source programming languages, it really is changing the nature of this field. Supply chain network design, it's kind of an obscure part of supply chain analytics, but I find it to be a powerful tool in devising and evaluating the organization's supply chain strategy, and hopefully today you just got a taste of what that might look like.
And thank you for listening. If you'd like to know more, please reach out.
Q&A
Awesome. Thank you so much, Ralph. And I just want to – that was a great presentation. I just want to reiterate to everyone, I see a bunch of questions coming in through the Zoom chat as well. If you could use the Slido, that would be super helpful, so we can kind of help see the questions versus comments in the chat.
But one I see was from Jerry. That is, how large of a problem can the open-source solvers handle from your experience? How often do you encounter models that would require a commercial tool?
I've ran some pretty large optimization models on open-source solvers. And if you have the right hardware, and you're kind of paying for it in your time or you're paying for it in your dollars, and so if you're willing to wait an hour longer than you would otherwise, it's well worth it. I haven't, both in my consulting and in my corporate work, I've not encountered a size of a problem where an open-source solver couldn't handle it, just like wouldn't compute.
Yeah, yeah. And so you can kind of shape your objective function to reflect whatever combined metric you may care about. So in this case, the objective function that you saw was purely a cost function. But I've in the past done things like, well, here's my cost. And here's, for example, a customer service metric. That is also a function of my decision variables. And you can weight them. Now, when you see the output, it's a number, it doesn't really reflect like, you know, dollar bills on the table or anything like that. But it can kind of, and the more heavily you weight cost versus customer success or customer satisfaction, that combined metric can lean towards one way or the other.
Thank you. And I know a few people had asked this and were curious about the Shiny code that you use. And if you would feel comfortable sharing your slides as well as maybe some of the Shiny code? I'd be comfortable sharing my slides. Unfortunately, not the code. That's kind of a trade secret thing.
And one last question. And then again, we'll cover additional questions at the end as well. I'm just excited to get to Laura's talk too. But the next question was, how did your optimization run so fast? Are you using parallel packages? I mean, it's partially running so fast because it's a small example problem. But no, I'm not doing any parallelization or anything. It's just running on my on whatever machines getting spun up on Shiny apps.
Introduction to forecasting demand with R
So a little bit about my background. I'm a demand manager, as Rachel mentioned. I work in the supply chain department. And in Mallinckrodt, we have a couple of different divisions. So I work for the specialty generics division. And we make basically generics that you might be prescribed when you go to the doctor or your nurse practitioner or whoever you might go to see. So a lot of pain management, acetaminophen, other things as well. So my main job responsibilities are stat forecasting, and then measuring forecast accuracy. So not just of the stat forecast, but also our commercial partners provide a forecast based on kind of market intelligence info that they've collected. They do not do stat modeling, as far as I can tell, but they do submit a forecast.
And then I wanted to add a caveat here that I only had about five months of experience regularly programming in R when I started the project. And I share that with you all to hopefully encourage people who are just getting started in R or Python and, you know, thinking about doing a project that will really involve R or Python. If you know enough to write a script, you probably know enough to do a project that will really impact your company. And I guess I didn't put this on the slide, but I'll mention it since I know Ralph mentioned academic background. I am an economist by education. I've worked most of my career in supply chain and really did a little bit of economic consulting, but found that I really enjoy supply chain. So here I am.
If you know enough to write a script, you probably know enough to do a project that will really impact your company.
How the project began
Okay, so how the project began. So imagine it's spring 2020. Actually, you don't have to imagine, right? It's probably all still fresh in our minds. And you're staying home, being a good citizen, and you're a little bored. So what did I do? Created a little Shiny app, a simple Shiny app for time series forecasting.
And shout out to the R-Ladies organization. I actually got the inspiration for the app from R-Ladies St. Louis meetup. And so I used materials that the presenter had provided to kind of dig into Shiny. I'd taken a few online tutorials in Shiny, but never really done much with it. So, you know, I'm working remotely and I want to impress to my manager that, hey, I'm being productive. So in one of our calls, I showed him the app and he was impressed with it. And he thought, well, maybe we can replace the SAS that we were currently using for staff forecasting with this app or something like it.
So the SAS that we were using costs about $200,000 per year, which for my company is not small change. We were definitely in a cost savings mode. It was also generating negative number demand forecasts. So keep in mind that that's what do you even do with that, right? We're not going to sell a negative number. The IT support for this company could not figure out what was going on. So it was costing us a lot of time and effort to basically modify these forecasts.
So after my manager's initial approval, I collected some demand data. We chose about 30 SKUs that we thought were representative of the business. And I did a rolling forecast accuracy analysis. We wanted to have an idea before we proceeded further, what kind of accuracy we could expect if we switch to the R forecast package. That was what I was using at the time. I'll go more into that a little bit later. And specifically, the ETS function, which is the exponential smoothing, commonly used in univariate supply chain forecasting, also the ARIMA function. And there's options there for just best fit algorithm kind of a thing.
So the ETS function was actually comparable to what was used or what we had enabled in the SAS. Various exponential smoothing models were selected by the demand forecaster. So I ran this rolling forecast accuracy analysis. And to give you an idea of how perhaps not efficient I was at the time, and I'm sure I'll be saying the same thing about myself in a couple years today. But I had three for loops that I used to do this. And anybody who's programmed in R for a while knows that's generally not the best strategy. But it did work. And we were satisfied that, hey, just doing like setting the ETS algorithm to best, you know, best fit, minimizing the AICC. That's the Akai Kei Information Criterion, the corrected one. That was comparable to what we were getting in terms of forecast accuracy from our SAS. Also, the benefit was, hey, we didn't need manual intervention from the demand analyst or manager. So we made the decision to move forward with this project. And then we had to get the buy-in from the IT group at Malincroft, not the, of course, the IT support at the SAS company, since it was in their budget.
Setting up the R infrastructure
So up until then, I had written some other scripts that I used for various data wrangling tasks, the kind of stuff you do in supply chain analysis, right, or demand planning, which is my specific area of expertise. And we didn't want to just have me running scripts on RStudio Desktop, as useful as the free version of RStudio is. We wanted more enterprise-capable options. So we looked into RStudio Server Pro, and I believe the most recent version of that is called Workbench. I'm not super up-to-date on that. Then we also contracted with Lander Analytics, and they helped with server setup and maintenance and then application installation. So it is a Linux server, for those of you who might be more Windows people. And at Malincroft, we're more of a Windows shop, is what I've been told by our IT group, so definitely needed a little bit more guidance in setting that up. And we had both a production and dev server that were supported with them.
Okay, so we're moving forward in this project. We made a decision to go forward. The IT analyst who was responsible for maintaining the data used for SAS forecasting needed to collaborate with me to replace the data management. So not the forecasting, but those of you who might have used the SAS at your company for forecasting or other things, note that the SAS can sometimes do functions that are not just the main function that you may have purchased it for. So we needed to do in-house solutions, especially generics, it's kind of old school in a way. So we have like an AS400 system. So we needed to create a bunch of tables, make sure we were maintaining those tables, jobs were running, that kind of thing.
So we had tables for both cleanse history and non-cleanse history. So cleanse being removing outliers, sometimes you might decide to not include all history for consideration in the stat forecast model, because there might have been a structural change, something like that. So we wanted to move away from Excel spreadsheets. There was definitely a time and a place for spreadsheets, but we just didn't feel that was in the best interest of the business. So we used the DBI package, which probably many of you are familiar with, and then the ODBC package, lowercase ODBC, I think there's a uppercase R ODBC package. And just using some basic SQL, we got those set up, and it was very easy to connect to the AS400 system, pull in those tables that I would need to estimate models and calculate forecasts.
We also changed our outlier cleansing process, as I alluded to in the second point, and that is because we had previously cleansed the history directly in the SAS application. So we just moved to a pretty simple process. We were trying to come up with something in the limited time we had. And we had an Excel upload. If you needed to remove an outlier, you'd upload an Excel file to our AS400 system. So probably could have been, well, we're changing the process actually now, and I'll get more to that in the future. But, you know, it worked in the initial stages, it worked, and sometimes you have to go with that.
Broader uses of RStudio Server Pro
Okay, so we were able to use RStudio Server Pro for other uses besides stat forecasting. As I alluded to earlier in my presentation, I had written other R scripts to do a lot of data analysis and wrangling tasks, the kind of things that you don't want to spend a lot of time on if you can. Measuring weight of MAPE. Weight of MAPE is what we use to measure our forecast accuracy. There are other metrics that your organization might use. Checking the commercial forecast upload versus the download. I am responsible for loading the commercial partners forecast to the ERP system, and depending on how things go, we might have kind of phantom forecasts hanging out in the ERP system.
So we were able to do that in just doing spreadsheet analysis, but it was fairly simple to write a program that did it in R. Also, last year with the – well, I said last year. I guess it's two years ago almost now. We had a lot of orders for acetaminophen with the start of the pandemic, and our product director needed some help in understanding how to allocate orders versus forecasts. And based on the forecast the customer provided, they were either going to get product or have to kind of go to the end of the line, so to speak. And there were some other ones. I won't – for the sake of time, I won't bore you guys with the details of those.
So it was very – it was simple to set these jobs up to run on the server using Cron R, and that's – as you would expect by the name, it's an R package which interfaces with Cron. So the scripts take a lot of time, all these jobs set up. We could compile a lot of data reports, no intervention, or maybe a little bit of just give it a check from demand managers. And then the real test is when I was out on short-term disability this past summer. The jobs ran well. I saved my coworker a lot of time having to cover my responsibility as well as his.
So my coworker left the company in November, so I've since been taking over his former duties. And I've been writing various R scripts to kind of hopefully improve some of the processes, and it really has made it possible. Scheduling these jobs on the RStudio Server Pro interface has really made it possible for me to do my job as well as his previous duties.
Project summary and results
Okay, to summarize the project, you're probably wondering how long did all this take. It took about 11 months. And I will say this, you could probably do this kind of project where you would totally replace the software you've been using in less time. We happened to have a lot of time based on when the contract with the SAS was going to be up, and so we had several months in early 2021 where we were running both concurrently.
There are definitely improvements to be made to the forecasting processes, and I can tell you that as well as anybody at the company. That's what I deal with day in, day out. But the initial goal of replacing the SAS with a solution at least as effective was completed. So we consider that the initial stage to be a success.
We measure on the lag three forecast accuracy, which basically takes into consideration. Essentially, it's sort of like lag four. You think about it takes history from four months ago when you're measuring the particular month in question. But it's been about the same, depending on the month, a little bit worse, a few percentage points. But we did meet our goal for 2021. So my boss is happy. I'm happy.
And the cost savings is definitely worth it for a slight trade off in accuracy. And we are still using pretty simple models like, you know, Arima, exponential smoothing models. So we hope that as we introduce new models, we will improve accuracy over time.
Tidyverts packages and forecasting tools
And then I alluded to this earlier in the presentation. So I started out with the forecast package in summer 2020. And then I found out, I guess it was probably old news, but it was new news to me at that time about the tidyverts suite of packages, which is kind of the replacement for the forecast package.
And so I switched to that. I switched over the code that I had written to that and took a little bit to kind of make sure I understood everything. But it's a great, great suite of packages that includes Fable, Feast, Sybil, Sybil data, I think. So I have found those to be really useful for automatic forecasting. If you have a lot of skews and you don't have time to go in and tweak parameters, figure out the best Arima model, figure out the best exponential smoothing model, for instance, like univariate forecasting.
The models they produce, the forecast for supply chain are stable forecasts. They don't require a lot of intervention. You're not going to get anything unbelievable or crazy, I guess.
Shiny app development
Okay, so I mentioned the Shiny app earlier. So the impetus of the project started with a simple time series forecasting app. We made the decision to forecast the script scheduled as a monthly job instead of an app. And part of the reason for that was it seemed simpler at the time, time savings. Also, it ended up working out, we weren't really ready to go with RStudio Connect at the time, which we would have needed to host the app for an enterprise, in an enterprise capable way.
So after that, I developed another Shiny app, kind of based on the first one, which is more, uses a number of different packages, the tidyverts packages, of course, DT, I think Ralph mentioned that in his app, to edit history, you can change parameters, basically interfaces with our A400 system via SQL.
So you can, through the app, you can interactively edit our data in our company's systems. And this allows me to, especially for those high volume, really important SKUs, to go in and pick out specifications I think are optimal. So we're hoping that this will give us more opportunity to kind of tune a better model for the SKUs that we consider to be, you know, A SKUs important.
This, the impetus for creating this app was actually my former coworker. And he wanted something that mimics some of the functionality that we had in our previous app. So he liked being able to go in, play with parameters, kind of test things out, see how you change the history, how does this change the forecast, that kind of thing, look at different accuracy metrics as he was creating his forecast.
Future state and planned improvements
So future state, it's always good to be thinking about continuous improvement, right? So I'm working on another Shiny app to visualize our generics data, that's our finished dosage data, with a pivot table, probably using the pivot tabler package, not sure yet, I might look into a few other packages.
And then we either look at monthly or weekly doses or bottles, depending on what the user would want. So I currently use this very slow tool, using a legacy system I will not name. And we use this for creating visualizations for the demand review slide deck, where we look at our customer demand, compare that to our, look at our forecast, see what's changed, look at our financials, that kind of a thing.
So this will help me more than anybody else, but I'm definitely interested in improving my processes. And then another project we started last month is looking at the commercial team forecast. So they currently don't use much statistical forecasting. So they are interested, though, in looking at their end purchaser demand and how we can roll that up under distributor customers.
Mallinckrodt is a manufacturer, so we sell to some big time distributor customers, you may have heard of in the news. And there's certain customers that are pricing contracts. And then those distributor customers sell to, you know, more of retail pharmacies kind of a thing. So we're hoping to get a better picture of our demand to kind of tie that out and build a better forecast for them in that way.
And then in terms of model improvement, those of you who are familiar with the tidyverts packages probably know that you can do hierarchical forecasting as well. We did explore that a little bit in the initial stages of the project, excuse me, when we were trying to understand what we wanted, how much we wanted to experiment with different kinds of models.
My boss thought it wasn't a good idea at the time. But now that we've completed the initial stage, we'll probably look into that a little bit more. And then also machine learning methods for time series. We would need to do that on weekly data. We don't have enough monthly data really to make it worth it. But model time is a good package. So that is something that probably this year we are looking into exploring.
Lessons learned
So to summarize, I just, a lot of this on this slide is common sense, but I think it's probably worth reiterating. You're interested in doing like a project where you replace a, you know, a SAS with an R solution, start sooner versus later.
Something that we learned sometimes the hard way is that the SAS will perform other functions besides forecasting. So you need to come up with solutions for data storage and management. And those solutions might not be R specific, they need to be compatible with, you know, whatever you're doing in R for forecasting.
So you might need to create a database, a data warehouse, maybe you already have something like that in your company or your organization. Also, project management 101, I guess, make sure you've an estimate of the time involved on your part, as well as many collaborators, because you don't want to come up to the end of the project and figure out that they don't have enough time to give you to complete the project.
And then I wish somebody had told me this ahead of time, learning some project management skills is helpful. Probably a lot of us on this call have good technical skills. When I was trying to map out this process, which is my subsequent point, I had really no idea what I was doing. And it got done, but it could have been a lot more efficient.
And as you probably expect, we learned the hard way. My coworker used the SAS for something I did not exactly know what he was doing. And I think there was a miscommunication between perhaps an IT analyst and him, not throwing blame, but we realized, oh, we never accounted for this.
Now we're practically finished with this project. We need to make sure we have something that will allow him to do what he did with the SAS. So you don't want to have our experience there. And then I'll finish. Hopefully this is not too pessimistic of a note, but with a paraphrasing of Murphy's Law, if something can go wrong, it will. So plan for all the contingencies. And since we're all in supply chain planning, that shouldn't be too hard.
With a paraphrasing of Murphy's Law, if something can go wrong, it will. So plan for all the contingencies. And since we're all in supply chain planning, that shouldn't be too hard.
Okay, that's all I've got. Questions? I guess, Rachel, I don't know how we're going to...
Q&A
Yeah, I can help with reading this. But thank you so much, Laura. That was an awesome presentation. Really appreciate it. I see a lot of great comments in the chat as well. And just want to remind people, if you could use the Slido link to ask questions, that is super helpful for helping us organize them.
Oh, sure. Sure. DT. But I know, and Ralph mentioned this in his presentation, R hands-on tables, another good one. And there are other ones out there. I was familiar with DT, and I already gone down that path. So I was just like, I'm sticking with it. If I was building it from scratch, maybe I'd do R hands-on table. But yeah, there are several good packages out there.
Cool. Thank you. Another question from Slido was, what type of data are you forecasting on? Invoice sales, demand, POS?
Demand data. So we have two divisions. Yeah, invoice sales. Yeah, it's by request date. So it's not shipment. It's supposed to be representation of what our true demand is. So not taking into account any supply issues.
So we sell business to business. So we supply a lot of the pharmaceutical companies you've probably heard of in the news. And that's all I can say about that. And then we also supply some distributor pharmacies. And that's our dosage product. So the kind of stuff that you might get prescribed for pain management or other conditions.
Thank you. But Laura, one question was, if you had unlimited budget, would you prefer one modeling in R and then passing results to Tableau or two modeling and displaying in R and Shiny?
So R and Shiny is a clear winner for me. So I have used a commonly – I can't name names, but a commonly used data visualization tool, which is kind of a click and drag and everything. And I just don't find it. I like to know what's going on behind the scenes.
And when I'm writing that Shiny code, I know how things fit together. But when I'm doing – if I was using more in Tableau or another tool, I wouldn't – and I know there's beautiful visualizations that can be done in Tableau. But I wouldn't have the same sense of control. So that's my personal preference. And I'm not saying it's the only – it's the right one.
Thanks, Laura. And I'm going to try and also pull people into the discussion and allow people to unmute yourself. I know this is dangerous with 300 people. But Eduardo, I saw you had a great question that you put into the Slido as well. And I was wondering if you'd want to read that one out loud and add some context there.
And I can start by reading the question too. But it was, Laura, did you have to deal with uploading back the forecast to ERP? If that was the case, were there any guidelines you had to follow with IT?
So you're talking in the stat forecast, I assume? I mean, I know I mentioned doing a check of an upload. So I'm assuming you're going with a stat forecast here because that's what we – the models estimated and calculated are.
So I actually worked – the nice thing about designing a system on your own – again, maybe I'm a control freak, but I hope not – is that I had a chance to work with the IT analyst. And he and I have a very good working relationship to say, okay, this is the kind of table we want to build. I need this field. I need this field. I need this field.
So we kind of had a custom solution built. I think within my – we use – within the service account, which we use on RStudio, and that's so it's not tied to my personal account so anybody can run if I'm on short-term disability or whatever.
There – I think we had to have, like, permissions set so that anybody could – you could write with that account to the table. That's a pretty simple fix. So, no, they were basically like, what do you need from IT? And I was like, here's what I want. Here's what I want. And they said, okay.
Great. Thanks, Laura. One other question that came through on Slido was, how do you source snapshots of your forecast for later accuracy analysis?
Okay, yeah. I kind of did allude to that, I guess. So we are – we – every – this is something that IT team handles. Well, I guess I could theoretically do this in R. But at the end of the month, before the table that receives the forecast is cleared, that forecast, so all 67,000 now lines of it or whatever, we forecast three years out.
That's just – nothing magic about three years, just how we do it. It's copied to another table that has – I think we store 13 months of historical forecast. And then when I measure statistical MAPE, the script pulls in from that historical table and it filters. I use some of the DDI interfaces.
And there's – in the parameter argument, you can say, I want to pull in based on this criteria. So you pull in whatever might be your lag three forecast or lag one. So that's how we measure it. But yeah, we do track our stat forecast accuracy pretty carefully.
Yeah. So for optimization, the biggest ones I use is the OMPR, which stands for, I think, Optimization Modeling Process in R. I'm not sure about the P, but – and essentially, it's a pretty intuitive way to take linear program or integer program optimization models and formulate them in R. And that's kind of modeling language package. And then there's some adjacent packages that actually then send it to the solver of your choice. And so if you go on the packages GitHub site, it's very easy to read. But I like that because it's built with a tidy versus centric mindset. So for example, you use pipes to add components of your model. And you can do whatever code you need to do to prep your input tables and then just send it into your optimization model, which is how I'm able to kind of dynamically change the inputs and then reoptimize on the back end. So that's primarily what I use.
That's a good question. I'm trying to think. Probably the first thing is I should become clean that I probably don't update my packages as often as I should. I am actually the only user of R in my division. I'm not sure about we have specialty brands, too. So in terms of managing across the organization, it hasn't been an issue at this point. If we do bring on another analyst or we have a current person who learns R and kind of decides to do some stuff with it, that'll probably become more of an issue. But I do use – isn't quite exactly your question. Maybe it's helpful. I use R3 – no, I feel bad admitting this, but 3.6.3 for some of my scripts. I know I'm really old.
The supply chain open source working group
Yeah, thank you. Thank you, everyone, for presenting, and we'll get back to the questions in a moment. But we also did want to take an opportunity with this group. There's obviously a lot of interest in combining open source software plus to solve the problems that folks in supply chain management have. The kind of too-long, didn't-read-or-didn't-listen version of what I'm about to say is if you consider yourself kind of one of two people, either keenly interested in kind of community organizing for supply chain management or keenly interested in contributing resources to the community, whether it be R packages or articles, kind of sharing best practices that other people could benefit from, I really encourage you to have a look at the link that I just shared.
It's an rstv.io link. Have a read over the kind of proposal we present there and, you know, consider participating and, you know, submitting your name. The longer version is, right, there's a lot of interest in supply chain management plus open source software. And so, you know, Rachel, Ralph, and I had talked a couple months ago now, actually, about kind of how to take this community to the next level. There's certainly a lot of industry groups as well as, you know, field groups that have, you know, set kind of templates that communities could follow.
So, you know, I have a lot of colleagues in R and Pharma, for example, and they have a nice conference that brings everyone together and shares resources. A lot of groups create shared packages with kind of common solutions. There's a lot of ways to approach it. You know, we propose a working group to kind of create a community hub for you folks where you could share resources and share best practices that you would all then be able to benefit from and develop together. So again, if you kind of consider yourself one of those two groups, I would really encourage you to have a look at that link and consider submitting your name. What we do after that is we'll follow up shortly after to kind of bring those folks together and continue the conversation about how to take this community to the next level.
Q&A
No, I didn't. And that's because it was less work for pretty much everybody. Yeah. So if you can make people's lives simpler, it's very easy to get by. And if you can convince them to do that. I was recently on a call with the director of IT, as well as my boss, who's the senior director of supply chain. And the director of IT was like, yeah, it's saving us $200,000 a year. It's less work for the IT analysts. And he was like, yeah, it's been great. So I think my old coworker, he, I think, liked having not to do as much work. Because I also had some R scripts that did his mate processing and other things for him, helping check his forecast upload, that kind of a thing. So make people's lives simpler, and they will be happy to change, in my experience.
So make people's lives simpler, and they will be happy to change, in my experience.
I did get a raise, yes, and a promotion.
Yeah, it's a great question. So this is kind of where the art and science come together. And so there's a couple ways. The way I primarily like to use is essentially use simulation to change the inputs. So for example, if the demand for Ames, Iowa was not this number, but rather we think it's from here to here, you kind of do a sampling of each of those, you run the optimization. And overall, you do this a bunch of times to kind of compare the outputs and see how different really are the recommendations. Like if it's always recommending these four warehouses, those four warehouses are a good choice. If there's radically different outputs, then you kind of got to dig a little bit deeper.
Sure, yeah. So compliant, I'm not sure I entirely understand the question. If you are able to get a little more color and put that in the chat or somewhere else, perhaps Rachel can give me some insight into that. Because we use statistical forecasting, we were using it, I was going to say, as more of a benchmark for our commercial partners. There wasn't a lot of compliance present out pharma, especially Opiates, which is one of the products that Mallinckrodt makes. It's very highly regulated, as you know. But the commercial teams, their forecast has to submit a lot more customer-level evidence to the DEA for quota. For statistical forecasts, we're basically saying, what does our demand say? So there wasn't as much of a compliance issue. And my company, I will say, for being a fairly large organization, people are pretty agile about, hey, if something works and we like it, we're going with it, as long as we're not obviously breaking any laws or circumventing regulations.
Do we have any more compliance with IT? I see someone says compliance with IT standards. Yeah, I mean, they were – I guess maybe – me, I'm very fortunate to have a really chill IT team. But they were like, yeah, let's go for it. It sounds good, as long as they have the time to do it, which they did. So – and then I see someone else say, compliance with my financial division will kill me if I do that. Well, I mean, I go back to my previous comment. If you can simplify their lives, maybe they would reconsider. So hopefully that's helpful.
Thanks, Laura. And I know I mentioned this before, but if you have a question and you want to – if you have a question, please feel free to reach out to me. I'll be happy to answer it. And I know I mentioned this before, but if you have a question and you want to ask it out loud, too, feel free to raise your hand on Zoom or even just unmute yourself and jump in. I know it's very hard to recreate that in-person feel virtually, but would love to hear from you all.
I'm happy to take this. Ralph, if you want to chime in, I feel like I've been talking a lot, but I'm more – I have less experience than you, so maybe it's more recent in my memory. Yeah, I'll – okay, I'll give my – I'm sure Ralph will have some interesting things to add.
I would say you want to be at the level where you can at least understand how functions work. I mean, Tidyverse is really kind of – I know people like Data Table, too. I personally am not familiar with it. But Tidyverse is kind of an essential, I would say, for supply chain. You're doing all this data analysis and manipulation, wrangling, all that kind of stuff. You should be able to comfortably write at least a fairly – a simple R script. And you do learn as you go along. Sometimes the best way to learn is to actually pursue something. I think once you have a deadline, that's where it all becomes real. So make sure that you've got something working before you give the cancel notice to your SAS or whatever else you're going to be doing that's going to be – involve money on the line, I guess. So that's my advice. But yeah, I mean, if you can write a simple R script, you're probably ready to start some kind of project. Go for it.
Yeah, I concur. I mean, the vast majority of work in supply chain is data prep and data analysis. So if you can start with kind of a Tidyverse education, that's probably best. And then just learning functional programming from there is good.
Judgmental forecasting really wouldn't involve any statistical modeling, per se. You kind of just say, I think this is best. And that's kind of somewhat of what our commercial partners do. I mean, I think they do some trending and things like that. I have not quite gotten a super clear explanation, which I will probably be learning more about that as I go into this project with them.
Delphi method, if my memory serves me accurately – and someone, feel free to jump in on the chat if I got this wrong – is kind of a consensus panel of experts. And that – I mean, that would probably more not fall under this kind of project. And there might be some products for which you want to do that. But that's – you know, at the mid-level manager, high-level analyst, data scientist role, that's probably not what you would be doing as much. Unless you – maybe you could put a variable in the model that – you know, I'm just brainstorming here that would account for that. But I don't – I don't – I'm not involved in that in my company.
I see Todd has asked a question on Slido that was, is the distance calculation line-of-sight or road network-based? What options are available to the solution? I think that might be for you, Ralph.
Yeah. Yeah, so it is line-of-sight, just kind of as the crow flies for speed. You can certainly do road distance calculation and use it. It's just a distance matrix input. And so, like, for example, you can use – there's some open-source options. For my work, I use Google Maps API because that's kind of the gold standard of quality and it's quite easy to use.
But yeah, you can do it either way in this particular demo. It's just straight line.
In generics, we cannot do promotion. So, we can't say, hey, we got a hydro promotion going on. Obviously, the EA would not like us if we did that. So, in that kind of industry, like a consumer packaged goods perhaps or something, you could – if you had a forecast when it was going to happen and then maybe a past time series of that, you would want to put that as an explanatory variable in your model.
I see one from Yoni. How do you describe your client's counterparts? What factors are driving your optimization forecasting models? This one might be for both Ralph and I.
Let's see. My counterpart – well, it's just me and demand planning now. So, my boss and I, he's got a pretty good technical understanding of things. The big concern at my company – and this is not every company – is forecast accuracy.
So, if the accuracy is good, if I could be doing the most complicated machine learning model that anybody ever thought of, it wouldn't make that much of a difference. But I do realize that that's not the same for everybody else. So, Ralph, you probably have a couple things to add about that or adjust about that.
Yeah. I mean, it's largely like saying, okay, here's how our model solving is happening. Like, what's underneath the hood in the business sense? Here's the three cost components. Here's how you might see the tradeoffs.
Sometimes it gets a little bit tricky because sometimes you see outputs that, oh, that doesn't make sense. Well, it actually does once you take into account all the constraints and things like that. But you just need to hopefully build the credibility first that they can take the hand wave explanation at that point.
Thanks, Ralph. I know we're getting to 30 minutes past the hour, so we have a few more questions here. And I know a lot of people have to drop, but I do want to say if anybody wants to stay on and just kind of chat with each other, we can stay on past the time, too.
Maybe pretend like we're all in person. But one of the questions was, how have you dealt with time series model forecasting when there has been such a large change in values, for example, the price of things?
So that's a really good question. I would say if you have pricing data, you could put that – like price elasticity of demand, you could put that as an explanatory variable in your model. I was dealing with univariate here. And univariate is not always the best choice if you have things that correlate well.
In my company, we have pricing contracts. Our price actually does not change very much, and I think our commercial team is not particularly – they don't like to change price very much, which means if you don't have enough X variation in your data, it's not going to be a good – you don't want to – you wouldn't include it in your model.
So let's see. I think that mostly addressed that. But yeah, if you – I mean, the moral of the story is I think I probably – maybe I sound like a broken record here. If you can do a regression or some other type of model where you have – you're bringing other factors, do it because you're probably going to get some more information there if you find, of course, good correlation between the outcome and the predictors.
Optimization algorithm and Q&A
Thanks, Laura. Ralph, I see you had answered this on Slido, but someone really wants to know this, and I see a few questions coming in with this. Which optimization algorithm did you use in your app?
So it's a mixed integer linear program for those who are familiar with operations research terminology. Essentially, you just – the OMPR package builds the model based upon your parameters and then sends it to a solver, and solvers – a lot of people – dedicated people work in those, but essentially, it uses a lot of different approaches to solving the problem and picks the best one essentially.
Great. Thank you. Mohit, I apologize if I'm pronouncing your name wrong. I see you had asked a question – or raised your hand as well, if you want to jump in.
Yes. So my question was regarding the correlation factor – correlation analysis, which Laura was speaking about. Like, if the price is correlated with the demand, like, you just put it as an external variable. So does that require any analysis beforehand? You put in those external variables, or you could just dump all the variables you're thinking which might be correlated and let the – just the algorithm decide which ones are best?
And – so what is the correct approach, as per your experience? I just wanted to ask that.
Great question. I thought I could teach a class on that. And it's funny you bring this up, because I see someone, Federica – hey, Federica, I think she's still on – from Book Club ISLR that I'm in, where they talk about different models – or different ways to select your best – you know, the variables you want to include.
I mean, a correlation matrix is a good – you know, for a start. Obviously, if you see something that's not highly correlated, you might want to consider dropping it. Academics versus business people approach things a little bit differently in terms of how they would think about a model and report that. I'm assuming your problem might be business. I don't know.
But, yeah, you could – there's different methods. Maybe I should say join the ISLR book if you want to find out some more about subset selection. But correlation is a good way to get started. And then think about does it have any theoretical basis, right? If it does, then maybe you should include it. At least try it out in the beginning. But without seeing data, it's kind of hard for me to give a more in-depth answer. So hopefully that helps you out.
Connecting after the meetup
Thank you so much, Laura. And I see there is also another question that is, what is the best way to connect with both of you, Laura and Ralph, after the meeting if people wanted to ask follow-up questions or connect? Is it LinkedIn?
LinkedIn is fine. I think a lot of us get a lot of spam in our inbox, but I try to respond to messages that don't look too spammy. I think my email might be on my LinkedIn. But, yeah, you're welcome to email me. I mean I'll try to respond. I'm actually pretty good about checking emails.
But, yeah, feel free to discuss. Isn't there the chat supply chain on the R for Data Science Slack too if you're interested? That's a good – I mean it's not super active, but there's no reason why it couldn't be more active.
Definitely. I was just going to mention that too. So after the first R and supply chain meetup, we did make a group on the R for Data Science online learning community. And so it is chat-supplychain. And if anybody wants to join that group, I'll put the link to join that Slack community in the Zoom chat too.
I see Federica put something in about the book clubs. So, yeah, if you – I would say those book clubs – I guess I'll give a shout-out to the – I want to give a shout-out to the book clubs. Really good way to kind of discuss with like-minded individuals different modeling techniques, especially if you don't have somebody in your organization that is also doing this kind of stuff or if you're working remotely and it's a nice way to connect with like-minded individuals.
Definitely. And I did just put my LinkedIn there too. Feel free if anybody has any other follow-up questions or wants to connect there or share your own LinkedIn in the chat too to network with each other.
Also, somebody had asked if it was okay to share an open job posting in the chat channel as well. And definitely. So if anybody on the call is hiring for supply chain professionals, please feel free to share that there too.
And thank you, Federica, for putting the link to the book club too.
Customer sentiment and demand forecasting
Yeah. So my question is that to how any organization utilize the customer sentiment to forecast or demand forecast for supply chain analysis? How any organization utilize –
May I ask what industry you're in? Consumer packaged goods or durable goods? I mean are you talking about maybe getting a – are you talking about like a government data index where I think there are – it's been a while since I did economic consulting, but I think there are data that you can download sentiment.
So, I mean, you could download that data. I think there's probably some R packages that could do that and put it as a predictor in your model. That's something I think maybe you could – if you really want to be creative and you had surveys from your own customers, maybe you could construct some kind of index that you read in that survey data and assign a numerical weighting.
You could, you know, use that or other kind of feedback from perhaps a website or something else. I'm not – I haven't worked in consumer packaged goods for several years, so I'm not super up on whatever might be the new techniques, but those are a couple ideas that come to mind.
Measuring forecast bias
Thanks, Laura. I do – Vasilios, I did see your hand was raised, so I'd love to just have that be the last question too if you want to jump in.
Have you also used – have you also measured bias to optimize your forecasts or just accuracy measures? And if you did, like, how did you do that in R, outside of R? Just some basics.
Yeah, yeah. I think – and I think you might – I think I'm going to miss that question in the chat earlier, so thank you for asking that. We do measure bias. Well, we have – so, the bias for – I think perhaps many of you on the call know this, but it's basically the variance over the demand, so some of all the variances.
So, that is basically directionally, do you under-forecast or over-forecast? It's actually a big point of discussion right now for our commercial partners. For stat forecasts, we more focus on accuracy, but yeah, if you have the data and how we use a lot of Excel – my coworkers use a lot of Excel spreadsheets, these giant spreadsheets that, hey, it is what it is, so I work with that.
So, we have OpenXLSX, so that's high times fast. It's a great package, but I can write that data in and we do track it, and it's important because I think a lot of times people do get focused on accuracy, and then I actually seen that one of our – one of our divisions, and they have a – they have this consistent bias, but they're like, well, we're only like 5% MAPE, you know, or rated MAPE.
It's like, we're great. One's like, well, there's some inventory building up there. So, yes, it's really important to do, and you should be able – if you have absolute variance, you might have created a variance field, so you could potentially use that and just calculate your bias at the same time.
One's like, well, there's some inventory building up there. So, yes, it's really important to do, and you should be able – if you have absolute variance, you might have created a variance field, so you could potentially use that and just calculate your bias at the same time.
Thank you so much. Thank you all so much for joining. I just don't want anyone to feel like you have to stay around, but if you want to hang out with us all and maybe just chat informally, feel free to stay on. I just want to say thank you so much to Ralph and Laura for your awesome presentation.