Resources
Dec 15, 2023 |
1h 2min
|
3k views

It’s Abstractions All the Way Down… - posit::conf(2023)

Presented by JD Long

Abstractions rule everything around us. JD Long talks about abstractions from the board room to the silicon.

Over 20 years ago Joel Spolsky famously wrote, "All non-trivial abstractions, to some degree, are leaky." Unsurprisingly this has not changed. However, we have introduced more and more layers of abstraction into our workflows: Virtual Machines, AWS services, WASM, Docker, R, Python, data frames, and on and on. But then on top of the computational abstractions we have people abstractions: managers, colleagues, executives, stakeholders, etc.

JD's presentation will be a wild romp through the mental models of abstractions and discuss how we, as technical analytical types, can gain skill in traversing abstractions and dealing with leaks.

Materials: https://github.com/CerebralMastication/Presentations/tree/master/2023_posit-conf

Presented at Posit Conference, between Sept 19-20 2023,
Learn more at posit.co/conference.
--------------------------
Talk Track: It's abstractions all the way down ....
Session Code: KEY-1161

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

You know the drill now. I'm about to introduce the next keynote speaker, JD Long, with a poem.

Introducing JD... Sorry, JD Long. Try and get your name right. Introducing JD Long, a legend in the field. Open source wizard, his knowledge unsealed. Since 2002, he pondered with might. Open source tools, can they take flight? Look at risk management, he knows it well. 13 years deep, with stories to tell. With R and Python, he crafts his art. Jupyter Labs, our studio, all play their part. In Richmond, Virginia, he makes his home. With wife and daughter, they brightly roam. A recovering lawyer, a philosopher teen, in this vibrant family, dreams are seen.

Oh, thank you all. Thank you very much. You know, it's funny. I didn't realize Hadley was doing the poems before everyone. And this is like, I'm an artist who's been put out of business by Chad GPT, because in 2000, I don't know, 13 years ago or something, I spoke at the R and Finance conference here in Chicago. I did a lightning talk, because that's run out of here in Chicago.

And my gag was, I did it in Seussian rhyme, and it was hilarious, right? Because nobody comes to a freaking finance conference and gives a whole lightning talk in Seussian rhyme, and now anyone can do it, and they hardly have to work at all. Ah, so frustrating.

Well, I just want to say hi to you all. I love being here. It's exciting to be back in Chicago. I helped start the Chicago R User Group many years ago, and it started as, we had some guys, they were all guys at the time, R and Finance guys and myself, and we were going to Jake's Tap and talking about R, right? And Jake's Tap is closed now. Ah, moment of silence for Jake's Tap. Or was it Jack's? It's Jack's Tap, I think, right? Anyway, I don't know.

Fantastically good times, and we decided we should just invite other people, right? We're having good conversation, and what we'll do is we'll do what we would like to participate in, but we'll include other people. And it's so lovely to see that the Chicago R User Group is still, you know, alive and well and kind of following that ethos of, you know, share the things you're doing and include other people, and that's really kind of an R community ethos and also, you know, an RStudio slash Posit ethos, and it makes for a fantastic community.

So speaking of that, like four years ago, I showed up at the RStudio conference at the time and gave a presentation, and I wore, you know, a shirt with this pattern on it because I tried to explain to people what I do for a living, and this was the best way to explain it because I'm an agricultural economist who works for a global reinsurance company, so it's easier just to say spreadsheets and bullshit.

And Mike Smith saw this shirt and liked it, and I promised him one, and then COVID happened and life happened, and Mike Smith, come get your shirt. This is for you, Mike. So we'll just lay this right here, and Mike Smith can come make his way down here and grab this, and if the size is wrong, Mike, we'll get you one in the right size.

So I need to, I do work for a financial services firm, so I work for Renaissance Free, and I am not my employer, nor am I representing them. I'm not discussing reinsurance here, and all the ideas here are mine unless otherwise stated. This isn't business, this is pleasure.

Community roots and a note to Hadley

And a little funny story. I was trying to think how long I've known Hadley and JJ, and I knew them separately before they were working together, and I stumbled on this. Back in 2010, for Hadley Wickham's birthday, so almost 13 years ago last week, I sent Hadley a copy of Generalized Additive Models, an Introduction with R, which had been on his wish list in Amazon because he was a poor, starving faculty at Rice University, and he was maintaining open source software, and not only that, he was answering my stupid questions on Stack Overflow.

And I wrote in the gift note, thanks for helping me kick ass with Plyer, now that's the precursor to dPlyer, Plyer and R, I appreciate the tools and the help you've given me on Stack Overflow. And this is the kind of community we have, right? And I just think it's hilarious. Hadley had no recollection of this. I asked him about this a couple days ago. No memory of this whatsoever.

But that doesn't keep me from this being my financial disclosure of my relationship with Hadley. And if anybody is wondering how you become a keynote speaker at Posit, the price is $68.78 and 13 years.

The idea of abstractions

Well, I enjoyed, so I had no idea when I started forming this presentation that Jeremy was going to be one of the keynote speakers. But I really enjoyed listening to JJ and Jeremy do this thing they called a two-way AMA, which I didn't know what that's called. I've been calling it having a conversation, and I'm just not really cool, right? So I learned that this is called a two-way AMA, and they recorded their two-way AMA, and it's on YouTube, and it's really fantastic.

And I was listening to this as I was driving from my parents' house to Nashville, Tennessee, and if my family hadn't been with me, I would have pulled over to the side of the road because I was listening to just the audio on my iPhone, and I would have pulled over to the side of the road and made notes. So I ended up listening to it and going back and listening to it again.

And one of the things that jumped out is they made the comment of if you think about how you help both new users ramp into things and make experienced users productive, you provide these abstractions, and there's a dial of how leaky you want the abstraction to be. Now, a bunch of us who've been around software, maybe worked in software engineering, but definitely talked about to software engineers, know this idea of abstractions and leaky abstractions.

In this community, a bunch of us come through clinical sciences or we come through other fields that aren't computation first, and I was thinking this term is really powerful, both abstractions and the idea of an abstraction leaking. These are really important concepts that I think we should more widely ingest.

So when Hadley contacted me and said, hey, you want a keynote? I'm like, yeah, what's themes or whatever? And he's basically like, I don't know, you use Python and R, maybe like some, I don't know, do whatever you're thinking about, right? Well, it just happened, I was thinking about this, and this has been one of the things I've been thinking about with my team and the people I work with, is how do we talk explicitly about abstractions, leaky abstractions, and how we deal with those leaks?

The law of leaky abstractions

So let's talk a little bit about abstractions and leaky abstractions. So first thing I did was go back, I thought I knew where this came from, and I confirmed that while the term leaky abstraction was around in the zeitgeist, it really didn't get traction in the tech community until Joel Spolsky wrote this blog post over 20 years ago, and he calls it the law of leaky abstractions, and the law is all non-trivial abstractions to some degree are leaky.

and the law is all non-trivial abstractions to some degree are leaky.

And what he means by that is an abstraction failure, sometimes a little, sometimes a lot, there's leakage, things go wrong, it happens all over the place when you have abstractions, right? So it means that you have this thing you're relating to, it's abstracted away so you have an interface, an API, a calling to a function, and you interface with it in a way and it doesn't do what you expect. That's a leak.

So the question becomes, what do you do? So you've got, if you want to be a master of any abstraction, not just a user, but a master of the abstraction, you have to understand what's under the abstraction. Now that's the only way you can truly debug or truly understand an abstraction is to understand at least one layer beyond. And I did a lot of research trying to figure out who originally said this concept of you've got to understand one abstraction below in order to be proficient and debug one abstraction, and everything I found was sort of like everybody just knows that. Like nobody attributes that thought to anyone, it's just in the zeitgeist that that is reality, so I guess it's like an axiom.

Computational and organizational abstractions

So when we think about abstraction, I want to expand what I mean, because many of you are probably thinking something like this, and I just like Googled computing abstraction, right? Where we have at the high level, you have like a high level language or an application, and then it goes through an assembly language program and the assembler turns it into machine code, and then it, like actually, then there's a bunch of hardware abstractions, and then an actual calculation gets done inside the hardware of the machine. That's how we often think about abstractions.

I want to expand that for the purpose of this conversation, because there's also organizational abstractions. So if you think of an abstraction as we've got some set of directives we're passing down, and we kind of don't care specifically how things get done in the next layer, we just want the next layer to do something, well that's not unlike organizational structures.

I come from corporate America, right? So I think, but this generally concept applies to your nonprofit or your software company or even your civic organization. At some level, we have a board of directors or some committee, and they pass down directives and priorities to the executive management. The executive management makes a bunch of choices and then passes things down to department heads who pass it to team leads, who pass it to team contributors, and if we continue to think about this, they then pass those on to computers in some way, right? Like they pass in because they use applications or they use high-level coding languages or something, and then all that other stuff from the slide before happens underneath this. It's abstractions all the way down, right? There's the title of my talk.

Why we need abstractions

So why though, like why do we need these abstractions all the way down? Wouldn't it be easier if we just had, like I grew up on a farm, and the great thing about farming is you have to do everything because you don't have staff. The worst thing about farming is you have to do everything because you don't have staff, right? So you become a master of every level of abstraction until you get in over your head and you have to have John Deere repair some piece of machinery or something, right?

But you become, all the way up and down the levels of abstraction, you become at least proficient. It's been hard for me to get used to organizations that didn't expect me to run up and down the stack, right? Because I want to run up and down all the abstractions. Well, the reason we can't always do that was really articulated back in the 50s by Herbert Simon. He's an economist, I'm an economist, so I've got to get economists in here.

He had this article in 1957 called Administrative Behavior, A Study of Decision-Making Processes in Administrative Organizations, and he coined the phrase bounded rationality, which I TLDR as head trunk only hold so much junk, which Gary Lawson captured in this cartoon where it says, Mr. Osborne, may I be excused? My brain is full.

We can only handle so many levels of abstraction and so many pieces of the stack before our brain overflows, and we can't make sense of all the pieces. And so we build these interfaces, and even if we are the person traversing the levels of abstraction, we would like to interface with different pieces of them and not think about what happens below them, even if we wrote what's below them, because it means when we're problem-solving here, we don't have to think about how the read-write is happening on the database. That just magically happens behind an abstract interface, and we don't have to think about it, and it allows us to work at the problem-solving level that's appropriate for what we're trying to accomplish. That seems to make a lot of sense, right, intuitively?

What abstractions are for — and what they are not

So let's talk a little bit about what abstractions are for and what they are not for. So one thing I want to point out that they're not for is they're not for gatekeeping, and I see this being done a lot, right? You're not a real data scientist unless you, you know, PyTorch or deep learning or whatever. Those are all different abstractions, different tools, that are used in certain places to solve certain problems. Those may not be your problems. Then you don't need to know that abstraction. You don't need to be a master of that abstraction. You don't need to be one layer below, and I watch a lot of early learners run around learning abstractions, learning tools, because they feel like if they don't know this tool, they're not a real whatever-it-is-they-think-they-want-to-be.

they're not for gatekeeping, and I see this being done a lot, right? You're not a real data scientist unless you, you know, PyTorch or deep learning or whatever. Those are all different abstractions, different tools, that are used in certain places to solve certain problems. Those may not be your problems.

That's really toxic, because you'll wear yourself out because this guy has already said you can't fit it all in your head in a really useful way. So don't let the learning of abstractions be like, oh, once I accumulate a big enough toolbox of these abstractions, then I'm a real whatever. That's just gatekeeping, and cut that out. You don't even have to know all the abstractions you use deeply. But you do need to know your limits.

So know which abstractions you really understand. Recognize when you're up against an abstraction that you don't grok, so you don't understand the abstraction. It's a breakpoint. You're like, I don't really understand what's going on beyond here. At that point, you have a choice. You can either learn that abstraction, learn what's really going on beyond it, so that you can deeply understand it, or you can partner with someone who's an expert there. Partnership and pairing with someone else and working with someone else is always an option. It may be harder in some organizations than others, especially if you're the only person on the data science team. You may feel some pressure to learn those abstractions, and that may be the right choice.

But if you're in a larger organization, and someone else in the organization is a master of that abstraction, you may not need to become the expert on database indexing. Contact Kara. She seems to know a lot about it.

And we have to choose, make that choice. Now, what I see happen a lot is people blame an abstraction for problems when they bump up against it. And often the problem is, they don't understand what the abstraction is doing. Now, that may be a leaky abstraction, but still, it's like, okay, my dashboard doesn't refresh fast enough. My database has a problem.

I had literally this one within the last year. And I worked with the Power BI developer, and we discovered that when using direct query in Power BI, which is how you access any database that isn't the one that's built into Power BI, it issues all the queries in series and will not issue them in parallel. So the dashboard had, you know, 13 queries that all took three seconds. They could have been run in parallel, and the whole thing refreshed in three seconds. Instead, it took 13 times 30, because it was running them in serial.

And the analyst had thought, oh, this database is crap. And I'm like, actually, Microsoft is shunting you into buying more cloud storage so you can shove your data into their platform instead of letting you use your own perfectly good database because they force your connection to issue queries in serial. And I'm like, that's broken, right? That's not a leaky abstraction by accident. That's a leaky abstraction by sales, right? And that should make anybody that runs into that one angry. And it made me angry.

That's not a leaky abstraction by accident. That's a leaky abstraction by sales, right?

And I'm like, we've got to get other tools. This is ridiculous. Like, we're being manipulated. We're being played. And my analyst didn't know it, right? Because he didn't really understand that abstraction. He didn't realize how to go into the database, get the logs, and see that they were happening in serial. But he does now, right?

But that's an example of don't blame the abstraction. Understand the abstraction. Understand what's going on. And then decide what you want to do with that information.

Dijkstra on complexity and separation of concerns

So I would, if I did anything where I dabble with, like, computer sciencey concepts and I don't quote Edgar Dijkstra here, I would probably be remiss. So he has this kind of great quote that says, Programming, when stripped of all its circumstantial irrelevancies, boils down to no more and no less than very effective thinking so as to avoid unmastered complexity to very vigorous separations of your many different concerns. Right? And the TLDR is, constrain complexity and separate concerns. That's what we're trying to do with abstractions.

And that's what you should be doing when you're writing functions inside your in-house libraries. Your first priority should be, how do we make it less complex? How do we separate concerns? And that's that, even if you understand and you built the pieces that are under certain abstractions, if you can separate those from your thinking while you think about this piece of the problem, you've got less of a sandcastle in your head to build the tools and do the analysis you want to deal with because you've got a lot of the complexity pushed away and you know that if you have a certain problem, which piece of complexity you then need to go look at. It's not all commingled together. So that's a really good concept and that was, you know, he wrote that down in 1975. And I'm not sure we've done a really good job embracing that. I know I haven't.

How fast do electrons flow?

So, let's do a fun example about thinking about abstractions. Anybody in here an intellectual engineer? Oh, more than I expected. I'm going to ask you guys, like, not to yell at me. Like, just stick with me. Just stick with me. It's a thought exercise. Don't rush the stage here.

Electricity is a fundamental abstraction for computing, obviously, right? We know this is all powered by electricity. We know, like, we've got gigabit switches in our office, so that's a billion bits of information a second. There's a bunch of wires and we don't really know how that works, but it's got to be at least going, like, millions of ups and downs of voltage a second, right? Inside your gigabit switch. That little light blinks fast, right? It feels like something like that.

And so, like, how fast do the electrons flow in our wires? So, just think about this for a minute. Don't yell an answer. How fast do they go? And I'm like, well, okay, well, we're flipping the little bits up and down, you know, a million times a second or more, maybe a billion in the wire, some big number. That's really fast. That's a lot of, that's got to be moving fast to turn it on and turn it off that fast. It feels really fast, you know. I don't open the refrigerator door and it's dusk. It's always, you know, light straight on.

When I was in middle school, they showed me illustrations that look like this, and I think the red balls are the electrons, right? So, the battery forces the electrons through, so those things must be going really fast. You know, and then the next year, the teacher did this diagram and likened electric current flow to, like, water flow, and, you know, resistor was like a restriction on the pipe, and we got that, and that was really helpful, so this all kind of points me to, this stuff's going, like, really fast. We're probably going near speed of light, but it can't be the speed of light. It's, you know, there's some resistance from the wire, and that's why wires get hot, but it can't be too much because they get really hot all the time, and they don't, so I don't know. We're probably pushing the speed of light inside these wires, right?

Yeah, no, it's 8 centimeters an hour at 1 watt. What? Right, okay, who was surprised by that number? Like, it kind of defied your intuition. The ones with your hands down, you guys are, like, smarter than me, and I'm glad you're here because I want to know smart people.

The problem is we conflate electromagnetic fields with electron movement, so the electromagnetic field moves really fast. The actual electrons move quite slow, so in an AC wire, when you're, you know, drying your hair, it's something like this. That's the speed of the actual electrons moving through the wire, which kind of blows a lot of people's intuition when they learned with these metaphors, like I showed you on the previous slide.

If anybody finds this interesting and would like to go read more, oh, by the way, if you want to take pictures of my slides or anything, that's great. Knock yourself out. At the end, there's a QR code that'll take you to GitHub repo that has the whole slide in there, so don't feel like you've got to capture the links or anything. But this is a link to a YouTube video that explains this whole thing, and it's really interesting, I found.

But why do many of us not know this, right? Because we use electronics en masse every day to solve all our problems. Well, the reason we don't know this is it doesn't matter. And not that it doesn't matter that we don't know it. It doesn't matter. It doesn't change anything. There's absolutely nothing you ever know where the movement of the electron matters. That abstraction never leaks. We never are diagnosing a problem, and we're like, I finally get the answer. It's because I thought the electrons were moving fast, and they're actually moving slow. That's never the problem we have, because this abstraction just doesn't leak.

Now, there's a whole genre of YouTube videos and clickbait that are basically like this. They surprise you with some piece of information you didn't know about something that you interact with all the time, and sometimes they're absolutely full of crap. Because we like this because it's surprising, because it's like a dopamine drop when we're surprised by information.

And at some point, though, it doesn't matter. So, you know, to understand abstraction, we build these mental models. We make metaphors like my water flow or my electrons being elevated and moved and dropping. Often they're wrong metaphors, and sometimes that doesn't matter because they allow us to understand the thing well enough to use it.

So I do my own, like, home electrical wiring. I'm rewiring. I redid my Jeep wiring. I'm doing a trailer now. I use computers. I've got a desk with, you know, power supply on it for testing Arduinos and controlling, you know, actuators with Arduinos and Raspberry Pis, and I understood how electrons flow through a wire. So my metaphor was good enough, even though it was technically wrong.

Floating-point math: a leaky abstraction

Well, what if our mental model is wrong and it does matter? Because, you know, it's not a universal case that it doesn't matter. So, buckle up. Let's talk about floating-point math. This is my favorite leaky abstraction. And by the way, I'm only going to have two slides with code, and they're both Python because we've got a big community. We're a very open tent.

You know, if you took, you know, grade school math, everyone in here did, at some point, the teacher, and if they were very creative, they did a pretty slide like this one to explain the associative property of addition where it doesn't matter how you group the things you're adding, you get the same answer. Very young. And this is how addition and multiplication work. And, you know, I think we all saw something like this. I stole this slide literally from, you know, a middle school review online.

But, we come into Python, and we go, okay, well, we got these numbers. They're real numbers. They're on the number line, right? We're going to take 1.11, add 2.22, and then we're going to add 3.33, and we're going to group them a little different, and they are not the same. And the reason they are not the same is floating point addition and multiplication are not associative.

And the reason why is illustrated when I format the print on these to show 17 decimal points. Because way out there in it-doesn't-matter-practically- to-you land, most of the time, out there in the 14, 15, 16, 17th decimal point, you can see these numbers are just a little bit different. And that's because floating point numbers are an abstraction. They're not literal point on the number line like we were taught in grade school about numbers. They're something different to make it work in computers. And this is why daddy drinks.

And it's things like this that will drive you insane. Because even though most of us understand this about floating point math, I'll tell you a very quick story about where I spent two days with an engineer last week, then it was a floating point math problem. But it didn't come at us as just value A equals value B.

What we were doing was we were doing a bunch of math, and then we rank it. And after we rank it, we then say, okay, I want the row number, and we use a second sorting number to say, basically, if there's ties, use this other value. Use this deterministic number over here to break ties. Because we want all of our systems to be deterministic, meaning that there's no random number generation here, so every time I run it, I want it to barf out exactly the same answer. No swapping values around, right? This is huge. It ingests like 5 billion records and barfs out 11 million after a bunch of aggregations.

And we were finding we weren't getting deterministic answers. Now, out of 11 million, maybe we would have 30. That would change every run. And we were like, that's not cool. What's going on? And as we dug into it, what we found is some of the condition that should have been a tie was like this situation where I was showing you where it's conceptually the same number, but they were flipping because, and the reason it was happening is we were on Spark. Spark's a distributed system. You don't control which executors or how the groupings are done because it's map-reduced. So every time we ran it, and maybe we ran it with a different cluster size, maybe one machine got the data first, little things change, and so when we aggregated these up, we'd get slightly different floating point rounding because of the way they were grouped.

And it became material because of that sort order problem. We thought we were handling ties with numbers, but we weren't. So, I mean, that was easy to solve, right? We just round those to some smaller number, you know, or we cast them into something with less precision, so we get rid of that noise out in floating point land. You know, but we spent two of our time for like, well, I think there's more people than that involved, for like four days trying to figure out why the hell these numbers keep changing, right? Why is there a ghost in my machine?

Well, if you're interested in this, this isn't a presentation about floating point numbers, even though it may feel like it right now. I recommend reading Julia Evans, you may know her as Bork, article about floating point numbers, and she does a number of fantastic computer science-y zines explaining these concepts using cartoons, and they're fantastic. And, you know, she points out that doing numerical computations on a computer inherently involves some approximation and rounding, right? That's one thing we should all come in this with knowing.

Organizational silos as abstractions

All right. Enough of floating point math. I'm an agricultural economist. Let's talk about silos. So, Cara mentioned silos in organizations, right? And so let's think about using our mindset of an abstraction, let's think about silos for a minute. What are if organizations are abstractions, what's the abstraction equivalent of an organizational silo? And I would say that's a over- restrictive abstraction, and it's an abstraction that when it leaks, you can't figure out why.

what that, what org abstraction leakage looked like. So, where did this number in the DB, in the database, where did it come from? If your organization is highly siloed, you may not be able to answer that question if your team didn't calculate it. That's an overly rigid abstraction.

Who do I talk to about solving this problem, whatever that may be. If you can't answer that, you've got a siloed organization, right? It's overly rigid abstraction. And how was this decision made, right? Non-computational, right? It may just be like, you know, we're making decisions about people, we're making decisions about a number of different things. How were they made?

Signs of leaking abstractions

the general evidence of leaking abstraction, what are those? Now, these are like, I went looking for, surely people have articulated this, and I couldn't really find any other authors or academic authors who had really outlined this. I am sure it's out there, and I am reinventing the wheel. But the thing I'm always looking for is unexpected behavior, uninterpretable responses, or logical errors where I, there's no explanation. And whether that's my computer or my organization, there's no signs of slippage or leakage in the abstractions.

So, I think of a bunch of this as really, in an organization, it's a communication problem, and communication is like beer. Communication is the source of and solution to all our problems.

Communication is the source of and solution to all our problems.

So, you know, if you run into where did this number in the DB come from, there's ways that we can improve communication in our organization to answer this, right? We can have the code for the ETL or whatever in version control, so Git, GitHub, whatever version control you use, and available to the people in the organization who consume the values, right? And that's really important because a lot of organizations do not share, read only is fine, but they don't share the ETL process for how they do things with the people who are consuming those things. That's insanity!

Right? Like, we can read code, many of us who are doing things with the data, show us how it was calculated because it may answer a whole bunch of questions we have about what we're trying to do. And, you know, a wiki page describing the process if there's a complicated decision that's made in how we do grouping, we always omit this, we always include that, that sort of stuff, put that junk in a wiki or documentation that's easily accessible.

So, similarly, like the question, how do I talk about solving this? Well, if everything in the organization has clear ownership, that one gets resolved really fast because you talk to the owner of that process or that system, and we often let ownership slip and things get orphaned, and then they just leak, like, all over the floor because there's no person whose job it is to explain why it's leaking or help stop the leak or help you understand what the system is doing. And then things like, how was this decision made? Well, that's better transparency, leadership communication. Somebody didn't communicate how the decision was made or what the process was.

And a bunch of times, those are easily solved by communication, and probably, for big decisions, they need to be communicated multiple times in multiple ways on multiple methods. So, a video, an all-hands call, an email. You know, one of the struggles I have is people tend to act like email is a reasonable way to access me when they put me on BCC. I don't read if I'm on BCC. The volume of email I get, I can barely read the stuff that's got my name spelled right. And everything else goes into a folder named, like, BCC Read Later.

Later. And I, every once in a while, I miss important internal corporate communication because I wasn't paying attention to that, but my email volume is much more manageable. So, don't communicate very important things to your organization with them all on BCC. So, sometimes we need multiple methods to communicate to make sure the message isn't dropped.

Drew Conway's Venn diagram and data science roles

Alright, with that said, that's a little bit about organizational abstraction leakage and how we might address that. I want to do a little history, and it wouldn't be a data science conference with an old man at the front if we didn't put up Drew Conway's Venn Diagram from 2010.

Now, funny story, I knew Drew when he was a PhD student at NYU, and I saw him, like, draw this out on a napkin, and it, like, completely blew my brain when five years later, like, the actuarial magazines on my actuarial department desk at work, like, had this diagram on the cover. I'm like, this is so weird. So, sometimes I feel like I'm the Forrest Gump of data science. It's like, I was sitting at the bar when Drew Conway wrote this on a napkin, right? I didn't actually do anything, but, you know, I saw it go down.

So, what's interesting about this, and the reason I want to bring it up, is the thing that was so revolutionary about this in 2010, inside of organizations, is these are three different abstractions. And they, one person was, not only wasn't expected to do this, one person in many organizations was not allowed to do this. I can remember many times in the early 2000 being told, I couldn't have coding tools on my machine at work because I was not a software developer. And I'm like, I don't think you understand what we're doing and what these tools are.

Right? And it's like, oh no, you're in the business. You get, you know, Excel. And I'm like, I can write code in VBA. You're not going to like it. Why do I have to use the shitty code when I could write good code, and you're just shunting me into that because it's the abstraction you're comfortable with. It's not the abstraction I'm comfortable with.

And what happened since then, so in only, you know, 13 years, is it's now totally acceptable, almost everywhere, except a few backwards government organizations, for people who are doing like data science-y work in the business, doing analytical work, to have first class programming tools on their machine, and they have rights and permissions to use those, and often install packages, and even if that's from a curated internal repo. But we can do these things that 13 years ago we were literally told we were not allowed to do. That's a fundamental change.

And I will assert that the single biggest business value derived from the data science movement in the last 13 years is making it legitimate to code outside of IT software development roles. Now, that's a pretty big assertion, right? A lot of value has come out of data science. But this is the, inside of these big calcified organizations, seeing all Drew Conway's diagram inside of magazines, and executives say, well, how do we get that? And I look over their shoulder and say, stop being stupid about the rules of what we put on our desktop, or give us access to, you know, coding tools through a JupyterLab, or through RStudio that's centrally hosted. You know, that opened up for these tools to permeate across organizations. And so these data science roles break previous organizational abstractions is my point there, right?

Abstractions must be permeable

I'll make a second assertion, and it's tied into what we talked about earlier. Abstractions will leak. Spolsky told us that. Therefore, abstractions must be permeable to allow debugging. So that's my thing of, put your ETL code where everyone can read it who might be using the results of it. That's allowing us to at least peek through the abstraction and see what we're getting.

This holds for both organizations and computers. So inside of organizations, we need to understand when those organizational structures leak, how do we deal, how do we peek behind the layer of abstraction? Now, for me, I've been in an organization, the same organization for an insane number of years. I've been at the same employer for 14 years. I have had the same boss for 12 years. My solution is easy. Because of that, I start calling people, right? Hey, what's going on with whatever? Or sending them a text message. But my analysts don't know who to contact. So I've got to be that to them.

So I've got to go to them and ask questions of, hey, do you guys understand what's happening? Do you understand why? And then we talk. Okay, some of you may have had questions or thoughts that you weren't comfortable bringing in the group. Feel free to hit me up one-on-one. What I'm trying to do is these things may be leaking for you. It's okay. Come to me. Talk about it. We'll deal with the leaks, and I'll help you understand what's going on. And this is germane, because we're right in the middle of acquiring another company. And there's lots of questions, right? And some of it I have answers to, and some of it I don't. But that's how we handle it.

So there's leaky abstractions all the way up and down your organization, and all the way up and down our tech stacks. All right, my assertion three. No single abstraction is right for everyone. I think we're going to need more abstractions.

The 80-16-4 framework

Now, we have found in our organization, and this is the part where I'm going to share some things that's not just my thoughts. My colleague, John Moore, and Peter Ilston, another one of my colleagues, came up with what we call the 80-16-4. And it really started as the 80-20 rule, and then we realized we needed to divide the 20 into two buckets, so we applied the 80-20 rule to the 20, and we got 80-16-4, because math is fun. And we like non-Gaussian distributions, given our world.

So the way we think about everything we do is 80% of our users in our organization are normal business users. Now, I'm going to present all this from a for-profit corporate organization. These should project into your world, and don't, even though I've got numbers up here, don't be obsessed with what that means. This is conceptual. The vast majority of your folks are normal business users. They want to use applications, dashboards, basic Excel. That's their tool stack. Those are their abstractions.

And then 16% or so are super users. Super users are going to want to do SQL against the underlying data that's collected by your tools. They may want to do a custom dashboard. They're going to do some advanced Excel, like pivot tables, and really processing the data a bit more. And then 4% are guru users. Your guru users are like, oh, I want to do that, but I want to do it 10,000 times, so can I hit the API and just do that directly, pull a result back, make a change, put it in, run your thing, pull it back? You know, that's a very different use case. Or maybe they want a library, a Python library, an R library that interacts with your corporate tooling. Very different abstractions needed by these three different groups. And they're cumulative, right? So the way I think about it is your super users, they will use the dashboards and the basic Excel, but they also want these other things. Your guru users are, of course, still using SQL, right? And they're still using your applications. They're using everything.

This has been incredibly helpful for us because what I had observed in our organization and in others is a tendency to, like, scratch the 80% itch and then kind of stop and make it hard to get the underlying data. And we had had at times in our culture where that wasn't the case, and then as we started specializing teams, some of this 16 and 4 was getting dropped sometimes. So, you know, we've changed kind of the definition of done on the things we build, saying we've got to make sure people from all three of these groups agree that they have what they need.

Now, it doesn't mean that everything we do has an API, right? That's not the principle. The principle is what the people in that group need, they have access to. And I try to be real careful, like, don't hear me saying, you know, build APIs everywhere, because then all of these, no, you're working backwards. Work from the principle, not from the implementation. We implement what we need to address the principle. How do you know, like, what your guru users need? We'll just make them an API. No, you're going to go ask them, right? Back to communication. And they may say, oh, if I just have access to the output data in SQL, that's more than enough for me. Great. Don't build anything specifically for them.

This is where, like, I really like the current movement for data products, because data products are used, data product meaning, like, it's defined, it's got a data dictionary, it's not, like, tied in necessarily to the whole corporate database, but it's a standalone data product, all the fields are defined, and in our shop, we have a link to a wiki page, and a link to the source code that builds that data product available for everyone. I love that. Like, even though when I first heard data product, I thought it was, like, more marketing BS, and I just wasn't excited about it, when I saw what I got from it, I'm like, oh yeah, data product, I want that. Because a whole bunch of my 16 and 4's needs are addressed by having a good data product, right? Because it's a good abstraction that we can peek behind, because of the documentation and the links to the source code. So it's fantastic.

So, like, the point there is multiple abstractions is a high empathy move, right? I'm here with my practice radical empathy shirt on. That's high empathy for who's actually using it. Building APIs blindly because Gartner says that's a good idea is not a high empathy move, right? That's just parroting, you know, what someone you believe to be a thought leader. Don't do that. Think. Right? Think about what people actually need, and build what they actually need. And more importantly, don't build stuff they don't need.

Big idea recap

And that's hugely empathetic. So, my big idea recap is abstractions start way up with leaders and go all the way down to hardware. It's abstractions all the way up and down. To debug an abstraction, you have to see what's below it. We're building mental models of complex systems, and that's why we need abstractions, because we can't hold it all in our head at once. You can't master all abstractions. Decide if you're going to learn or if you're going to partner, but don't decide to blame the abstraction you don't know. Ensure an abstraction is for everyone. My metric for that is the 80-16-4, but don't be dumb about it. And more importantly, just like Hadley helped me years ago on Stack Overflow, help everyone kick ass, because

almost nobody really wants to know how to code or do these things. We just want to solve some problem, as Kara said. Or as I say, we just want to kick ass. Right?

So, here's the QR code that'll go to these slides. There's my LinkedIn and my Mastodon. If you guys want to hit me up there, I'm happy to talk to y'all. We've got a few minutes, so let's get Hadley and do some Q&A.

Q&A with Hadley

Thank you, thank you.

So, I had one question to warm you up. So, obviously, I'm like a huge fan of abstractions. A lot of my work is just creating abstractions. But one of the other things I enjoy is, in my personal life, discovering abstractions that make something outside of work interesting. Do you have any examples of that in your life, where you've discovered an abstraction that you really love? Oh, yeah. It's so nerdy. Let me think how to summarize this.

I like old machines, and I got an old Jeep that has no moving part on it. That's an original Jeep. So, it's got a Chevy small block in it, and it's had the carburetor replaced with a throttle body fuel injection. So, that's a fuel delivery device that sits where the carburetor could go, but it's got little jets in a computer, like a modern vehicle, but it's squirting into a 1969 small block Chevy engine. And it is so fantastic to read the discussion boards and the people debugging throttle body electronic fuel injection, because they bring their carburetor metaphors in, and they don't work, and they're just mad in hell at this thing being broken, that it doesn't work right, and it's because they don't understand the abstraction.

And I think about this a lot. I find debugging the issues with my electronic fuel injection pretty straightforward, because it's not really a complicated system. It's got two sensors. You just sense the air flow, and you sense the oxygen at the tailpipe, and it's a pretty straightforward control system. But if you don't understand some of the nuances, you just bang your head against debugging it, and so it's very much like this.

Can you suggest things to watch for that kind of signal that maybe we've fallen too far down, like some particular abstraction rabbit hole, and we need to pull back up or look at another abstraction?

Yeah, that's a good point, and I wish I had these guidelines, and I might, if I had them, I might anonymously send them to the Spark development team. By the way, I use Spark a lot, and this talk was largely inspired by banging my head against a bunch of things in Spark. It is the leakiest abstraction of any useful piece of software. I've dealt with software that leaked more, but it wasn't useful. Spark is incredibly useful, and it leaks like a sieve. The abstractions are so leaky.

Spark is incredibly useful, and it leaks like a sieve.

I think we should always... This is some real Stephen Covey seven steps of highly effective people advice, but we need to keep the end in mind. What problem are we solving? The goal isn't more abstractions. The goal is compartmentalization of concerns and the problem, and if we're spending all of our time redefining new abstractions because we have an abstraction fetish, that may not be healthy, but it is a little bit gray sometimes to know the right level abstractions, or if you build an abstraction that's too compartmentalized...

I think one of the examples that's floating around right now in the zeitgeist online is the discussion of microservices and how lots of companies, the development teams, have looked at what Amazon and Google do and are like, oh yeah, everything has to be a microservice. What is a microservice but a very specific abstraction, right? They build thousands of these, and they're not Amazon, they're not Google, they don't need that. That could have been solved with a query, could have replaced five microservices. That's an example of losing sight of the problem and following the advice blindly of microservices are good, or APIs are good. So keep the end problem in mind of what problem we're actually going to try to solve.

How's it going to be used? One of the things I do is I love user personas, and people in marketing use user personas, and they're often quite vague. They'll be an imaginary person. I've used this in other talks, but my favorite is Ann Taylor, the woman's clothier. There is no person named Ann Taylor, but you couldn't tell that inside of the design meetings at Ann Taylor, because she is a real person. She has a dog, dog has a name, she has an address, she has a job. They completely describe her, and that's who they make clothes for.

Marketing does that because they don't know their customers, so they create usually a few different personas that they are targeting. I watch software development teams inside of our organization. We've got under 1,000 people, and I have sat in meetings where they debated, what does this user persona want from this? I'm like, that's Robin. Call Robin and ask him. Why are we messing around with dreaming up our... Put a name on every user persona if your organization is small, and then ask that person, is this solving your problem? They may go yes, or they may go no, so one way to stay out of that is make sure what problem you're solving, who you're solving it for, and ask them.

The 80-16-4 role

We've got a couple of questions about the 80-16-4 role. The first one is, how can we keep our empathy for that 80% and protect them, and how do we avoid them from just being seen as these low-skilled, low-status people, the disposable post-docs of academia?

It depends on your organization. In my shop, I have empathy for them, but I don't build most of my tools most of the time are not for them. They're using tools that put things into systems, and I'm accessing it in the system. I want to make sure that they're taken care of by our software development team, but they're not mine to take care of. They need sets of tooling. In some ways, they may not be your responsibility, depending on how your organization is laid out. That's somebody else's responsibility. If they're your responsibility, you've got to figure out how to have empathy for them and make the tooling they need, but don't try to make them a 16 or a 4. Don't try to make them a power user or guru if they don't have any interest in being.

But if you get some of those regular users, and we see this, we'll get users that are like, is it possible that I could write a query to get this data out? We're like, yeah, let's talk about it. I can show you how to do that. We'll move people, but it isn't a career prerequisite that people be moving from the 80 to the 60. It's not a career path map. It's an organizational structure. If you're responsible for them, figure out how to love them. If you're not responsible for them, eat lunch with them so you understand their world, but I don't sit around worrying, are they getting their needs met? That's somebody else's job.

You actually answered both of the questions. Twofer. I like it.

Data warehouses, lakes, and lakehouses

I had a question for you about some specific metaphors of today and how you think about them. I think the data warehouse, the data lake, and now the combination of the two, which is the data lake house. Oh my god. I have dropped so much profanity mocking this marketing language around these things. Let me tell you how I think about it.

The data warehouse means to me, or the way I assume people mean, and I'm afraid people use these differently. What it means to me is some thought has been given to creating fact tables and dimensions and star or snowflake schemas that cover multiple areas of the business. It's usually a very big lift to create a data warehouse. It takes a lot of top-down orchestration to figure out the right granularity of the data for your fact tables and which things are dimensions and all of this. It's a super highly structured but really powerful, but the problem becomes it's really hard to get new stuff in because it's a big change and you will never change the granularity of it. Once that granularity, the grain of the data, it's not easy to change. It can be done, but it will be a big lift to do that. It's going to be slower moving, but it's super valuable.

The data lake approach is a little more you've got a system that squirts data out, capture it. It may be logs, it may be some sets of results. You're just accumulating those in a way where they could be queried later. Then data lake house is like, let's try to do a little bit of both all at once and we're going to see, and often what that means is you create a bunch of ETL processes in your data lake to extract things that are interesting and drop them in a schema where people can use them. That's usually what it is, is ETL on top of your data lake.

That's great. A lot of these are marketing terms to try to get people to talk about it. I'm back to what problem are you trying to solve? If we understand what problem we're going to solve, it may be fine to do one or it may be we need the other.

Very few people are building huge data warehouses in my limited experience because of this big cost low agility feature. Very slow to build and they're very slow to change and they're costly to build. Because it spreads many parts of the organization, everybody hates them fractionally in their own special way.

I think that's part and also storage has gotten so cheap. We can just kind of forget about dimension tables. We just repeat our data because we have columnar databases so that's not near as costly and storage is cheap. We end up doing lots of big flat tables stuck there that are useful. We just think less about data modeling and you'll know when you're thinking too little about data modeling and you start sucking and you're like maybe we should have given some thought to data modeling and you go back and do it and we're all figuring this out sort of emergently and organically. Hope that answered.

How leaky is dplyr?

I think we've got time for one last question. I'm not sure whether this is for you or for me but I'm going to give it to you. How leaky do you consider something like dplyr?

I literally told Davis at dinner two nights ago that the thing I like about dplyr is for SQL-like transformations. So kind of the normal stuff you might do with SQL. I consider dplyr as like an appliance. One of the definitions of an appliance is it just does what it's supposed to do. You don't think about it. You're dishwashing. You don't think about how it works. It just works. Dplyr for a lot of that sort of stuff I think it's very unleaky. I think if you start trying to do data frames inside of data frames or whatever that foolishness is you let people do Those probably leak but I don't know because I don't do them.

It's really good for a bunch of stuff. You guys have slowly morphed it over time like how you change the cast and melt to the pivot longer, pivot wider. That's been actually that has taken leakiness away and taken friction away. You can change an API but it's really helpful in reducing leaks. I consider dplyr not particularly leaky. What do you think about that? How would you answer that?

The thing that I think about the most is dbplyr. dbplyr leaks like a sieve. Yeah. We are trying to translate our code to SQL. Yes, it leaks at some level. It leaks like a sieve but amazingly it works. Yeah.

I think the thing that's most surprising to me is most of the stuff you're trying to do to data like filtering and aggregation you can do it in R, you can do it in JavaScript, you can do it in SQL, you can do it in Python. It's all basically the same and we can translate between that with pretty high fidelity. That's pretty surprising. It really is kind of surprising.

Where it leaks is kind of akin to how SQL leaks in that you start accumulating the steps and then at some point you get unexpected behavior. That's not unique to dbplyr. That's a little bit of the underlying SQL. I have the same problem in SQL. Then you have a permeating the membrane problem because at some point you go, fine, dump the query out. I dump the query out and you say, oh God, what have I done? I have my DBAs tell me I can tell when you're doing a db query because it's got really crazy SQL that no one would write. So when it comes time to solve some of those problems, some of them aren't yours, some of them are SQL, but it can be a little tricky. It's harder.

Yeah, that makes me think a fun talk would be. We could spend probably 30 minutes just talking about how leaky and abstraction SQL is. Oh, we really could. I was tempted to jump into that, but I was like, no, I want a super succinct example and floating point was just so succinct. Thank you all very much. Thank you.

Closing announcements

Before we go for lunch, just a couple of quick announcements. This is kind of the last time we're going to be all here together in person, so I just wanted to start by saying thank you.

First of all, I really wanted to thank all of our speakers. They have been working on their talks in conjunction with our training partner for like weeks now, and I think you can really see the results of that and the uniformly high quality of the talks this year. I'd also like to thank all of the sponsors. If you haven't stopped by the lounge already, make sure to do that. Most of them have got booths there. Find out the cool stuff they're doing. They're here at Comps because we believe, and they believe, that they can help you solve your problems.

I'd also like to give a big thanks to all of my colleagues

at Posit, the colleagues who are here both here in person solving the many logistical problems that have occurred so that you never experience them, and to the many employees who are also working online as well. In particular, I'd really like to thank Caitlin and Brian who run our events team. They have done just such a fabulous job at Comp this year.

And finally, I'd like to thank all of you all. Obviously, the Comp wouldn't happen without your enthusiasm and love for R and Python and data science, and it's been so wonderful to meet so many of you here in person. Thank you.

Announcing posit::conf 2024

Now, you're probably getting sick of clapping and you really want to hear the next thing I'm going to tell you, and that is, where is Comp going to be next year? And that is Posit Comp 2024, or in an effort to embrace our Python colleagues, we're also going to call it From Posit Import Comp, Comp 2024. But seriously, it's going to be in Seattle next year.

If you look closely at those dates, you might notice it is one day shorter than usual, so next year we're going to be experimenting with just doing a single day of workshops. We're trying to bring the overall cost of Comp down. We're decreasing the ticket prices a fair amount, and also just making the whole thing a little bit cheaper by reducing it by one day.

I also have an apology to make to the members of the vampire community who are with us this year. We're going to have many windows next year, and there will be a lot of daylight as well. So as always, the cheapest possible price to get a ticket is our Superfans price. That's $6.99. That's going to be on sale until October 6th. We'll be sending out an email to everyone who registered in the Conf in the next couple of days, so you can sign up for next year if you want.

So thanks again. Please enjoy the rest of the conference, and if I don't get to see you or meet you personally, I really appreciate you being here, and good luck with data science and the rest of your life. Thanks. Thank you.