image: thumbnail.jpg
People over computers: engineering leadership | Elliot Murphy | Data Science Hangout
We were recently joined by Elliot Murphy, a VP of Engineering at Posit, to chat about engineering leadership philosophy, balancing maintenance versus innovation and risk, non-traditional career paths and the value of lifelong learning, and data science tool breakdowns including Posit Cloud, Posit Connect Cloud, and shinyapps.io.
In this Hangout, we explore some philosophical approaches to engineering and management, where Elliot stresses that focusing on people is more important than computers, and human relationships tend to outlast relationships with a given company or employer. Elliot discusses his role as primarily listening and synthesizing the brilliance of others, creating a supportive environment, and making space for people to take on risk and propose new ideas. He also highlighted the value of clear communication when addressing high-pressure environments (like when you're sunsetting a software product).
Resources mentioned in the video and zoom chat:
🔗 Cognition in the Wild (Book discussed regarding joint cognition) → https://mitpress.mit.edu/9780262581462/cognition-in-the-wild/
🔗 datapasta (R package) → https://github.com/MilesMcBain/datapasta
🔗 AI Capabilities in Positron Workflow Demo → https://pos.it/workflow-demos
If you didn’t join live, one great discussion you missed from the zoom chat was about the surprising value and broad resources offered by local libraries, including free access to services like LinkedIn Learning, paid periodicals like the NY Times, and streaming platforms like Kanopy. Attendees praised libraries as the "best resource we hardly hear about" and key providers of freedom of information. Do you love your local library?!
â–º Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
Hangout: https://pos.it/dsh
LinkedIn: https://www.linkedin.com/company/posit-software
Bluesky: https://bsky.app/profile/posit.co
Thanks for hanging out with us! 💛
Timestamps:
00:00 Introduction
02:04 "Can you tell us a little bit about your journey to where you are?"
04:10 "What does a VP of engineering do? And what types of things are important to you from a philosophical standpoint for engineering and management?"
06:21 "What about your job lets you focus on the people?"
06:33 "Is there also a pressure to innovate?"
09:28 "How do you argue to leadership at a strategic level that that maintenance work is important to take up people's time?"
11:48 "How do you balance engineers and data scientists wanting to incorporate best practices with fast prototyping?"
16:46 "What distinguishes science from engineering in data and also more broadly?"
18:28 "Any resources or any books that helped you bridge the gap between a non traditional background and your current position?"
23:00 "What is the secret to creating a good environment for the team?"
25:23 "Are there basics that you feel analysts or data scientists should be aware of in terms of engineering that would help their work?"
28:47 "How do you make decisions about when to either sunset a project or change directions?"
31:20 "What can those doing maintenance work do to add value to the organization to be safe from being displaced by automation?"
35:02 "What do you think many data scientists or analysts are missing about engineering infrastructure today that could simplify their day to day workflows?"
35:52 "How can a professional transition from a business intelligence role to a data scientist position?"
43:47 "What is your mantra or best practice to crack these professional certification exams?"
45:22 "Tell us the difference between Posit Cloud, Posit Connect Cloud, Posit Connect, and shinyapps.io."
49:03 "How can a team avoid the trap of over-engineering?"
Transcript#
This transcript was generated automatically and may contain errors.
Hey there, welcome to the Posit Data Science Hangout. I'm Libby Herron, and this is a recording of our weekly community call that happens every Thursday at 12pm US Eastern Time. If you are not joining us live, you miss out on the amazing chat that's going on. So find the link in the description where you can add our call to your calendar and come hang out with the most supportive, friendly, and funny data community you'll ever experience.
Can't wait to see you there. Well then let's go ahead and introduce our featured leader today. I am so, so happy to be joined by Elliot Murphy. Elliot is a VP of Engineering here at Posit. Elliot, it's so nice to see you today. Could you introduce yourself and tell us something that you like to do for fun as well? Yeah, so I'm Elliot. I live in Florida on a little island on the east side of Florida, which is a pretty boring little island except that NASA is on the top half. And so I can actually feel the windows shake when the rockets go up every week. Blue Origin, SpaceX, Boeing, all of those rockets. And something that I like to do for fun is just be by the water. So there's dolphins in the rivers and canals back here, and I just like to be outside with nature.
How often does that happen? How often do rockets go off and the ground rumbles? It's easily every week. Sometimes it's every day depending on the launch schedules. That's really surprising to me. I think of launches as things that happen very, very rarely. There's some big ones that get some press, right? For sending people to the space station. There's a lot of commercial activity as well, putting some more internet satellites into orbit.
Elliot's background and journey
Wow. Okay. Well, it would be great to get some context about what you do as a VP of engineering, but also a little bit about your journey to where you are. And I'd love to talk about that because you have had a really interesting path. You did not take the traditional go get your undergrad and your master's. Can you tell us a little bit about your journey? Sure. Yeah. I owe a lot to my mother and grandmother. My grandmother actually got me a computer when I was eight. So I started learning the program when I was eight years old. And my mom wouldn't let us have a TV. And so that was what I did for fun. And I worked at a lot of different software jobs over the years. I tell people that I'm a lifer. I'm going to keep building software for my entire life because I really, really enjoy it. But yeah, I tried and couldn't quite convince myself to finish college. I dropped out several times, but have just spent a whole career building software for different people.
And I've been at Posit now about three and a half years and really, really enjoying the people here and the mission. For a while, I worked on more general purpose systems. I worked on databases at MySQL. I worked on Linux distributions at Canonical. I worked on Ubuntu. And I loved that. But then I wanted to work on tools that were a little closer to people. I've always worked on tools. And so I absolutely love getting to see all the ways people use the data science tools that we produce here at Posit.
I love that. I have more questions for you, but I'm getting the feedback that you sound a little bit quiet. So I was wondering if there's any way we could either everybody, maybe I could turn myself down. Hold on. I'll move my mic a little further away. Maybe Ellie and I can be close to the same volume. I'm not sure. Everybody let us know if Elliot sounds a little bit better now.
Engineering leadership philosophy
Okay. What does a VP of engineering do? And what types of things are important to you from a philosophical standpoint for engineering and management? Yeah. So I think any engineering leader is really responsible for assembling and looking out for the team. And so at Posit about half the company is engineering roughly. We've got that half of the company divided into four groups. And I support one of those groups. So all of our teams, the teams that I'm working with directly are focused on cloud and online services. So Posit Cloud and Connect Cloud and Academy, those kinds of things.
And I have always tried to, I like to say that people are more important than computers. The technical expertise is super important and you have to be fluent there. But every single time that you can try to create a good supportive environment for the people, the rest of it is a lot easier to deal with. A lot of the systems that we are working with, are working with, there's some pressure to keep them up, to keep them running. So it can sometimes feel like a high pressure environment. And so focusing on the people relationships, it's weird, like over the years, I feel so old saying this, but like the relationships with people tend to outlast the relationships with a given employer or a company. There's so many companies that I absolutely loved working at that no longer exist, even, but I still have relationships with some of the people that I worked with there. So the human relationships are super, super important. Maybe that's part of working in technology is technology is just fast moving and fast growing and evolving. And sometimes that means that things grow and burn out over time much more quickly than in other industries, right? Where insurance company is going to be around for a hundred plus years by the time you can get to it, which is my experience. I've worked for a hundred year old companies.
And I have always tried to, I like to say that people are more important than computers.
Maintenance versus innovation
What about your job lets you focus on the people? So I would love to talk about maintenance for a second, because you said like, okay, we have this pressure to keep things running. Is there also a pressure to innovate? And is there, are those competing pressures between maintenance and innovation? There's always pressure to innovate. There's always interest in, in innovating and you want to find the new thing and you want to, you want to accept that, right? Like it's important to account for and include new technologies, better ways of doing things. But it's easy to, I think it's easy to forget about the value in maintenance. When we were talking the other day, we were talking about like the road system, right? And like, you just count on the roads to be there, but so many, so many nights and weekends, there's crews that are maintaining the roads and like that has to happen every season. There's so much infrastructure that we, that we count on. So I really look for ways to give ourselves credit for doing that work and to kind of celebrate that maintenance work that enables everything else to run smoothly.
There is, there is something that is really interesting, which is like, there is so much innovation that has happened, that seems to work really well, that hasn't yet been put into practice. That's what I wanted to talk about. Yeah. Cause you, you have some stories about this where you are, you're like, I have seen innovations that should be improving people's lives that are a decade behind an actual application. Yeah. Yeah. I used to work and absolutely loved working there at a behavioral health and mental health startup. And the state of the art in mental health care was 12 years ahead of the common practice in clinics. And there was so much value in bringing up, you know, something from five years ago and making it available to people. And so I think, I think there's just a huge amount of value in identifying the good ideas from last year, the good ideas from five years ago and helping more people have access to those.
Yes. And this translates for me. So exactly to data science work and engineering work, because I, every time I mentioned something on blue sky or LinkedIn, or here on the data science hangout, that is a package I've known about and used for years, at least 10 or 15 people will go, oh my gosh, I had no idea that existed. I had no idea that how long has this been out there? Right. The data pasta package is a big one. Every time I talk about data pasta, I get a wave of people that are like, wait, what? You can do that? I'm like, yeah, apparently you can do this. And it's existed for years. Miles's package is amazing. But there's so much like that.
And I feel like the, the maintenance work that happens, it has to happen. And we have it in data science as well, especially around documentation. But the push is always to move on to the next thing. How do you argue to leadership at a strategic level that that maintenance work is important to take up people's time, right? Because we, we look at this dev team and we're like, oh, they're doing too much business as usual, BAU. How do we reduce the BAU? And I know that you have told me efficiencies can be found. You can sort of outsource some of that maintenance, but there's always a trade-off cost, right? There is. And I'm, I like to say I'm a fan of reality, right? So like, let's get real about what benefit we're actually getting and what, what the cost is of maintaining something. Even if I think about, you know, around the house I've got a shed or, you know, something that I get some value out of, but then there's a maintenance cost also. And it's not always like I pay a certain fee every month. Sometimes it's a little bit lumpy, like you have to do something every year and you can accumulate, especially easily in software, like more systems and more things and more vendors that you're using and more software that you've built and is now kind of sitting there supposedly running quietly. But all of that represents like this growing maintenance burden. And so I really, really like thinking about it from like a budgetary perspective, like we're spending this much time on, on these systems. And do we want to seek efficiency on that maintenance or do we want to actually remove some of those systems completely and consolidate? Or do we want, do we think it's well worth it and we should actually increase that, the maintenance, because we're getting so much value out of that when we can see how we can get value for years to come out of it. So people have been so supportive, you know, in other parts of the business at every organization I've, I've been at, when you're able to, to participate in the conversation, thinking about the health of the overall business and accept that there's trade-offs and just involve them in the conversation. It doesn't need to be like this technical debt is so important. It really should be about the health of the overall business.
Balancing best practices and risk in innovation
Well, this is a great topic to bring Javier in because Javier had a question in the Slido and I would love him to ask that live so he can sort of add details around it. Javier, are you available to ask that live?
There he is. Hey, yeah. My composite shirt from a few years ago, so. Nice.
Yeah, my question was basically around, I feel like there's a balance between engineers, you know, data scientists wanting to incorporate best practices in everything they do, but sometimes that is like a monumental effort that feels like a lot of innovations that come out of like very fast prototyping or tinkering, or, you know, more of like a minimalist type of production style, like, or development style. I haven't, I don't have a good answer for this. Like, I haven't found what the appropriate balance is. I have seen big sort of best practice projects just go like off the rails. Like now we've created something so complex that what we thought was going to make it easier for others is maybe even more challenging now. So, you know, kind of an open-ended question for you.
Yeah, there's, I could talk about this for, for days. I love the question. I think it's really important to, and this is not something I came up with, right? This is, this is just something I've, I've seen over time, to be explicit about wanting to take on risk. It's, it's funny business, business owners and leaders understand this very well that most profit in business comes from risk. So it's, it's not that they're afraid of risk. They just don't want to take foolish risk. And especially in engineering or in software development, it can be, or security or, or any of those things, it can be easy to think about removing risk because a lot of our daily work is about removing risk and standardizing things and like making them more predictable. And it can be easy to forget the value of the risk.
And so I think about this as like a dev environment versus a production environment. In even the most risk-averse organizations where lives are at stake, you have exploratory and research environments and you want to create those and, and, and, and encourage and provoke risk-taking in those environments. And then you just want to, I think a, a reasonably structured way of identifying which are the things that you want to move out of that environment into a safer, more predictable environment. And what are the, what are the additional investments that you want to make? So you don't want to make all of those. Dave Snowden likes to talk about like safe to fail experiments versus fail-safe experiments. So you want to do some things very, very cheaply because you want some of them to fail or you're not exploring enough. And then once you've identified some things that have really good potential, then you want to spend the extra time and money to sort of make them safer or more predictable and so on. So I think even the organizations that like have this identity that they carry of being extremely risk-averse spend a lot of time trying to find ways to invest in innovation in certain parts of the business. And if you can label it correctly, it can go a
Dave Snowden likes to talk about like safe to fail experiments versus fail-safe experiments. So you want to do some things very, very cheaply because you want some of them to fail or you're not exploring enough.
long way towards getting permission to do those things. Yeah. I love, Elliot said to me the other day, there is dignity and risk. And I feel like that belongs on a pillow somewhere, maybe a little needle point. Uh, there's dignity and risk. I often think about parachutes when I think about risk. I come from like a line of pilots and people who had parachutes and I grew up with parachutes all over my garage as a kid, right? And whenever I do something risky, I don't want somebody on my team who's going to take away my parachute. I want them to pack and repack my parachute several times for me so that when I jump, I'm probably going to make it to the ground in one piece. So the person who's just going to take away your parachute, uh, is less helpful, right? Help me take this risk in a safe way.
I want them to pack and repack my parachute several times for me so that when I jump, I'm probably going to make it to the ground in one piece.
Um, Javier, that question was fantastic. Thank you for asking. And there's a poll up in Slido. If you would like to go weigh in, um, on that at the top, there is a little button where you can switch between Q and A and polls. You can go answer that question. And it's, do you have a development versus production environment at work? Do you just have production or do you have like dev test prod? Um, the majority of people are saying, yes, they do have dev test prod, but the number that is saying no, as I'm watching, it is going up the percentage. I have worked at several places that have just had production, absolutely no safety or guardrails at all. Um, okay. I had a question from Noor that was fantastic. Noor, would you like to hop in and ask your question live? Cause I love the philosophical take.
Hello. Can you hear me? Yes. Hey, okay. This is more of a sort of broad question. Uh, but, uh, and I said, this is more of a philosophical question, but for you, what distinguishes science from engineering in data and also more broadly, or do you feel it's more of an org specific, uh, specification on that level of granularity?
Great question. Um, I've, I've only really experienced as at like an org specific ways. Um, when I was working at the healthcare startup, um, science was what they did at the university and, um, engineering was what we did taking some of those papers and, and put, you know, implementing them in code. Um, but I don't, I don't really have a very sophisticated take on it. It's, it's sort of an arbitrary distinction in my mind. I guess, I guess maybe it's, it's, um, if you think, if you think about the, the daily experience of the person who's, who's doing a certain type of work, one is probably much more exploratory and the other one is probably much more about bringing something to an audience. Um, that, that is sort of already a well-formed idea.
All right. Fantastic. Thank you so much for your wonderful question is always, um, and I had a question that was from direction. And I think this is interesting. I would love to hear your take on it direction. Do you have the ability to unmute? Let's find you. Where is she?
I think she's unmuting right now.
Hello. Hi, I hope I'm audible. Yes. Okay. So my question was that, uh, that elitist from a non-traditional background. So any resources or any books or anything that helped him bridge the gap between a non-traditional background and where currently the position he's working on, right. There has to be something like any reading habit or any practices, these kinds of things apart from coding, right. There are things that soft skill that, uh, in combination with data, they like bring the best. So yes.
Yeah. So I mentioned that like, I owe my career to my, my grandmother and my mother. My grandmother bought me the computer. My mother got me a library card and brought me to the library every single week where I would, I was homeschooled and I would bring, um, the maximum amount of books you could check out from the library every single week. And, um, as I'm sitting here and it's tidy on the camera behind me, but, um, next to me, I, I literally have four foot high stacks of books because I don't have enough bookshelves in here yet. I just moved, moved houses a couple of months ago. And so, um, I have tried to fill that gap with reading, um, just compulsively is probably the right, the right word for it. Like my whole, my whole life since I was a kid, I've just read as much as I could.
Some of the stuff that I have, um, enjoyed the most has been reading, reading books about, um, complexity and also a little bit of, of stuff on, um, like joint cognition. So I, I, I can barely even describe the ideas. I'm not like a smart enough person to keep up with the ideas in these books, but the, there's this book called cognition in the wild that was really fascinating to me. And it's about the, it's about this idea that, um, people working together in a system can form this super brain and no one person in the system knows all of the parts, but together there are decision-making activities happening and there is value being created by, by the group working together. And I just love this idea so much that you can not know everything and you can still contribute to progress. And, um, even, even with like a flawed understanding of the world and incomplete set of knowledge, you can still contribute to progress.
And I just love this idea so much that you can not know everything and you can still contribute to progress. And, um, even, even with like a flawed understanding of the world and incomplete set of knowledge, you can still contribute to progress.
And one of the stories in that book is, um, some research that was done on Polynesian navigation, where they have these, um, this, this mental model of geography that doesn't match with what we know today from science, but was still useful in being able to navigate to far away islands. And, and there was this concept of like an ETAC of birds. So there's like these ETACs are segments of the journey. And with very imprecise navigation, they were like trying to find an Island that was far away. Well, the birds leave the Island every day and go look for food. And so they go about 30 miles away from the Island. And so as, as long as you got within like 30 miles one way or 30 miles the other way, and you saw a bird, all you had to do was just chill out and wait for the sun to go down. And then the birds would fly home and you could follow them in the last 30 miles. And so like that, that idea has been so incredibly helpful, um, in like trying to work up the courage to, um, work in these different places without like a fancy education and like often feeling inferior to the other people in the room. Like just that idea of like, if you can get the direction right, um, and you can participate in a good group of people, uh, you can, you can still help. You can still help make progress, even though there's an awful lot that you don't know or understand.
What a fantastic question. I was wondering if we were going to get the ETAC of birds into this conversation and we did it folks. Okay. Um, I put a Slido question into Slido. I don't know how it's going to show up for you on your side, everybody, but, um, it's what books have shaped your work today and why? Oh, of course. The first answer is Statistical Rethinking by Richard. Um, I always say his last name wrong. Mackl, Mackleridge. Somebody can correct me if it's wrong. Um, so everybody go leave your books that you love there and why I would love to see them. And I think everybody can read everybody else's results as well. All right. Direction. Thank you for that question.
Creating a good environment for the team
And I would love to move on to an anonymous question that I just love. It's what is the secret to creating a good environment for the team? You mentioned early on that your part of your job as a VP of engineering was to take care of the people. How do you do that?
Yeah, I don't, I don't know if there's part of it is giving them the test versus prod environments to play in and safely fail. Right. Are there other, some other things that you consider, um, prioritizing? I, I do as much as I like, I don't, I don't feel like I have a great answer to this. I spend as much time as I can talking to the people doing the work every day and trying to experience a little bit of the work for myself. Um, I think of my role as mostly like listening and synthesizing all of the brilliance from other people. Um, so it, it's less about shaping it and more about like letting the good things that people want to do happen. Right. And endorsing, endorsing things, um, making some space for people to take on risk, making some space for people to put forth proposals and ideas. But, uh, yeah, it, the team is, is the one who does all the hard work and like comes up with the good ideas. I, I'm, I'm just here to support.
I think you gave the best answer that you possibly could have given because my answer to everyone's question they ever ask me is to talk to people. And your answer was, I talk to them. I listened to them and I love that so much. Thank you. Anonymous question. Asking your question was great. Um, okay. Let's take a look at Slido here because my little queue of things in my brain are being interrupted by the answers that I'm reading of these books. Um, because someone said ISLR, fantastic. 10 out of 10, one of my favorite, favorite books. Um, but there is an anonymous person who left a whole list. So everybody go check out what some of those are. Some of them I recognize like, um, extreme ownership and leaders eat last, the Simon Sinek book. These are great. Thank you. Whoever you are, anonymous person.
Um, okay. Let's hop over to our questions again.
Engineering basics for data scientists
There's a not anonymous one that says, are there basics that you feel analysts or data scientists should be aware of in terms of engineering that would help their work with supporting and collaborating with the engineers on their team? And I really like this one because I feel like in data science and analytics, um, oh, and this is Jeff Pennell, not, not, not anonymous. Jeff, feel free to pop in here and give me, give us some more context. And this is a great question because we often are on the statsy side and we feel like we're not engineers.
Yeah, sure. No, I, I just had forgotten to log in when I put the thing, like, I should probably put my name in here. Um, no, this kind of came about actually recently with, um, I was having a conversation with some of the, the engineers on our team, um, actually around, uh, shiny because I was doing some work with Cardo and then somebody kind of up the food chain. They really wanted, like, can you make this interactive? I'm like, well, I can, but we don't really have a shiny infrastructure. We use Tableau for a lot of things. And so I started having those discussions with our engineering team and just like, what would it even take to build something like that? And kind of like the gap in my knowledge between what they would need to do to do sort of like an internal implementation versus, you know, kind of what these visions I had in my mind. I was like, okay, well, what are some of the basic things that we should even keep aware for kind of working in these collaborative environments?
A great question. Um, we could list off a million DevOps skills and it'd be amazing if everybody in the world learned all of those things, but I don't think that's realistic or fair at all. So the, the thing that I kind of want to say is, you know, so the thing that I come back to, and I think this is important for, for scientists or analysts or business people or designers is to get comfortable with, um, Git and version control. And it sounds so simple and it's definitely got its moments of being annoying to work with, but it's such an incredibly powerful skill that is very relevant, even more relevant with all of the AI stuff happening. It's relevant, um, no matter how sophisticated or how simple, um, your deployment environment is, um, even if you only have a single environment, just adding version control to it can still get you a lot of the benefits and make it a lot safer. Um, and even if you're working solo and alone, um, being able to try a couple of different things and look at the notes you had going back to, to last week and last year is really, really powerful. So like that, that would be my one recommendation.
When to sunset or redirect a project
Fantastic. All right. Thank you so much for your great question, Jeff. Let's hop over to Arsenis. Arsenis, you had a great question in the Slido. Would you like to ask that live? All right. Thanks, Libby. Thank you for, um, joining us and kind of chatting about this. Um, yeah, my question is, um, about projects that I, I think of as Frankenstein's monsters, what they become. So you start a project, um, and it becomes something that's, you know, maybe through a scope creep or whatever, it just kind of becomes much bigger. It blows up, but you realize that it's kind of lost the thread. It's kind of gone off the rails in some ways. So how do you make decisions about when to either sunset a project like that or, or change directions, especially in the, you know, in recognition that some of these projects might affect external users, things like that. So, uh, to, uh, for a little more context, I'm thinking of projects like a code base that's has a lot of patches or whatever, or a database that has lots of data sets that have, um, you know, that overlap things like that. So, yeah.
Yeah, it's not, it's not easy. And, um, I think the reason it's not easy is because of, of realizing that you're going to disappoint some people. You're going to, and it doesn't matter which decision you make, you're going to disappoint some people if you stop supporting it and turn it off. And you're going to disappoint some people if you keep struggling down this path, which is maybe not worth it. Um, so I, I think that's what makes it hard. Um, the only, the only thing I know to do to try and deal with that difficulty is open communication, try to describe why it's hard or try to describe why it's expensive and try to describe what the benefits seem to be or what benefits seem to be missing.
Um, and, and put forth, try, try and put it in like understandable terms.
There's still going to be people who are upset that it's turned off. I had to shut down an entire product line at a, at a previous company. Um, and we were, we were making like $500 a month on this service and we had maybe six or seven companies that were using it and they really liked it. They were happy with it, but we couldn't get more. And so we were losing money on it every month. And I actually grappled with it for about a year before finally just like getting, getting, um, brave enough and confident enough to go like, listen, we're going to shut this down. We're going to shut it down on this timeframe. Here's the best recommendation we can make to you for where to go instead. And some people were still unhappy about it, but, but just clear communication was the best thing I could think of to do.
Fantastic. Clear communication is a good, um, piece of advice, no matter what area you're working in. Well, that was a fantastic question, Arson. I thank you so much for asking it.
Maintenance work and automation
I had one that kind of went back to maintenance work. It's from an anonymous asker. Um, and it says for those doing maintenance work, like monitoring and already existing data science, modeling pipeline work, um, what can they do to add value to the organization to be safe from being displaced by automation? Um, and we had a reply on here that says, get involved in other people's projects and developing the next skill before anyone else, like high performance computing and AI. What do you think about this, Elliot? Um, do you feel like people doing maintenance work are in danger of being displaced by automation and what can they do to be safe from that?
What can they do to make sure they're adding value? I think it's very valuable to do maintenance work. I think it's valuable to do maintenance work.
I think the ability to go into a legacy system or an older system and learn how it works and identify like how to keep it running is incredibly valuable. And then sometimes the gap is in, um, making that visible to the rest of the organization or sharing the knowledge. And you actually do want to be replaced by automation, I think. Um, and so the best way to do that is to be the one who's replacing yourself with automation, identifying ways that automation could be used and documenting what you've learned. So like it comes back to that communication again, right? So like if you understand a legacy system and you're doing some maintenance work, um, documenting what work is required and what investment would be required in order to reduce the maintenance cost over time makes you an incredibly valuable participant in the business, right? And so then there's another system you could go do that same work on, and you could, there's other areas of the business that you could investigate. So I think if you are good at maintenance work, you can be incredibly valuable.
So I think if you are good at maintenance work, you can be incredibly valuable.
And there's so many systems out there, right? Like if you look at the lifetime cost of a technical system, including software and people, right? Most of the cost is in the maintenance phase of it. And so being good at that part of it and understanding what it needs to be kept up and running and documenting and sharing it, I think makes you super valuable.
Do you think that part of the advice here is to learn about just how valuable your role is so that you can talk about your value and then also being vocal about what you're doing? I think that Rachel and I have talked about this before on previous Hangouts as well, where like, we've had to learn the lesson over time and explicitly that good work does not always speak for itself. You have to speak for your work. For sure. Yeah. You have to let people know what you're doing and remind them why it's valuable, even though you think that they should just kind of implicitly know. They probably don't.
Yeah. And there's so much trust that is earned when someone comes and says, hey, we should get rid of this system because we could do this thing and that thing and then combine it with this other system. And when you're talking with that person the next time and they're like, we shouldn't change the system, it would be way too expensive and difficult and not worth it. We should preserve this existing thing. You trust them a lot more because you know, they're thinking about the big picture. And so anytime that you can be talking about the big picture really, really, I think, increases your ability to get credit for your work.
Fantastic. All right, everybody, let me know. Let me know in the chat if you feel icky about talking about your work ever. It's been a barrier for me sometimes where I'm like, oh, I feel like a show-off talking about my work. But I think that you can show people in a learning and teaching way much more than a show-offy way.
What data scientists miss about engineering infrastructure
All right, we had an anonymous question that was, what do you think many data scientists or analysts are missing about engineering infrastructure today that could simplify their day-to-day workflows? So this is similar to our other question, but specifically about the way that data scientists or analysts work. That's a great question. I don't... And this can be to everybody in the chat if you have thoughts, let us know. I don't think I have an answer for this one.
I have a question in Slido as well that is a little bit more about data science. And I would really love to put this out there to the group and let's talk about it. So this question is, and I will paste it here.
Here we go. Anonymous question. All right, this question is, how can a professional transition from a business intelligence role to a data scientist position? Even with a master's degree and a background in data science, it can be difficult to make the switch. What type of projects or work do you suggest get a data scientist position nowadays?
I would love it if we could talk a little bit about this. Renato says, let go of Power BI.
We had an anonymous reply that says, try and become competent in a data science language and try to show that competence by developing a GitHub repo, even if it's just building a shiny app, using APIs and demonstrating a range of end-to-end knowledge, which is great advice. Which is great advice. Yeah.
Elliot, do you have any thoughts for us on this? I know that you are in engineering and not specifically data science, but you build tools for data scientists. And I'm involved in a lot of hiring. And I think this is really a question about how hard it is to know about good opportunities and how hard it is for people to hire other folks into those opportunities, right? It's kind of a matching problem. I know when we posted a job opening recently, within hours, we got a thousand applications. And most of those were spam applications. And it was actually very hard. At Posit, a human reviews every single application. And it was very, very hard. We actually were trying to write some custom software to help us filter applications. And we put annoying questions in the application process, trying to help us filter out bots and AI submitted applications. Completely made up profiles.
So, what you're talking about is being qualified, having a good background, and having trouble getting noticed. So, there's kind of two different phases of the job. Once you get into the interview process, maybe if it's an exhaustive interview process, you've got 10 hours with different people at the company, right, of conversations. And then they have to decide, you know, a significant investment in choosing between a whole bunch of people who've applied based on very minimal information. So, I like this reply of, like, try to become competent in X and show that competency publicly, right? So, I'm abbreviating some of it. That is so powerful. Because once you get past the problem of, like, getting to the first interview, having something you can show to demonstrate your competency makes the decision so much easier. These are just other humans who had to talk to, like, 50 applicants for 45 minutes each and try and make a semi-intelligent recommendation on which person to hire. So, showing something publicly is a really, really powerful way to tip the odds a little bit more in your favor.
So, showing something publicly is a really, really powerful way to tip the odds a little bit more in your favor.
It's not a guarantee. And it's a lot of extra work that is probably not fair that you have to do. But it's a hard problem. It's a mess right now, like hiring on both sides. And I think that is reasonable advice to try and improve your odds a little bit. It doesn't feel fair that you have to do that, but...
It doesn't. But also, the advice, again, that I will, like, navigate us back to is talk to people, right? You said that relationships often outlive companies. I think that that has a really big impact on getting jobs. I had asked, I think in another data science hangout a couple of weeks ago, or maybe last week, one of the questions I asked was, have you ever gotten a job on the hidden market? Like, have you ever gotten a job that was not posted somewhere? And a lot of people said yes, including me. And I see Renato is shaking his head right now, you know, like camera on crew is shaking their head. Zachary is shaking his head. John's shaking his head. I will raise my hand there too. I've gotten a lot of positions in my life that never, ever saw a job board. They were a conversation. They were a relationship. They were an awareness that I existed. And that would never have happened if I hadn't talked to people and done what I liked to do out loud.
Working out loud and speaking in public
Do you have any stories about working out loud, Elliot? Actually, I feel like you do a lot of talks in places. I remember looking at your LinkedIn and seeing like pictures of you speaking. I'm like, Elliot must do a lot of speaking. Yeah. So like, that kind of connects back to this question about, so I was actually laid off from the healthcare startup because we ran out of cash. And so I started a consulting company and I started giving talks at conferences because that was the most effective way to demonstrate expertise and would get a lot of work from people coming up to me afterwards going like, oh, can you help us with this problem that you were talking about how to solve, right?
I think that is another option for breaking into a different career or to demonstrate competency in public is to like give talks or do consulting work. It kind of lowers the stakes a little bit. You're not asking for like an entire full-time job. You can start working with multiple companies. And I think you'll get a lot of additional information about what it's like inside some of those companies. And maybe you don't want to work at some of them, or maybe you really do, right? Like you'll learn something.
What a good piece of advice. I was just talking with, I run a small data science freelancers community because I used to be a data science freelancer and my heart is still very much there with that community and those people. And one of the questions I posed in our last meeting was, what are the tangible benefits you have gotten from speaking? Because speaking, especially giving a data science talk requires so much effort. It's like months of work and you agonize over it and you're creating these talks and these slide decks and you're practicing. If anybody here has given a talk, let me know how long you worked on it. And if you worked on your slides up until the moment you were on stage, right?
And is it worth it? And everybody kind of like stopped and was like, well, no, I don't know if there are any benefits. And then I started watching light bulbs go off and they were like, oh wait, yeah, there have been. I met that person at that thing after I gave that talk. And then I ended up having a contract from them. Sometimes the downstream benefits are a little bit further away. So my advice would be if you have the bandwidth to start getting involved in giving talks, even if they're just lighting talks, do it now instead of way down the road when you're in a dire situation and really need the exposure fast, but do it either way.
All right. That was my soapbox moment. And I'll get off it now.
You don't need permission from a conference organizing committee to give a talk. Last night, I was watching a five minute video on YouTube from someone explaining how to like adjust pool chemicals. Cause I'm trying to learn how to do that, right? So, you know, something that other people don't know, regardless of where you are, what your background is, and you can share it without needing to get permission first.
Yeah. As a piece of career advice, I guess, how do you help people or yourself get over that feeling of no one cares about this. Nobody asked me and everyone probably already knows this.
I don't think it ever goes away, right? Those feelings are still there for me. You just do it anyway, right? Yeah.
Do it anyway. Do it scared. I know that's tough advice sometimes.
Professional certifications
Okay. Well, I saw one more question that was in Slido really quickly. Let me go grab it. That is about your many, many certificates. It says, I see Elliot managed to complete 10 plus professional certifications. What is your mantra or best practice to crack these professional certification exams, certification exams? Well, I definitely do not want to get an A on any of them.
So if you get a high score, you studied too hard, you wasted time. Okay. And, and, you know, I've, I've done a lot of reading over my life. And, and so, um, I, I read the practice exam and then I take it. I have also failed multiple times, um, taking these certification exams. I don't think the exams like are a super great, like proof of expertise. I think they're a useful communication shorthand to make other people feel more confident that you could pass like this, this testing hurdle. Um, and so I've really tried to minimize the amount of time I spent on them. Um, I'm, I'm not looking to like get a high score on the exam. I'm looking to get the lowest possible passing score.
That's great advice. Uh, and I would never take it because I am a try hard. If there is an a, I am going to get it. I'm going to blow it out of the water. Uh, tell me in the chat if you're like me and you just go too hard all the time. Grad school was like 14 hour days.
for me. This is why I don't have my PhD because I thought, can I do it in a healthy way? No, so I'm not going to do it. Um, okay. There's a question that came up about, uh, technical stuff. Are you up for this one? It's tell us the difference between posit cloud, connect cloud, and posit connect and shinyapps.io.
Yeah. Um, so there's this, there's this like saying that, um, software reflects the shape of the organization that it came out of. Um, I think if we were building everything today, we would just call it posit cloud and it would just be the set of data science products that you get in your browser. Um, but they were built over eight or 10 years, um, a different piece at a time. And so that's, that's why they're still a little bit fragmented. Uh, I don't know if I should say this in this forum, but I will, um, at one point earlier this year, we had six different publishing platforms that posit was operating. Um, so, so we're down to five and my goal is to get us down to one or two. Um, it's still going to take another couple of years to get there. Uh, so, so today just like to actually answer the question, posit cloud is where you can get our studio in your browser. There's, there's free accounts. Students use it. Sometimes there's also some paid accounts. Uh, it works pretty well. Connect cloud is where you can, um, publish, um, your data science, uh, products, applications, um, to a hosted platform that we run. So it's like the cloud version of connect and there's free accounts. Um, there's paid accounts, there's org accounts you can, you can share with other people inside your organization. And then posit connect is the older, um, and, and more, more feature complete product that is where you can install it on-prem in your, in your own, um, company's private deployment. And so connect cloud is the one that we host. Shinyapps IO I think is older than all of them. And that's where you can host shiny apps. Um, so it was the very, very first one connect cloud will be the place that sort of like tries to pull together all of those different publishing capabilities together into one place. It's like a, it'll be like a super set of all of them. That's, that's where it is today and why it's not consistent yet. And we're in, we're working towards making it more cohesive.
I really like this answer because it's something that, um, brings up a point that a lot of people don't realize. Um, and maybe I didn't realize either when I first was learning about data science and statistics was that our studio slash posit is not a super ancient old company. It is a fairly young company that still is doing growing and moving and shifting. And all of the things that you're supposed to be able to do as you're going and moving and shifting. Um, so if you make the assumption that our studio or posit is extra old and big, you think, Oh, well, they should have all those things consolidated by now, but no, we might just be in the process of doing that. Does that sound right? Am I, am I correct? Or am I off base? Totally true. There's so much we want to do. Like we have pages of notes here about like all of these good ideas that we want to build. And we, we agonize over like, which one do we need to do first? Right.
Yeah. And, um, I, you know, there are probably some other incorrect assumptions that people have about all kinds of businesses. I'd love to hear your assumptions in the chat. Like, what are your assumptions about posit or about our studio? Maybe we can debunk some of them. We have, we have five minutes left. Um, but let's, let's move on to another slide. Oh, question. And it is from you. Who is asked this person is, are some of us asked from an engineer's perspective, what makes a good minimally viable product and MVP and how can a team avoid the trap of over engineering?
MVP and avoiding over-engineering
Ooh, I've been over engineer big time. Yeah. One question that's helpful. I think is like, what is the cost? What are the consequences? Um, when it fails and that applies to, to data, to, to, um, software projects, to security projects, um, to a construction project, like what is, what is the cost when it fails? And you don't want to spend more on preventing a failure than you would lose from the failure itself. And that can really help like bring the conversation back into balance. If I'm, if I'm don't want to publish, um, an early version of a product because, uh, I'm afraid it'll make me look like I'm not a professional and it's like incomplete and, and, and not feature rich, or it has some known things that it can't do. The cost of that is really like what I think other people think of me. Um, but the benefit that I would get from publishing that early version is some people might really like using it. And some other people might point out things that I hadn't thought of. And it's incredibly expensive to get those outside ideas and feedback. And so it's worth a little bit of embarrassment to get some like novel insights on how you could make the thing better.
And so it's worth a little bit of embarrassment to get some like novel insights on how you could make the thing better.
And then there's other cases where like, yeah, you're not over-engineering. You really do need to make the thing safe. You're talking about like, Oh, one, one way we did it with connect cloud was like, publish the thing as early as possible. Wait until we were a lot more confident in it before offering paid plans. And that, that felt like the right sort of responsibility and, and proactive, um, you know, risk balance. Yeah. Work for free is a controversial thing, but I think that that translates into the freelance world a little bit where, you know, you, you build a few websites for people for free. Cause you're like, what if they stink? What if they're awful? And then you get some good feedback and you say, maybe I could do this for money.
Career advice
Um, all right. Well, we only have about a minute left, but I would ask you if you have a quick nugget of career advice that you could share in 60 seconds. Um, no pressure. Yeah. I, I think to, to take on risk is like the thing that to try to do something new, to try to do something hard and, and to be willing to embarrass yourself, like so many people have so much to offer and, and oftentimes you're the one, only one who's stopping you from like trying something new. And so, so take that risk, take the risk, do it scared.
oftentimes you're the one, only one who's stopping you from like trying something new. And so, so take that risk, take the risk, do it scared.
All right. Thank you, Elliot. This was so much fun. Um, if you saw amazing things happening in the chat that you don't want to lose, please go save the chat. There are three dots in the top right-hand corner of your chat, probably depending on your system, and you can go save the chat there. Um, this will be on YouTube in a couple of weeks. So you can look out for that. The rest are on our YouTube playlist. Elliot, thanks so much for hanging out with us. Thank you. This was great. Good. I hope everybody had fun. I cannot wait to see you next week where we have a community hero of mine, Tan Ho. He's an engineering manager at Teamworks Intelligence Soccer, but he is also just a community powerhouse. If you are part of the DSLC, uh, formerly known as our, for a DS, um, learning community, then, you know, Tan, if you are part of the Twitterverse, the Blue Skyverse, whatever, you probably know Tan. So come hang out with us with Tan next week and ask him all of your burning questions. We cannot wait to see you. Thank you. Have a fantastic rest of your week and weekend.