image: thumbnail.jpg
Inspecting websites to find JSON data APIs | Marcos Huerta | Data Science Lab
The Lab is an open, messy space for learning and asking questions. Think of it like pair coding with a friend or two. Learn something new, and share what you know to help others grow.
On this call, Libby Heeren is joined by Marcos Huerta, a Data Science Manager at Carmax, as he walks us through the guts of websites looking for data we can play with. He shows us how to find hidden REST/JSON APIs by using the web inspector in Safari/Firefox and then how to get what's necessary to pull the same data programmatically in python or R.
Hosting crew from Posit: Libby Heeren, Isabella Velasquez, Daniel Chen
Marcos's urls:
Website: https://marcoshuerta.com
GitHub: https://github.com/astrowonk/
Resources from the hosts and from participants in the Discord chat:
š Postman: https://www.postman.com/
š Insomnia (open source alternative to Postman): https://insomnia.rest/
š Baseball Savant website Marcos is using: https://baseballsavant.mlb.com/gamefeed/?gamePk=777076
š Isabella Velasquez's blog on using {polite} R package to help scrape Wikipedia: https://ivelasq.rbind.io/blog/politely-scraping/
š Festivas Mac app Marcos used to add the lights to his desktop: https://festivitas.app/
š Ted Laderas blog post on parsing JSON in R: https://laderast.github.io/intro_apis_json_cascadia/#/how-does-r-translate-json
š New rvest read_html_live() function: https://rvest.tidyverse.org/reference/read_html_live.html
š yyjsonr R package: https://github.com/coolbutuseless/yyjsonr
š tuber R package: https://github.com/gojiplus/tuber
š WikipediaR R package: https://www.quantargo.com/help/r/latest/packages/WikipediaR/1.1/WikipediaR-package
š rookiepy python package: https://pypi.org/project/rookiepy/
āŗ Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
The Lab: https://pos.it/dslab
Hangout: https://pos.it/dsh
LinkedIn: https://www.linkedin.com/company/posit-software
Bluesky: https://bsky.app/profile/posit.co
Thanks for learning with us! š
Timestamps
00:00 Introduction
03:05 Web scraping vs. API calls
04:12 Server-side rendering vs. client-side JSON
06:12 Warning: Rate limits and business ethics (ahem)
08:39 Demo: Baseball Savant website
08:57 Using browser Developer Tools and the Network tab
12:15 "What is curl?"
13:30 Importing curl into Postman
16:03 Generating Python code from Postman
16:50 "Are there open source alternatives to Postman?"
17:50 Using the generated code in Python/Jupyter
22:28 R packages for JSON (jsonlite, yyjsonr)
25:09 Demo: Massachusetts Lottery website
28:17 Example: scripts Marcos automated with Cron jobs
30:17 Handling logins and cookies with RookiePie
32:19 Demo: CNN Election Data
34:26 Inspecting ESPN's website
36:58 "Can you scrape YouTube?"
38:19 Finding hidden JSON in CardsMania history
45:00 Benefits of API inspection over Beautiful Soup
46:59 New rvest function: read_html_live
50:40 Inspecting LinkedIn and finding GraphQL
53:58 Encouragement on handling API pagination
Transcript#
This transcript was generated automatically and may contain errors.
We are going to go ahead and get started. I'm so excited to have everybody hanging out with us today for the data science lab. My name is Libby. I'm a data community manager here at Posit, and I am joined by Isabella and Daniel. Isabella, would you like to say hello? Isabella is here from Posit.
Hi, everyone. I'm Isabella. I'm on the DevRel team at Posit. I'm really excited to have you here. Thank you so much for joining.
Hooray. Daniel, would you like to say hello?
Hello, everyone. I'm also on the DevRel team at Posit, and I guess I'll see everyone in the Discord chat.
Yeah, Dan's going to be our Discord gremlin today. And I'm joined by Marcos Huerta. Marcos, would you like to say hello to everybody?
Hello, everybody. I'm Marcos. Happy to join Libby and everyone today.
Yeah, we are getting together every Tuesday at the data science lab to be a much more open, transparent, screen-sharing, messy place to hang out with your friends and code. This will be like a, hey, my friend wants to show me how he does something, and we're going to code together.
What this means is this is a place for you to stop and ask questions, to say, I can't see your screen. Your text is too small. Wait, slow down. Do that again. Or can you explain that a little bit more so I understand it? This is not a presentation or a talk. This is us like hanging out and coding together. And I really, really hope that you are open to asking questions and learning things today.
And if you are someone who knows a lot about the topic that we're covering, I hope that you will share your knowledge and your resources and extra tidbits about what we're talking about in the chat, because that is what this whole thing is about. This is a space for everybody. It doesn't matter what your years of experience are. It doesn't matter what background you come from. You are in the right place, and the Discord server in the data science lab channel is the place to be.
So if you just join the Discord server, you can head to the general channel. There's a little message saying that we're going to be in the data science lab channel. You can click that channel heading name, and it will take you directly in there. And I see that we are joined in the Discord chat by some alumni of the data science lab. Isabelle is in there talking and saying that she's just here for the mechanical keyboards. We might talk about mechanical keyboards today. We're going to try not to.
Let's go ahead and get into it. This is dangerous. Marcus, I just put together one of my keyboards, and several of my keys don't work, and I'm like, oh, I'm going to have to take it back apart.
I bent some pins when I was doing my hot swap keyboard. I bent some pins, and I didn't realize how important it was to make sure the pins were aligned. So that was a real problem. Yep. There's some foam in the bottom of my keyboard that's messing up the pins. So I'm going to go rip my keyboard apart and take the foam out.
Introduction to private website APIs
Okay, everybody, let's get started. What we're going to talk about today is the guts of websites. So when we want to get data from websites, we are often looking at web scraping using, like, Beautiful Soup or rvest, maybe rvest plus Selenium. By the way, rvest has a new, like, read HTML live function, which means you don't have to use Selenium all the time, which is cool.
But there's also a more manual alternative to this is an API call that's not an official API, right? There are some websites that have official APIs. They're great. We can call on them and grab data. Most websites aren't going to have an API that's structured for the public to use, right? But that doesn't mean that that website's not receiving data through API calls. That's what Marcus and I are going to be digging through today.
So I say we go ahead and get started. Marcus, can you give us a rundown of what we are going to talk about today at a high level? Because I think it's mostly the guts of how API calls happen on the website and postman.
Yeah. So, yeah. So, like, opening statements, like, yeah, Libby said it very well. Like, I think, I don't know, 10, 20 years ago, most websites or a lot of websites rendered server side, right? They would do a bunch of stuff on the server and then the server and then your computer would retrieve a HTML table or whatever from the server. And that's when you could scrape that with Beautiful Soup, right? You can do a request.get or HTR and R. My R is very good. I haven't used R in ages.
But you can basically pull that in and then you've got this basically giant string of HTML and various tools like Beautiful Soup or other things can parse those tables and extract the data, right? I don't know when the trend started, but now a lot of times what happens is the server is not sending you HTML. The server is sending you data. And JavaScript running on your computer is taking that data and building the pieces of the website for you.
So the table you see on a website is not, if you went and, like, curled that from the command line or if you request.get or whatever, programmatically with some retrieval code that does not process JavaScript, you would just get a bunch of JavaScript tags and you wouldn't actually have a table, right? So there's two reasons to do these private APIs. One is because, like, it's cleaner and better than trying to do Beautiful Soup. And two is sometimes Beautiful Soup won't work, right? Because, like, unless you'd have to, like, maybe save the source from a web browser or something to kind of get that generated table.
So what we're going to do is we're going to look at a couple. I'm going to share my screen here in a second. But we'll look at a couple of websites. We will use the developer tools. And I'm going to probably mostly be showing Safari, but I'll also show off Firefox and Chrome briefly. You're going to be using those developer tools, looking for what is your browser actually going to get and seeing if any of that is JSON. And if it is JSON, then we will figure out. You can obviously save that data directly if you just want to play with it. But if you want to get it the next time, we'll show you some tricks to maybe getting some code that will let you kind of hit that over and over again.
Another big caveat is, like, this is all fun if you're just trying to do some analysis. Like, hey, I really want this data from this baseball game or from this government website or whatever. Do not build a business around, like, the tools. Because eventually if you're hitting this thing, like, if you hit it once every now and then, no one's ever going to notice. They're just going to think you're a web browser. If you're, like, banging some private API, just pounding away at it, like, with, like, a, you know, cron job every minute. Like, they're eventually going to figure that out and probably block your IP address. So, like, don't do that. Don't build out, like, some big, don't think, ah-ha, now I have a great business idea. I'm going to create this weird API and do this. Don't do that. Because they will probably eventually block you.
So, is that a good overview? Yes, I think that's great. I think the takeaway for me is, like, with great sneaky Snoopy power comes great responsibility. Because if you like being able to sneakily grab data off of an API and then you ping it incessantly for, like, weeks on end, you will ruin it for all of us. They'll lock it down.
I think the takeaway for me is, like, with great sneaky Snoopy power comes great responsibility. Because if you like being able to sneakily grab data off of an API and then you ping it incessantly for, like, weeks on end, you will ruin it for all of us.
A lot of these sites, well, not all of them, but a lot of them will have rate limits built into the API that are kind of designed for what a human would do clicking around the website. So, if you start using, you know, request.get or curl on it, you know, a gazillion times, like, you might start getting, like, you know, not 200 responses. You'll get a 400. You'll get a 500. I forget what the code is. But you'll get some HTTP code that's basically, like, you've hit the rate limit. You'll get something other than a success. Yeah. And that happened to me recently. I forget what I was trying to do. But I was starting to hit rate limits.
All right. Well, let's do it. Let's do some screen sharing. And for anybody who is hearing curl and wondering what that is, we'll talk a little bit about that. And I would love to hear in the chat what your experiences are with web scraping because I use this method that Marcus uses. Marcus taught me this. And he, like, fielded a ton of my, like, where do I find this questions, like, years ago. Because I hate web scraping. If I dive right into web scraping, I don't always know what I'm doing.
This glimpse in the background, if you do do web scraping or use, like, hitter to call APIs, this will help you understand more what you are doing when you're doing that. Yeah. See, another R package that I mispronounced. You said hitter. And I did not know the type pronounced H-U-T-T-R. I would just think H-U-T-T-R. Isn't there, like, a little person hitting a baseball on the sticker or something? That makes sense. That shows I haven't seen the hex, I guess.
Exploring Baseball Savant in Safari
All right. So here we are at Baseball Savant. I'm in Safari. I will feel free to follow along on your own web browser. And so, you know, it's got this nice, cool site. This is, like, a particular game, right? We've got pictures and exit velocities, all this cool data. So these are all kind of tables. This is, you know, this is all getting constructed with JavaScript. So the first thing we do is you want to inspect element. And if you were in a different web browser, you might want to do something differently. But you want to kind of go into the ā so a lot of this is, like, source code. But what we actually want in Safari, anyway, is the sources tab. So one thing you will usually have to do is refresh because you want to get fresh requests.
Yeah, you do command R for the refresh, too, I think. Oh, yeah, thank you. Now, sometimes you might ā in Safari, sometimes you might see what's called an XHR request. That's frequently a thing you want to look for. But sometimes they don't show up as XHR. They show up as just fetches. So we click on these. And, oh, look, what is this? It's JSON. So here we see trending players. We see some JSON data. That is making ā well, nothing on this page, apparently. But that is some data. We've got a schedule here with a bunch of stuff in it. We've got this GF, which probably stands for game feed. And that has a ton of data. Look at all this JSON, right? So that's pretty exciting. And then we have some other smaller data set. And, you know, these actually are telling you what they are, picture yearly averages, right? So this is the start of everything.
Now, the first thing you can do is just ā let's look at the game feed. And you can just ā in Safari, you can just right-click on this and save it. I think Safari is the only one where you can right-click and save a file, though Libby was telling me that she thought that was the way to do it. Yeah. When I'm on Windows and I'm using Chrome, I can right-click and save object. So I'm going to say this is gamefeed.json. Or copy it. I'm going to say gamefeed.datalab so I can find this file later.
And then let's open up VS Code and ignore all of my projects there. And let's just open up this one file. Can everybody see Marcus's screen with the text size and everything? I will make things bigger, yeah. Zoom in a little bit. Someone who's reading the chat, tell me. Everyone can see everything on my computer right now.
I love Marcus's Christmas lights. I know I'm distracting us from the point here, but look at his Christmas lights all around his computer. That is the app called Festivus, if anyone wants to buy that. Look at this, 100,000 lines of JSON. So it has an app called Festivus, which is a Mac app if you want to decorate for the holidays.
Parsing the game feed JSON
All right, so here's this giant ā and I formatted it, and let me make the font bigger. So here's this giant JSON file, right? And this is a whole other dark art of trying to figure out, like in these nested JSON files, like how do you want to get out what you want? Like, for example, it looks like we've got the scoreboard as a key, and then we've got a list ā sorry, a dictionary. But then within that dictionary, we've got a list of dictionaries. So I don't have time to go through it. We've got some win probabilities by inning. This is all pretty cool, but you can see that it's kind of buried in here. Like this key is in stats, scoreboard. I mean, there's all these kind of keys kind of tucked away.
So there's a whole dark art to taking these JSON and trying to massage it into something you can turn into a DataFrame. But that's probably a little bit beyond the scope. The point is there's a ton of data here, and we want to play with this data, right? So the first thing you could do is just save it and load it into Pandas or a DataFrame or Pollers or whatever.
Using cURL and Postman
I think what I want to do first is show, like, maybe you just want to get this data directly into your machine as opposed to kind of saving it as a file like I did. So, again, in Safari, one thing we can do is right-click, and this is definitely in all the other browsers. Copy is cURL, right? So what is cURL? cURL is this, right? cURL is, and I'll keep trying to make this bigger. cURL is just a command line program on Linux or Unix, and it just basically will download stuff, right? It will download the text or it will stream that. If you type this into, like, a terminal in the Mac, it would just spit out all of the HTML. I'm sorry, all the JSON because we're hitting the actual API here. So this cURL is the API, and these are a bunch of cookies and a bunch of various headers that you may or may not need. So you could just paste this into your terminal and get some stuff out.
In fact, I think I did that last time Libby and I were doing this. So let's just pop this in here. Well, let's copy this again and drop that in there. Don't worry about this. It's pretty mad. Why is it mad at me? Oh, it's binary output. That's interesting. Okay, well, ignore that. Anyway, so this is what it looks like. So here's the link, some refer, some cookies.
So the real trick now, we don't want to be using cURL command line stuff. We want Python or R, right? So this brings us to an app called Postman. Postman will really try to make you have an account. I always ignore this and use it without an account. So I was doing this earlier when Libby and I demoed this. So I'm going to go to import. Yeah, Postman really gets me. I thought for a long time I had to have an account, so I avoided it. So I pasted in that cURL. I can just slow down and do it again. See, it says paste cURL to import, and then you paste it in. And now it has translated all of that into ā can I make this font bigger?
Yeah. Everything is based on ā what's that thing called that VS Code is based on? I don't know. I can't remember. Someone remembers. Anyway, the command plus works in this app, too, because it's also based on whatever that web framework is that makes all these apps. Anyway, so now what has it done? The params are just this ā the params just means this question mark thing, which is some kind of identifier for the game. And then the headers are all those things we saw. And if we click send, we should get back a bunch of JSON. And here's the JSON that we saw, right? Yes. And so those are like each of those little check marks. That was in our cURL, each of those little dash lines that was like dash H something. Yes, exactly. And I'm trying to make this go away. Can I make this go away? I think that's as small as it gets. I'm trying to make it go away. But that's okay. Oh, there we go. I got rid of it. Yeah, there we go. At least we can see your text. And also I heard electron is what they're saying. Electron. Thank you. Yes. Well, electron ā they both have electron apps, so command plus just worked in both of them because they're both basically the same thing under the hood.
All right. So now we've got all this JSON data here. One thing you can do, one thing I like to do is I start ā I just start unchecking things like does it really need this cookie or is it going to get mad at me? Will the request work without the cookie? No, the request still works without the cookie.
Do I really need this like user string with all this Apple stuff, you know, pretending we're Safari? Yeah, it still works. Sometimes you turn ā do we really need the refer? Probably need this. Let's see. Do we need it? No, we don't. So you can kind of start turning things off and seeing if the request still works or if it doesn't because sometimes you really need that cookie. It sets a session cookie and it won't work without it. It really needs the refer or whatever.
Some of these things you don't want to turn off because it's telling it like what kind of data it accepts, so I usually leave the smaller ones alone. Sometimes, you know, the cookie can be really important. For example, I have a little system by which I get some data off of the Kia website for my car, my electric car. The data is all there. The website only shows you like four columns, but if you do what I'm doing here, you can see like the real data structure. But I can't access that without a cookie, so I have to get the cookies out if I want to download it.
Generating code from Postman
Anyway, this little button here, which is very hard to see, is the code button, and this is where it will turn this request that you imported into code. So Python request is what I'm going to copy because that's what I'm going to go show off in Jupyter here in a second, but you've also got everything. You've got PHP. You can go with non-request Python if you want. There's base Python. There's hitter. There's rcurl, right, and it will just generate the code for you. It's not that complicated because it's really just a set of headers and a link. It's basically headers and a URL, but nevertheless, if you need code for different things, you can.
You can go back to curl, which is kind of funny because that's what we imported. It's all like cleaned up now.
Oh, we have a question from Nick as well in the chat, which is like, are there open source alternatives for Postman? Do we know? That's a great question. The other app that I've played with before is RapidAPI, which I don't know if it's open source or not. Is it another Electron app? I think it is. I think it's yet another Electron app. Oh, maybe it's not Electron. Command plus is not working. So RapidAPI is a similar app. I think there might be some Mac ones that are at least not Postman. I feel like there's one more that I've played with once, but I couldn't find it again. So RapidAPI is another one that you can use, which I believe has a Windows version as well.
Adam is saying Insomnia? There you go. Insomnia? I don't know what that is. There are plenty of them. Thanks, Adam. To me, the trick is can it generate the code for me? Because it's kind of just very convenient to have that ability to generate the code.
All right. So let's see if I can get a Jupyter Notebook. But you could just like look at the curl and construct it yourself. You could look at the curl and construct it yourself. Hitter or, yeah, which is often where I hit a barrier because I'm like, I'm clearly doing this wrong. Or I'll get a response I don't understand. Firing up a Jupyter server here. All right. Cool.
And once again, everyone can see all of my directory structure. That's fun. Let me just open up a single. That's okay. So the, oh, great. Dan just put a insomnia.rest link in the chat for us.
Finding JSON in Chrome vs Safari
And this is like the way that I prefer to do things also because I like to be precise about where I'm finding things. And I sometimes, if I'm in Chrome or something else, it's not in the same spot. And I figure this is a good point to like stop and say that is in Safari, we opened up, we did the right click inspect, and it's underneath this like nice folder structure for us. But often when I am in Chrome, I have to go to the network tab to the XHR where it's fetching or like fetch slash XHR. And sometimes this payload is in different places and you really have to click around.
Yeah. So let's actually, we got time. Let's look at, let's, let's look at quickly what it looks like in Chrome since a lot of people use Chrome. So if you would do this in Chrome, inspect, refresh, I never hit command R. Libby keeps telling me to just use a keyboard shortcut. I'm a keyboard shortcut person. I know. So here we go. It's sources. Didn't you say sources and network? Network. It's network. And then fetch. So there it is. So here's the same thing, right? And you just have to kind of be comfortable poking around. Once you, once you do it once with whatever browser you prefer, you'll kind of get used to looking and looking for it. So here's the same data once again. And you can right click and copy as curl.
I don't see it. The harder thing is like, it's not, yeah, there's no save it's copy. So I ended up just, I paste it into a text doc and go from there or I'll save it as a doc. Jason. Yeah. So, so, so that you get the idea. All right.
Parsing JSON in Python
So now we're coming over here and there's so many, I cleaned up my windows and now it's already a window. You can close the Chrome. Yeah, that's a good idea. Yeah. And then we'll go ahead and close Chrome. Everybody hates me for it. All right. So here we are, we've done the same thing, but now we have this response that Jason, which is going to be a dictionary. So let's just do a response. It's interesting that we finally get relimited. We might have the, the code defaults to something that's like requests something or not response. Do we need to change it to response?
Maybe I did need some of those cookies. Maybe I was wrong. Let's go back to postman. Let's turn my cookies back on. And I refer back on let's not get swift code, but let's get Python requests. Let's copy this. Let's come back over here. Put that in there. Let's see. So let's go back to postman. Let's come back over here. Put that in there. Let's see. So is this going to work? It's really not happy with me. That's what I know. Everyone's seeing in real time, the things that are going to happen when you're trying to do this on your own.
And it seems to be working. There we go. I think I was using an old version of Python. I mean, that's the problem. All right. So now here we go. So now here we have a bunch of JSON. Basically we'll parse the JSON for you if you're using requests in Python. And so this is now a giant dictionary, so I'll just call this. But the thing that it prints for you at the end of your code is not response.JSON. You have to change that. Yes, that's true. That's true. It'll do response.text, which will just give you the raw string. And by response.JSON, if you're using requests in Python, we'll basically turn that giant, basically essentially import the JSON library and turn it into a giant dictionary for you. So now we have a giant dictionary called game feed.
And now we can look at the keys and we see all these things. And maybe we can look at home team data and see what's in there. And then you can start playing around with it and trying to figure out what you want. To me, what's interesting is the probabilities. I've looked at this data before. So let's see. It's basically a JSON parsing. You have to decide what you want to do, if you care about. I think we liked scoreboard, right? Scoreboard had some fun stuff in it. Yeah.
Oh, and David is asking if anybody knows of a better R package for JSON like to TSV or JSON to CSV other than JSON Lite. JSON Lite's what I use. Does anybody use anything different? Let me know in the Discord chat. And also, I really like the, like, if you're parsing JSON in RStudio, you open up your JSON objects and it gives you this nice little, like, nested structure with little dropdowns that you can open. It's, like, nice and visual. Highly recommend. That's what I use.
So at some point you're going to have to, depending on what data you get, you know, it might be a very simple structure, which is just a nice list of something that you want. It might be more complicated where, for example, what I really find interesting in this data is the stuff that's making these probability graphs, right? I think that's kind of fun, right, because you have this probability of who's going to win going up and down. And so that's kind of what I would be interested in grabbing, right? So I would have to go in, find where that is, because there's only one game, right? This is a single, this is the, but we only, I believe we're getting the information just in a single game from a single day. So we'd go in and try and find that, find that data and turn it into something and turn it into a data frame or what have you. But it might be in stats, not states, stats. So there they are. We want WPA. We want game WPA.
So now we have a list of dictionaries and that's what you need to do something like turn it into a data frame, right? So now we have a data frame, right? Home team, away team, win probability. Basically it's a real-time estimate from the MLB computers as to how likely one team is to win the game or not based on the score and the ending, et cetera, and the situation usually. So that is essentially the technique, right? And we can now apply this technique to a variety of, of websites and, and see what, see what data is out there, right?
And we can now apply this technique to a variety of, of websites and, and see what, see what data is out there, right?
It's, it's, to be honest with you, the, the getting to this, getting the data, getting the JSON is probably, once you've figured it out, it's a pretty easy step. Then there's like traversing the JSON structure, which is like dictionaries and lists of dictionaries and lists of dictionaries. And sometimes, like, for example, there is an actual official API from Virginia legislature that like returns, like, it's hard to describe it. Essentially there's a list buried down in there that I want, but I also want identifying keys from above that, that, that tell me like what build this is or whatever, or what the votes are. And so I have to end up doing this elaborate dance of like extracting this list and then pulling in keys from the, from the, from the one or two levels above. I would love to build some kind of universal JSON smart parser thing that will like kind of do that for you. I haven't, it's so hard. I haven't figured out how to do it.
Exploring more websites: Massachusetts lottery
Let's look at another one. Here's the Massachusetts lottery. Powerball's big. If anyone is excited about Powerball. So let's. Oh yeah. We have Ryan in the chat said that there is another package that I'm going to say is YYJSONR. Like letter Y, letter Y, JSONR. I've never used it, but it exists and we should go investigate it. That's cool.
So we're still getting fetches. We're not getting in Safari anyway, but we're getting interesting things like hot and cold numbers and cold numbers. And you can see a lot of stuff from a lot of stuff that you're seeing on the screen here. It's being, it's actually a bunch of JSON. We have upcoming draw dates, draw schedules, the numbers of the latest draw, things like that. Maybe we're interested in like instant, instant games. We've seen this thing pop up. Is that showing up over here? And yeah, sort of. Here we've got a games dictionary and it tells you all that it's showing. Basically I think it's everything that's here is now kind of rendering here. Some of you were curious about the, these images, whatever. Have I ever potentially done this for a lottery website before? Maybe I have.
So this is like showing like kind of all the different prizes and different things. So you could play with that too. So in the same deal, right, you could copy as curl, go over to Postman or RapidXR, your favorite app, generate the code, switch to a code base, play around with it, automate it. So that is kind of the recipe.
Does anyone, what do you think we should do? Should we try some more websites? Should we take more questions? Yeah, I would love to take questions. Just remind everybody that you can ask questions and stop us when we're going very, very quickly through things and potentially go through just like another round of this and be messy and have it, you know, cause problems.
Real-world automation: conference schedules and lottery data
This is how I, if you've ever seen, in the last couple of years, I created like an ICS file that was of the PositConf schedule. Schedules are usually JSON objects somewhere and you can go scrape them and do stuff with them. I'm always like, oh, I wish that there was like a Google calendar file that I could use that I could just upload and have all of the events separated and all of the talks separated and all of their like abstracts and stuff in there so that I can choose between them. And so this method is how I did that for the past couple of years. I do have a video probably on YouTube somewhere of how I did that the first year. And then the important lesson that I learned was I went back the next year to do it and the JSON structure was completely different. So I could not completely reuse all of my, my code.
Yeah. Even though it was like the same, the same website. So there are, there are always going to be challenges. And this is definitely not something that you're going to set up and like, like Marcus said, like build a thriving business on something that will just be plug and play forever. So no, I was going to say, it's just like be okay with the manual part of this. And you'll be all right.
The only thing I actually automate with this is it is, it is my, I'll go ahead and show off. It's this, it's this site. So for the Virginia lottery a few years ago, I built out and this is like pre, before even about shiny and dash and all these things, I built out this kind of manual, this manual site that's like a bunch of ginger templates that I wrote myself. And it, it, it actually mostly does beautiful soup stuff and scrapes various scratch off web pages for the lottery. But like Virginia lottery, the Virginia scratch off things, but like who's image here. And like this just, I needed to know what were all the games. Cause like the, the Euro was predictable based on this number that I need to scrape, but I didn't know what the list of active games was. And so sure enough, somewhere on the Virginia lottery website is a tiny little Jason list. When you, when you go to look at all the lists of scratchers, much like I was showing the Massachusetts site is a list. And so I hit that list once a day, right. To make sure I have like a new games. I know if there's a new game or not. And so I know what to scrape. Right. And all that runs once a day. Right. And so far. Still works.
But what I'm doing is I'm looking at the ratio of unclaimed to claimed tickets and trying to see which, which games are the best deal. None of them are a good deal, but like, which one is the good deal every now that you do get positive expectation value. But right now it looks like we don't have any games with a positive expectation value.
Did I miss the way that you schedule that to run every day? So I have a server, a digital ocean that hosts all of my, my website, all my dash apps, formerly a shiny server. And, and it also has, so it has Cron. And so, and so I sometimes, sometimes I run these on my own Mac cause my Mac mini never turns off. So sometimes if it's more computationally intensive, I'll start, I'll just spin up a Cron job on my Mac at home.
Like for example, I use some language model stuff, like the small ones, like sentence transformers that I talked about it and positive. And those that's just too much to run on my little digital ocean server. So I'll run those on my Mac and push them up to wherever, wherever that data needs to go.
Using cookies for authenticated sites
But I was going to say is there's an advanced advanced. If you really want to get advanced about this, for example, I mentioned that I have this, the Kia website has, has stuff that I, that I want for myself. It's my own data, right? And so that is obviously not accessible unless you're logged in. Well, the way it knows if you're logged in is a session cookie.
And if you have something like rookie pie, which is an event, this is a, this again, I don't know if this exists for R or anything like it, but rookie pie, it basically, if it only works for Firefox, it says it works for everything. If Firefox, you're logging in Firefox, rookie pie can see all the cookies for a site that's from Firefox. So I can log into the website and Firefox, and then I can run my script. Rookie pie goes and grabs the cookies I need and then does all this request stuff that I was showing you and gets the data I want and the way I want it.
So if you want to get, if you're looking for advanced like tactics for doing stuff for like sites that you log into and you don't want to manually be cutting, cutting and pasting cookies every time you use it, rookie pie, potentially it might help you. It claims it works for all the browsers. I've only actually got it to work with Firefox. I started to wonder if it should work for Firefox. Is this, that means that your cookies are sitting in a text file somewhere on your computer. But nevertheless, it works. I use it. So rookie pie, and maybe there are other things like if you search for cookies, you'll find a bunch of packages. And maybe, maybe there's similar ones for, for R, and another one for Python or for R.
Exploring websites live with the community
I had a question. I put it in, in the chat, like what are some websites that we could attempt to snoop to show how hard they are. You're asking the community. That's great. And one of them that's really, really hard. The first one that got mentioned was LinkedIn.
So things that need a login are hard, but LinkedIn is hard for multiple reasons. It's just really, really locked down. But also like Marcus would have to log in and show his like personal stuff in LinkedIn. I'm not sure. A lot of websites are tough, but they, they do usually have similar things, but yet then the copies become really important.
Another one that I've used before is the, the CNN election data. So here is now you actually see it under XHR. And so, here's some results from Virginia from, from we had elections in November and you can see that all these different hits, the different rate, every race is a different link. Basically, a GBA, Lieutenant Attorney General, Lieutenant Governor. I'm sorry. I guess GG is governor. And so this is similar thing and you can get, and you can get it. I think this is like full data by County. So you could probably download this from the County, but this is a nice JSON five version. Same deal. You can copy as curl. You can bring over to postman or, or insomnia or your favorite, your favorite thing. Import it, get it. This one actually has, looks even has a little secret API key here. And then you can do the same thing.
It's good to point out that when you paste in to postman, you are switching to the headers tab. There's like six little tabs down there. No more than six, seven.
Did we hit one? That's not going to let us. Maybe three or four are not modified. Interesting. It might. I think it's, it's not sending the data. It's just telling us, you don't need to get this data again. So I have to, I'm gonna have to do something to say, no, really give me the data. To like, tell it that you want to refresh.
We had a lot of suggestions like Reddit. Reddit is also, they've made it super hard. Lauren says ESPN might be a good one and SeatGeek. What is SeatGeek Ryan?
Brian also made the point. A lot of websites that don't want you to scrape their stuff are going to change their end point URLs pretty frequently so that you cannot programmatically go through. Cause it, it's almost like a revolving. What is it called? When you VPN, they're like, we're just going to make our IP not static and you can't hit us.
Yeah. All right. So we're on ESPN. We're on ESPN. We are looking for things and not areas and fetches. I'm not seeing a lot of times. I would say some of the stuff you'll see in here is completely, it's not useful at all. Right? Like you're going to see JavaScript and images. And some of them will be, there'll be data, but you were like, well, I don't really care about this. Right. So you just have to look through it. It's hit or miss. Like sometimes it'll work. Sometimes it won't. Sometimes you'll be like, yes, this is exactly what I wanted. This is not, this is just like, this is what, this is information about the ads they put in the ad carousel. Like, I don't really care. Let's look at Aaron Judge.
So much of what I wade through when I'm doing this is ads and like underlying information for ads. Yes. That's all tucked away in there.
How do you know that when you're finding JSON, because like when I'm looking visually, I can see the JavaScript it's like JS and in other browsers, it looks different. But how do you know from just like the file structure that you're clicking? Well, this one says JSON. So I kind of, that was JSON though. That's not working, but I mean, it usually JSON looks like this. It's like, it's like kind of like a, like a Python dictionary or whatever. So I just kind of click through things and like this is HTML, right? This is JSON. This is, I don't know what this is. It's some kind of list of, of, of audio codecs. So that's not anything that's an MP4 file. So, you just have to kind of keep looking. And also you can look, also you can see that on here is the domain, right? So like if the domain is whatever this is and it's an MP4, you probably, that's not going to be JSON, right? This is some kind of, what do you call it? Content delivery network, audio file or whatever.
So it looks like, it looks like ESPN has a ton of XHR requests and there's almost so many is that it's impossible to tell what the heck any of these things are. It doesn't look like they have a lot of data. I have found that the bigger, the website, the more stuff is crammed in there. And so sometimes a smaller website, that's a simpler website. You will have more luck with, but the logging in thing is, is not always a barrier. So like the key of stuff, I'm sure that you have to log into that, right? So that's a, a instance where you're like, you can totally get in there. I recommend just trying this. Like I'm interested in this data. It could be data about you. You could need to log in and try it.
Isabella asked if we can scrape YouTube. YouTube YouTube rate limits you so fast though. You gotta be careful. I mean, all kinds of people are trying to write all kinds of things to like help you like, either download YouTube videos or, or, or playing without ads. And so I don't really know if there's a lot of data to scrape there.
I would rather just Wikipedia. Wikipedia has an underline. It's a familiar text, right? I do the tables and Wikipedia constructed service. Constructed. I feel like the tables are pretty easy or sometimes easy to scrape on beautiful soup, but then sometimes I absolutely fail. Dan says, Wikipedia ends up being pretty easy with CSS selectors. I think it depends on how the tables are constructed.
I mean, it looks like a server side. I don't think this, I don't think these are HR requests. Cause it's all about where do they make the table? So they can start the tables with JavaScript on your computer or are they to construct them on the server? Cause it can start on the server. You just curl, you just download the page directly and then you use beautiful soup or whatever your favorite HTML parsing library is. It's when they don't as well. It's when they don't do that, that, that they're going to have to send the data as JSON. And then you have a chance to get it.
Cards mania and the origin of JSON inspecting
I mean, a fun one that I like is, and this is something I do with my friends a lot is we play a game. We play cards on cards mania and see if it's going to work or not. So I realized a long time ago, this is a game I played with my friends a while back. That there is a hidden JSON structure and a game history. So let's see sources XHR. Why is that working? It should be here. I need to click on this. There we go. So here's, so this ridiculous JSON structure is basically the entire history of this game of Oh, hell that we play. And so I wrote a bunch of code to partial lists, cause this is like telling you what order you play the cards in. This is the queen of hearts. This is the king of spades or whatever. And, and then I built a whole website around it for my colleagues and I to, to analyze how, how well or not so well we did when we were playing against each other in cards.
So that is that kind of, that might've been the start of my whole journey of JSON inspecting, trying to find hidden data structures. That one, we just basically do it in the browser and then we have a interface to upload the file. And then I, then I go through and personal.
that might've been the start of my whole journey of JSON inspecting, trying to find hidden data structures.
Chat updates: YouTube, all recipes, and Wikipedia
I will say just for anybody who is watching and is not in the discord chatting, I thought I would give some updates on what's happening there. There are lots of questions about like YouTube. Could we scrape YouTube? And there are a lot of ways to do this. YouTube has its own API. You can just sign up for an API key and go look through stuff. But also like there is an API. I mean, there's a, a transcript package for Python, but it's going to like figure out that you're not a browser pretty quickly and it's going to block your API. And you have to like use a proxy to get around that there's instructions for it, but it can be really complicated. I really suggest like just going through the legit API.
And then someone had asked about all recipes.com, which I think is a great suggestion. Cause I wonder how they set things up. All recipes might be one that's like Wikipedia, where it's like built on this side. I would guess it's just a bunch of static texts that they generate in like a CMS or something as opposed to like a bunch of data-driven things. Sometimes it can depend on how the website is organized to on, on how, how pre-packaged their data is for you in JSON format. Like whether or not you get one recipe or you can access like all the recipes in nice nested JSON, you'd be surprised.
Sometimes it's, it looks like it looks like there's a little quick, the one just tells the, just like creates like images, like see this is, this is JSON, but it's nothing useful, right? It's like, it's like links to SVG files. And then this is just JavaScript. I mean, the more text there is, the less likely they are to, I mean, the reason why they like to do server side rendering of tables, especially pretty tables is because I think it's, I'm guessing it's more performant to just, here's a big JSON structure of numbers. You on your site, you, you know, CPU and memory or whatever, take those numbers and turn it into this pretty table, right? Right. Locally on your computer. So I think that is the impetus for this kind of, you know, back-end you hear a lot about back-end and front-end developers. Well, the front-end developers are writing the JavaScript that makes a pretty site and the back-end people are making the servers that, that, you know, send you all of this, all of this JSON data.
We mentioned Wikipedia. Jason says that there is a Wikipedia R package, which I have never used before. But that's like the good thing about having a website that actually has a dedicated API for it too, is like sometimes people have built wrappers for you so you can just access stuff.
And then someone had also said, Isabella said that there is a Goji plus package by Goji plus called Tuber, which has like a get captions thing for, for YouTube. So you could go explore that. I will also say that the Ragnar package is, I think maybe going to get an update that will help you more easily grab YouTube captions. Don't quote me on that. If it doesn't happen, I'm sorry, but yeah, watch the like Ragnar package space because I myself really, really want to build rags on YouTube, like transcripts. That would be amazing.
Looking at SeatGeek
What are we looking at now? Is this SeatGeek? This is Ryan's suggestion. I was looking at the chat, trying to see, okay. Trying to see any refresh. I don't even know what SeatGeek is. Is this for tickets to concerts? Yes. It's like a secondary ticket market sort of deal. You know, like you can, you can sell your tickets on there. There's some interesting stuff here. We got a, whatever this, this looks like some event. So this is not like ticket master. This is like, this looks like eBay for tickets. And then there's an affordable events end point, right? And it looks like it has more things like some games, like some basketball games. And then we've got, there's quite a few things here. It's worth playing with. If you're interested in this sort of thing. I think this one might be interesting. I think, I feel like there's all kinds of frustration around the ticket prices lately and ticket availability.
Someone in the chat was saying they had scraped something to help them figure something. What was that? Someone in one of the early posts. Let me go back to the beginning of the, of the day. Of our wild chat, right? And it's a ticket master alternative, or is it like a reseller? Can you resell a ticket master?
And Ryan said that he just did a Ragnar run. So hearing Ragnar is giving him some PTSD. My, our family has a ranch where we actually host Ragnar runs. It was Kevin. He said, he scraped to use car websites. Hey, I work at CarMax. I don't know if I approve of this. So you could get a decent deal on his truck.
I mean, I mean, I mean, this is something that businesses do deal with. We do. You do see if you are a business, you don't really want people scraping all the prices off your site. But if you are a business you want, you would like to scrape all the prices on your competitor's side. So it is a tension there of, you know, you would like to know what your, what your competitors are doing, but you don't really want people to know what you're doing. But of course, I mean, it's kind of hard to stop, right? So that's kind of what I care about, about API users versus website scrapers.
if you are a business you want, you would like to scrape all the prices on your competitor's side. So it is a tension there of, you know, you would like to know what your, what your competitors are doing, but you don't really want people to know what you're doing.
Let's see if I can find it. What are the benefits of using this over going straight to something like
Beautiful Soup? So you're like, okay, I have this website that I want to scrape and you don't know anything about this inspect side. You're just like, I just know that I want the data off of this website and I see some tables and graphs and I think they're being made by data. What's the advantage of doing this versus hopping straight into something like Beautiful Soup?
Well, one is that the data is, if you do this way, the data is structured, right? You don't have to, you're not trying to figure out, you know, find every TD cell, you know, I know people use Beautiful Soup, but like you're, I mean, Pandas has a built in like find table on a website thing. So that sometimes can work. But fundamentally the big, the main thing is, is that if you try and just download, you know, a website, you just get a link to a website you can read and you try and just download that directly with requests or hit her or whatever. There won't be anything there because, because you don't have JavaScript running to ingest the data and assemble it.
So you have to do something like Selenium or whatever, which is essentially running a web browser for you. The web browser is telling the thing it's the JavaScript is turning it into, into Beautiful Soup and then you were turning it back into data. So, so if, so why go from data to HTML back to data when you can just go straight from data to data? That's kind of my philosophy.
so why go from data to HTML back to data when you can just go straight from data to data? That's kind of my philosophy.
Again, there's some sites where there is no data stream where the server is sending you the table, in which case you do have to kind of scrape it away. So I think, I think baseball references like that, for example, I was trying to get some, I was trying to do, I was trying to figure out people were saying that the, one of the World Series games was like one of the best games ever. And I wanted to, you can take that win probability kind of zigzag and like integrate it and figure out like how much, basically how much, how crazy the game was, but how much swings there are. And so I was trying to do that. And I think I had to do old school Beautiful Soup parsing because there wasn't a, I don't think there was like a API call that they were doing. If I remember that correctly.
So getting data off the, getting data off the web is kind of like knowing more than one language. It's really useful to know how to do it in more than one way, because you are, there's not one thing that's going to work across everything.
The rvest read_html_live function
I did want to point out one more time, Ryan had put it in the chat. I will put it back in the chat again, that if you are using R, the rvest package did not have this initially, but it is new and exciting. And it's the read HTML live function, which does what Marcus just described Selenium does. When you run this, it opens a headless browser on the underside for you, pretends that it's Chrome using Chrome mode, I think. And then that way it can like actively look at something that before you wouldn't be able to, right. You would be trying to get something that's static, but you were looking at something that's live. You're getting an error. You have to use Selenium to do this. You can now do it in just rvest, which is really exciting. So definitely go check that out.
Samer Hijazi has a YouTube video about this. He put out a little update. That's how I learned about it. So definitely go check his YouTube channel and I'll see if I can find his channel and link it.
Looking through the, I'm looking through the chat. Interesting.
I'm going to go find that. Sorry, Isabel. Isabel saying to make the, to make the Postman window larger? Too late. I think we did. I think we did. And we checked back in and we were like, Hey, that's, it's working now. Yeah. I don't know why this one isn't working. I'm a little upset that, oh, there we go. There's some data here. I guess they, I guess we hit the refresh limit. So now you can see the structure of data from the, from this governor, governor from, I think it's going to be every race. Anyway, the recent elections here. I I've used this one before too. It actually even says politics, that API to CNN. Oh, it's so clear.
Yeah. You have, so you said that you have a website where you do stuff for Virginia races? Oh, yeah. So they actually have a real API, so I don't have to, I don't. Oh, okay. I used to have to scrape things. It's for, it's for legislation. It's this, it's called recorded vote. But I don't have to, I don't have to pack anything. I used to have to scrape one, a couple of pages, but the new API, I don't have to anymore. So all this is coming from a real API. I mean, I hit it once a day or whatever. This is all coming from a real idea.
It's also hard to see. Ryan had asked if the, I guess their read HTML live works on shiny app. Dot IO slash positive cloud because our selenium didn't work. I don't know. Yeah. I mean, anytime, anytime you're having a good run selenium, I feel like I feel like you've lost. I mean, I don't know. I mean, I mean, I mean, sometimes it's necessary if you, if you really need like a particular site, but like if you're running a headless browser and you're turning and you need, like you, you basically need the JavaScript to run. Then I feel like there's probably some hidden struct data structure there that you'd rather tap into instead. And that's not necessarily always true, but it feels like I, I try and avoid selenium, right? If I can, I either want to just use request that get to beautiful soup, or I want to go look for the, the, the hidden API.
I think the last time I tried to get selenium to run, I wouldn't, I was complaining about something. And so I didn't even, I didn't get very far. I definitely, maybe I'm just a bad luck. Dan says, plus one, if you're using selenium, prepare for sad. That's why this is so like learning how to dig through these tabs and stuff. Um, it's so, so valuable because you can avoid that roadblock of sad. Sometimes I got four different people mentioned said electron when I was trying to remember what electron was called. Everybody figured it out.
Exploring LinkedIn's network data live
Well, I wanted to see if I could like, log into LinkedIn and, and not show you guys all of my stuff. I think that I can, as long as I don't click anything. Um, and then I can show you guys what it actually looks like. If I do attempt, okay. Because I have no shame, and I don't really mind. People seeing my stuff. All right, let me share my screen really quickly. I know that were at the end and just. Attempt carcass can be my guide.
I'm on Chrome. Nobody hate me. Um, I'm going to inspect. And then I'm going to go to the network tab here. And then I'm going to go to the network tab here. The 1, the thing that might be interesting is like, click on like your network or something like, click on my network to see all the people you're connected to. If you, if you're willing to do that, because that, uh, that might have some interesting. Oh, wow. Look at all the things notice as I'm like going down and it's loading because you have to load more, you're going to get more. So we are not going to get the full lake. Yeah, but it might be a paginated.
So here's another trick, which I didn't talk to you about. I should have, I should have thought about this more. So sometimes you will do something and you'll find an API endpoint and you'll cut, copy it and it will only show you 50 results. And you're like, that's weird. But if you start fiddling with the request and the payload, you might be able to like work around that limit, right? it might be, it might have a setting like, you know, uh, you know, a max. Yeah. I might have a max or a limit that you can change.
Um, and if we go back to like the regular page here. And I'm going to do my command R to refresh. Hey, well click. Why don't you put one of those graph QL ones so people can see the hellscape that is graphic. Sorry. The, oh no, it's all like, well, that's, that's the, that's just, that's just, that's just Jason. If you click on preview, I think that'll look okay. Well, not that. Okay. But copy the actual link, like copy the, uh, what do you call it? You're going to have to guide me. Right. Click on one of the graph QL, the query things on the left, right. Click on that and then copy as curl. Right. Okay. Cause you, I was like a weird, like SQL like thing. I don't have anywhere to stick this. I just put it in a text file somewhere or, um, so we can see what graph QL looks like graph QL. I am not a fan of graph QL. Is anyone, anyone in the chat? Like, I have no opinion on graph QL. Whoa. What am I looking at? Why is this so ugly? That's an excellent question. Is it by those little cookies? This is the cookie. This is all cookies.
Look, if I got rid of that, how many could try to see what the actual payload is. I'm trying to see what the actual payload looks like. Can you scroll down a minute? Yeah. Uh, I don't actually see a payload. So weird. Anyway, graph QL is kind of this quasi SQL like language that you can, um, that, that some APIs use. And so instead of writing, um, instead of there being like the JSON payload or a bunch of question marks in the, in the, in the, in the, in the URL, there's like this kind of quasi SQL JSON language you have to write. Um, and it's, I really don't like it.
Handling pagination
Yeah. But if you do hit something paginated, um, and YouTube's API is a great example. It's like 50 per page or whatever. It's going to give you a token that corresponds to the next page. And then when you write your loops or whatever you're doing, um, you just have it take that token for the next page and insert it into the next page token of whatever your API call is. And you can page through it that way. Don't let that stop you. Like use chat to BT to help you navigate the pagination stuff and understand it. Um, that had really, really slowed me down when I was first hitting API. So it was like, I hate pagination. It drives me crazy.
Um, it's so funny that Josh Starmer was the one that popped up on LinkedIn. Cause he's going to join us on the hangout on Thursday. If you're a Josh Starmer fan coming out with us on Thursday, last hangout of the year, it's going to be fun.