Transcript#
This transcript was generated automatically and may contain errors.
All right, everybody. I am so excited to introduce our featured lab manager for today, Karan Gathani. Karan, would you like to introduce yourself and tell us a little bit about what you're going to be talking about today?
Sure. Thank you, Libby. Hi, everyone. I'm Karan Gathani. I'm a QA engineer on the Shiny team. And today, I'm hoping to talk about one of the projects I've been working on, which is using LLMs to generate Shiny apps, which I think a lot of people might have tried. But then trying to evaluate when they do generate the Shiny apps, how do we gauge the quality of the Shiny apps? And like I said, I'll go through this thing where we started off with one thing, and then I found some discrepancies and started digging into that. So I'm hoping to share those things, because for me, that is super exciting and a thing that I did not think of when I started this adventure. But yeah, we are exploring and discovering new and new things. And so I'm excited to share those things with you folks today.
Well, we are excited, too. This is a place where we do tons of screen sharing. So I say, Karan, go ahead and share your screen, and let's get started. And in the chat, I would love to hear who is using LLMs for making Shiny apps. Hubert is already saying, yep, I use Cloud for this all the time. I think a lot of people do. I, myself, like using it to create utility Shiny apps that I will use and throw away. No one's ever going to see them, but they're going to help me do an interactive task, whether that's data labeling or something else.
Generating Shiny apps with LLMs
Okay. So for folks who haven't ever created a Shiny app using LLMs or agents or whatever, it is pretty easy to get started. Right now for this demo, I use OpenCode. OpenCode is just like an agent harness. It allows you to talk to any LLMs, provided you have a subscription for that. And they have some free things. But the goal of this is it's similar to how you use Cloud Code or you use Codex or like something else, right? There's so many options. But this allows me to quickly change different models. So I can actually look over here, I can quickly go and select different models. And there are some free models and there are like the proprietary models. It's pretty easy to do that.
And so for this example, let's say I'm selecting one of these. And for me to actually just create a Shiny app, it would be create a Shiny for our app using, and I have a data set. This data set is essentially an Airbnb listing of an actual neighborhood. And it has like all the ratings and prices and stuff like that. And right now, like I said, I'm not trying to actually look for anything specific, just point out to the CSV. And that allows the user to look at some interesting trends or whatever. Like I said, you can change it to however you need it. This is all based on vibes, right?
And so the good thing about this is while it's like going on, and this is using a DeepSake Flash, which is like one of the cheapest models, you can actually see like how many tokens are being used, what's the cost being used, like how much you are spending and stuff like that. So you can like do it in real time and go through that and do a bunch of those things, right?
And for me, though, when I'm actually evaluating one of the questions I frequently get is like, how are the LLMs doing in generating like Shiny apps or like in certain tasks? And that's one of the areas that I'm super interested in is how are the models doing? Like people have like different kinds of models that people are using. And like people have loyalists just like sports team. People are like, I am on team Anthropic or I'm on team OpenAI or I'm on team whatever, some other open weights models or whatever. I'll go over that later.
And so for me, it is actually like looking objectively, which is like, can I show evidence on which model is doing better than this? Like if there's a new model that comes out, can I actually say, is it better than the previous one? Because some of the models are getting more expensive as they are coming out. And so one of the ways in the Shiny team we want to do is we want to evaluate how these apps are being generated or how these apps look like.
Right now you can see within OpenCode, it will try to like create that. It'll give me like, hey, you want to run this. Some of the models that I have noticed is like the cheaper models, they'll run something or like they'll generate an app, but they won't like run it to see if it's working correctly. Some of the more expensive models will actually like go back and like, okay, this is the work I did. Let me actually see if it's actually working. And so one of the things I have noticed is usually with the cheaper models, sometimes occasionally you will have like issues.
Did I miss my terminals? Run the app and figure out the issue. But yes, I don't want to harp too much on this one. The goal of this is like sometimes the apps will generate something, they'll run into an issue. And then you have to go through several iterations, which like I said, you can use a cheap model. You can spend time and fixing it, but your time is precious as well. And I think like the more time you spend in actually trying to direct these models, I think you are spending a lot more effort and time.
What are evals?
And so one thing that we do is anytime we are actually coming up with like any LLM integration within the Shiny team or we want to do something called as evals. So if you want to write in the chat, if folks have ever used evals in your products or in any LLM integration they all have ever done.
Are you gonna make a poll? Yes.
All right. Did that work? Yes. Okay. I'll add one on Zoom as well.
Well, I'm gonna say no. I'm in the no camp. It looks like most people are in the no camp.
Live feedback on this poll is that over 80%. No.
Perfect. So, okay. So let me actually go back to just explaining quickly what evals is. So evals, let me, you know what, let me actually show my whiteboarding skills, which are terrible because, okay. So essentially an eval will be like this. Eval. If you want to sound fancy, you can say evals or with a Z. I don't know. Whatever works for you. So evals essentially has different components. And this is like similar across any evals you will see in the wild. And this I think is a really important concept that I think is catching up more and more. And so evals essentially has four concepts.
So it has a concept of a data set, has a concept of a solver, has a concept of a scorer, okay. And then the overlying of a task. Okay. And I'll break it down into simple words. So what data set essentially is, is data set, think of it as like a paper, like a score card, which gives you the question and the corresponding answer. So if this is a test sheet that you want to give for a pop quiz or something, what is the question and what is the corresponding answer? That is something that you would end up giving. And so data set essentially is this like key value pairs almost. It's a, you can have, it's like a CSV format or JSON format. And essentially it is like, which is the question and what is the corresponding answers. Since these are LLMs, they are smart enough that they do not need a singular answer. You can describe your vibe.
they are smart enough that they do not need a singular answer. You can describe your vibe.
And I'll give you an example of how this would be. So once you have that, you feed that to a solver. So solver is essentially this LLM, which like I'm representing it with this robot, like beep, beep, boop, boop, or whatever. So this one, it, you give them the question. They don't have access to the answers. You just give them the questions. So they give the question or whatever the question is like create a shiny app or whatever. And then it will actually take that in and do whatever calculations it's doing. And then it will actually spit out the answer.
So this is one LLM and it can be any model you can think of. It can be Opus, it can be OpenAI, it can be OpenCode or whatever. And then it spits out that answer to the scorer. And scorer is someone, another LLM that's wearing glasses. So it can actually like look in detail on what answers this one gives, but the scorer actually has access to the test sheet as well. So they can actually take a look at this and it'll be like, okay, let me look at the answers that were provided to me and let me look at the questions. And then they'll be able to compare and then they'll be able to give a rating or a ranking or scoring.
And the scoring usually is, so it uses something called as complete, has partial and it is incomplete. So if it's complete, you get a one point, it's partial, you get 0.5. And if it's incomplete, you get zero. And for all of these, you get scored and you can have 20 questions, you can have 300 questions, whatever. And then overall you'll be able to see. And the goal is to keep on running it multiple times to see how well the LLMs are doing. Because LLMs are mostly are not deterministic unless you lower the temperature quite a bit. And so to see if they are actually trying to be in the ballpark.
And like, this is a different concept than when people are like writing unit tests, because unit tests you are essentially, or any kind of test you are, it's like, okay, what is two plus two? And then you're like, the answer should be four. This is not that, because LLMs are not, I mean, you could use LLMs for two plus two, but that's not what LLMs are like the most powerful at. It is like something that is used to synthesize data, or like create code and stuff like that. And judging them is also hard because it's not as straightforward.
The vitals package
And so to show an example of how this might look like, I'll show you an example in Positron. And so there is a package developed by Simon Couch on the Posit. And he created this, which is vitals. And it's essentially like checking the vitals of your LLM integrations. And the way, like I said, I'll go through the key concepts, which is you had the input and the target. So this is the data set. And so this is essentially asking if this is the capital of France, what is the correct answer? And again, it could give a one word answer as Paris, or it said like, oh, the best place in the world, Paris, or whatever. And as long as the LLMs know, because it's an LLM who's judging it, they can deduce that Paris is somewhere contained, they don't need to do a regex or something like that.
So something like this, wherein it's something like a little more on the fence with like, should pineapple on pizza be allowed? I don't know, might drive a lot of people off. But again, like you can have whatever answer you expect the LLM should be giving kind of. And then you can go into like much more fuzzy topics where in like, okay, how do should we tackle if there are three different things and maybe something with like a little more like ethical dilemmas. And again, you have to provide what you think the answer should like be closer to kind of, because the LLM will give you an answer no matter what. And you can actually give and more context you give it, the easier it is for when it is actually judging it.
And so essentially, so we create the task. So you remember task was the overarching like bucket, we create the solver, we create the scorer, and we create the data set. So those are the three concepts, like I said, and we have three questions, we have three answers, the input and the target, we have the solver and scorer. And I'm using a different model for creating the like, who is solving it, and who's judging it, it's just different models kind of, you can change it across different providers and stuff like it's really easy to mix and match. You can also have a panel of judges. So they will actually, you can have
multiple judges over your judging. Okay, and so over here, if I wanted to run this, I would just just do I think, yeah, just ask if Yeah, I have. So I don't know. Yeah, so smart. Okay. All right.
So so we are you see, it's just solving that. It's working through all the examples, and scoring it and we look at the results as well. And the results are should be over here. Let me stop my code. Let me look at eval report. So it's it can create an HTML file. And we can go live and check on this. Okay. So it looks at that. This is the target. This is the answer it gave. And you can click on each of them, but it will show you that this is the question you asked. And this is what it gave. And then you can see how the judge is actually reasoning as well. And the explanation on why the judge gave an answer, which is like, this was a complete because it thought it was really close to the target. And so that's what it ended up coming with. And like I said, some of them will go into details. And you can always give more instructions. So like, if you're going to lengthy, you're just blabbing kind of like just talking about a lot of stuff instead of being towards deduct marks. And yeah, it will follow those things. But yeah, so we have a different judge doing that.
And so this is like the concept of evals, which I think as people are switching into like relying on LLMs for every single thing, or like a lot of things, I think people want to be confident that it is not going to hallucinate or it's not going to give you things that will embarrass you in front of other people. And that's why I think one of these things we wanted to do is the evals kind of. So this is the concept of evals that I showed. And this is like an example. It exists on Python, as well as R side, R side use vitals, Python use inspect. Vitals is built on top of inspect. But same concept. And you will see other packages that do something similar, but these concepts are universal of data set solver and scorer. They might use a little different terminology, but essentially, this is the whole concept. Okay, so this is the just evals in a nutshell, what ends up happening.
And so this is like the concept of evals, which I think as people are switching into like relying on LLMs for every single thing, or like a lot of things, I think people want to be confident that it is not going to hallucinate or it's not going to give you things that will embarrass you in front of other people.
Evaluating Shiny apps generated by LLMs
And so one of the things we wanted to do in the Shiny team is when we actually have a Shiny, like we give the task of LLMs generate Shiny apps. And we want to do the evaluation, we want to do some judging, right. And so we started off with like judging on like four different things, like requirements, fidelity, like how maintainable is the code? How good is the visual quality? And like, how robust is the code? Like, does it do defensive coding, but then it can handle, like, it should not like this flop, when you have, like, CSV or like an input file that is like, not properly structured or whatever. So it does it like do defensive practices and stuff like that. And there's a lot of these things, which are like, all the stuff that you would see, like, oh, you should use these for best visual design for dashboards and stuff like that. And we'd like follow those things. And then asked it to like, judge, right? Like, and so I ran it across. So your answer, there's this repo that I work on called shiny gen, it just allows me to generate a lot of different, like the same task across different LLM models, right? Like, and so we'll have like 76, like the same task given to different 76 different, like scenarios, and then trying to judge who is doing better kind of like judging what scoring they get.
And so as an example, for us, it is super easy to actually just run something in like, given the same task, like I can actually run, like which framework I want in R or Python, which models I need, I just need to, like separate them by a CSV. And this is I'll talk about the skills and vanilla one, which data set I want to have some data sets that they can use, which prompt I want to share over your and then what is the maximum iterations I wanted to like keep on iterating on. And then this is a quality score of the judge. Like if you do not get that score, go back to the drawing board and keep on working.
But yeah, so I think we have it. I have to stop for just two seconds, because I think Isabella and I are both baffled by what we are even seeing. Is this a GitHub action? But yes, this is amazing. Yes. So but it is essentially doing an evals under the hood. Okay.
All right, when you construct the workflow, does it automatically build that drop down and their run workflow? Yes, this is my question to get that part on the right. So you can when you design your workflows, you can design that you want to drop down with any values you want. So anytime there's a new model that gets released, it automatically I'll add support to it, and it'll automatically like show up over here in the LLM. So I can show you the workflow, but essentially, we have like a bunch of these workflows that you can run. And then you can select like what options you want. And anything under options is something you can select from a drop down. Okay, amazing. I will I will not detour you any further. But we definitely need a data science lab episode on GitHub action workflows, because my mind feels a little bit blown, but I'm gonna let you continue.
Yeah, yeah. But like, just to show you an example, like, again, like, I'm not gonna go too much on these. But when we run these, again, they do end up like running for quite a long time, because we are trying to run it across multiple things. And so when we select that, like we'll get through like, what was the cost of each of them? How much long it took? How many iterations and like you can see, it's running for Shiny for Python, Shiny for R. And we'll run it across all of these models, kind of to actually just see like how the models are able to deliver and stuff like that.
Running evaluations in Docker and introducing skills
And so one of the things that we want to do, because we want to be as scientific as possible when we are like doing the evaluations. One of the things the Shiny team is trying to and this is the whole concept of how it works. When the user just decides they want to start this evaluation, it goes through everything runs in a container, because we don't want to like, have like a personal configuration be involved, because people might have skills or anything that might affect and like not be reproducible. So we want to run everything on a Docker container. And this is where we actually do the control and treatment.
And so let me actually show you on a mirror board, how that would look like. So essentially, one of the things that you will hear is something called a skills. I don't know if people are aware of something called a skills that is like a word that is being thrown around a lot in the AI world right now. But skills essentially, if I can like draw is imagine you are getting a bagel every day on your way to work and you're like, can I get my and it's like maybe everything bagel with like, just regular cream cheese. And that's what you would get. So which is like everything bagel. This is the poppy seeds or whatever. And you get a cream cheese. A skill would be you tell the person who's making a bagel like can I get an extra toasted and can you be like a little light on the cream cheese because I'm trying to watch my calories. That's a skill. So LLMs can make a bagel, but like that specific thing like extra toasted and light on cream cheese. That's a skill because now you want it to be reproducible every single time. And so that you can be a skill and you're like anytime you make a bagel, make sure it's extra toasted. So that's a skill. You are mentioning it to the agent or the LLM and then make sure it's less on cream cheese because I'm watching my calories or whatever. So that's a skill that you can add.
So one of the things when we do skills or when we are trying to make these skills is to actually see if the skills are actually even better, like making lives easier or like making the dashboards even better. And so what we do is when we give the prompt, we'll give the same prompt to an agent or an LLM with skills that has access to the skills and one without. Maybe we are just over engineering and maybe it's not something that's even helping the case. And you would always think you are making things better, but that's like a really subjective view. And so this one, no skills. And so we'll get that. So when we run those things, we'll do it across. So we have two arms, we have with skills and without skills. And skills essentially is like the best way to have plots, the best way to do tables, the best way to have value boxes and stuff like that. LLMs can generate value boxes, but maybe if you want to make sure you use icons or font awesome icons on your value boxes, that is a skill because you want it to be structured a specific way.
And so we do that. We run that. Within that, it will take a screenshot. And then if you have judging enabled, the judge will actually look at that code. It will also look at the screenshot and it will actually evaluate and give it scoring. And this all went pretty well.
Human vs. LLM judgment discrepancy
One of the things I noticed when we are doing it for so many different iterations, when we had the 76 kind of things, and I asked it to rank it based on which dashboards, because if you just give it in isolation, it will be like, oh, this is a 10 on 10, this is a 10 on 10, this is a 10 on 10. And you're like, I don't know which is better than if you are saying everything is a 10 on 10. And so when you ask it to do relative ranking, one of the things I realized is like, I wouldn't agree with what it thought was like a number one or whatever. So I did an exercise wherein I actually created the top 20 that the LLM thought were really good dashboards, created a Shiny app, wherein it was a double blind test, wherein users have to look at the Shiny dashboards that are generated, and then rank it in the order of like, top to like the bottom 20. And people would like essentially like move it from like, if B is better than A or whatever. So they'll rearrange it that, and then they can submit their ranking.
When they submitted that ranking, this is where the shocking part happens. I created a Quarto dashboard is the model, when the LLM actually judged as number one, was like, number, I believe, 17 on what like, the what most people were judging as. So I was like, okay, there's a clear discrepancy on what LLMs are thinking a model is like, like a good dashboard. And what humans are thinking are quite different. So again, when you look at the number two that the LLMs thought were good, it was like number 11. And so this one, when I ran it, I ran it through two judges. So I ran it through, I think I believe at that point, it was Opus 4.6. And then I gave it to GPT 5.5. Like, do you all agree with this ranking? And then both of them said, yeah, this looks really good. But when I gave it like the double blind test to users, they did not agree on whatever was showing up.
When they submitted that ranking, this is where the shocking part happens. I created a Quarto dashboard is the model, when the LLM actually judged as number one, was like, number, I believe, 17 on what like, the what most people were judging as.
And then number three, I think was getting a little closer and stuff like that. And the reason one of the questions is why even test all these models, right? And I think as we are progressing in the LLM age, one of the things at least most people might have started to notice, and I at least see it, is there is a lot of emphasis now on how much are we spending? And can we be more conservative on the amount of money we are spending on LLMs? Or can we get an idea of how much we are spending? And do we need to spend top dollar to get the best quality? And this exercise was a way to answer or even figure out if we can actually try to answer that question. So you can see how the discrepancy is just too wide at this point, what the LLM is rating or the LLM is judging as the best quality and what users are thinking as the best quality.
But okay, and so this one actually you can see if which ones are using skills, which one I use in vanilla, if it's a Python or R or whatever. But anyway, so this was the first iteration that I realized like, okay, so there's clearly this difference between what users are judging as a good-looking Shiny dashboard. I do feel LLMs were pretty good at figuring out like, at least the more expensive LLMs were figuring out where in, hey, is this code maintainable? Is it robust or whatever? But in terms of visually, they would start falling flat really quickly, like some of those that they would actually think it was good. When you would look, you'd be like, that's so basic, like, it's like pretty meh. Like, sometimes they would fail in rendering and the LLM would think it is like, oh, yeah, I like the minimalistic design. Like, there's nothing rendering because it didn't work. Yeah, yeah. So like, so those are the things that
like, I think humans have a way in like, able to find out like, this does not look right. And I feel like LLMs were like, LLMs were really good at expressing what a good app is, but they were, I felt they were not good at actually judging what a good app is just by looking visually and stuff like that. And there's like explanation I can do on why that is the case.
But that's at least something that I initially saw that was happening here. So that process ended up actually kicking off like a new thing that we ended up doing that I can share as well.
Responding to audience questions
So let's see. I think this relates to like, one of the questions that we got on Blue Sky, actually. So for the judging that you use for this experiment, was it only focused on aesthetics? Because the question was, like, quality gaps in LLM generated Shiny apps often show up in reactivity, logic and module structure. Curious what you use to evaluate good.
Yeah, so when we do the screenshots, because if you look at across the board, every model has a different version of what makes a good Shiny dashboard, like when it's actually doing it. So some of them you'll see, they'll be like, take everything, just one page. Some of them will actually break it out into different nav tabs. And so for me, when I'm asking the LLM to judge, so in order to take a screenshot, I would be like, okay, make sure you like check something in a sidebar. So like the reactivity graph kicks off, and then something gets updated and then click through every nav tab. So if the reactivity would have broken, we would have caught the screenshot in that case, right? Like, because that would like be able to see if the reactivity thing was broken.
But like in terms of the question, in the judging, like I said, we do it on four criteria. And the criteria for the judges is like, this is the visual one, but we also do based on code maintainability, like how well it is and stuff like that, we'll check. And this is just in general, in any dashboard or any apps people are generating, like how robust is it, how maintainable the code is kind of, we do that. But again, like I said, when we do that, like the other ones I was able to like agree on what it was able to find.
But the delta I saw was the highest in the visual and UX quality. And that's where like that investigation like started, it's spurred off that investigation. And I started like digging more and more on why that is happening kind of. So I hope that answered the question. Yeah, very cool. I'm also curious, did an LLM write this? The judge? Yeah. So this one is yeah, like in the point, breaking it down into 10. But essentially, I was looking at like, how do you judge like on online, like you can, how do you judge a good app, and then you have like a bunch of articles, and I was like, alright, synthesize it for me, and give it a grade of top 10 or whatever. So okay.
Discussion on OpenCode and model types
Any other questions before I can go do something else? Yeah, we had a question from Nathan earlier on about OpenCode. There was a lot of excitement about OpenCode. And I have never seen that before. That is so cool. The question was how this OpenCode compared to Posit Assistant.
From what I remember, you can add a custom or local LLM as a back end and Positron, which also kind of adds that agent harness. Yeah, so OpenCode, again, it's just an agent harness that you can use. The reason I use OpenCode to test it, like I'll use Cloud Code, I'll use Posit Assistant, depends on different things. But one of the things I feel Posit Assistant is really like, I think, in terms of like the tweaks that they have made to Posit Assistant, I think it is much better at like looking at graphs and being able to tell if like, there are tweaks that are made in Posit Assistant to be able to like render or like actually be able to info a graph that is being shown up, which in OpenCode, I think it's like you can do a lot of things.
The reason I use OpenCode and I just just use it for this one is I use a subscription called OpenCode Go. I can show it over here. And it is, again, like I wouldn't like recommend people use that if you are using like sensitive company information or whatever. But OpenCode Go is like a subscription plan you can use. And it's like five bucks for the first month and 10 bucks for every month after, but allows you access to all the open weights models and they get updated right away. So you will be able to test any open weights models that come in. And I can cover what open weights model mean. But it is like again, it is so for me, it is like a flat fee that I can use it for testing all these different things instead of paying individual by tokens kind of that makes it much easier for me. And OpenCode Go is like one of the few providers that allows you across like I can buy a subscription, but then when will only allow me when this one allows me all the top open weights models.
So for folks who might not know, and I don't want to. So we have like these different models. So you have something called as proprietary models. I did as this and this is like open AI and Anthropic. And then you have something called as open weights models. Weights. And then you have something called as open source models. So the open source and open weights are not the same. Okay, so, so we have these things. So open weights models would be something like GLM 5.2, which is like the new rage that people are talking about. It's minimax M3 and like bunch of different things like deep seek or whatever.
So think of as proprietary models as like trying to think of an example that people would be able to relate. Let's say you go to Starbucks and you go order your coffee, like whatever, espresso or whatever. And Starbucks gives you that, whatever, espresso or like latte or like that rainbow latte or whatever, like the pink, blue, all the shades of the rainbow on your latte or Frappuccino. And, but they won't tell you the recipe. They'll make it and give it to you. So you get what you asked for. And this is your whatever Starbucks, I can draw the Starbucks, whatever you get this thing, which like loads and loads of whipped cream, which is don't get me wrong. It's really tasty. I just have to watch my calories.
Okay. Open weights models is they will actually give you the coffee, but they will allow you to actually adjust the shots of coffee. They'll allow you how many pumps of syrup you want. So essentially there are these different knobs you can use that they can allow you. So you can tweak it a little bit based on how you want your coffee to be. You can do a little bit. You can't do one thing. They won't tell you is how they source the beans. They won't tell you where they got the syrup from. They won't tell you. The only thing you can do is like based on what you got is you can adjust like how many pumps of syrup or whatever coffee you need or whatever. If you need it like French toast or I don't know, whatever coffee terms people use.
Open source is everything is open. They'll tell you where they source their coffee beans from. They'll tell you where they, how they made their coffee, the whole recipe, everything is out there. So there's not that many open source models. There's a lot of open weights models. Open weights models, you can actually host it on your server and you can run it locally. Some, yes, like the smaller models, but the bigger ones you cannot, you can store it on a server and run it. And so if you want that data to just stay with your company, you can use these. Open weights models do have different kinds of licenses. So you have to check if you can use it for commercial applications or not. So, but this is like the open weights models and open code go allows you to use all the open weights models. They claim the, the open weights models are like stored somewhere on a server and it won't be trained. I don't know with 10 bucks, I'm not really sure if like they hold true to that. But again, like I said, if you're just trying new things, I think it's good. I cannot vouch for like how safe it is for your like company data kind of. So you just want to be a little skeptical anytime you have something like that. But it is really good to actually try. And open weights models have come a long way. I'll show you one of those examples.
But yeah, so these are, these are the difference between open weights proprietary models and open source models. Most people, when they're talking about like, oh, open, like people use these interchangeably, but they are not because open weights, they don't tell you where the beans came from. They don't tell you the recipe.
Human ranking experiment and skills evaluation
So, okay. So, okay, cool. Going back. So I did that. And then one of the things when, so on the Shiny team, we are working on like the skills to see if the dashboards are being like made better. And we wanted to see. So, okay, before I actually cover the scale, let me actually show you the, no, maybe. Okay. So this is like the whole range of pricing that we have. So you have something that's like really cheap, like a deep seek, we flash to like a really expensive, like whatever, the Claude Fable or whatever, which I did not get to test it, which I like, even I don't usually test things on like the most expensive models, because this is not rocket science, like Shiny app. I mean, you can make rocket science with the Shiny app, but you don't need to just spend a lot of money to make, like I tested on Opus, but I did not feel like I wanted to test with Fable when the access was there, just because you can like just dump in a lot of money. And at some point you will get things that are like similar.
One of the things when we test, we do not give explicit instructions. So if I go back to my Shiny generation, we'll give something that's a little more vague. So it will be like a build a professional grade Shiny dashboard for this data set. And then like these things, because if I be super prescriptive, it becomes really hard to judge because everyone will look the same. It's like saying, like, make sure everyone wears Nike sneakers tomorrow or whatever. And it's really specific, like whatever Jordans, I don't know, but like Jordans, the 1990 Jordans or whatever. And then everyone just looks the same. And it's really hard for me to judge what creativity did the LLMs bring. And so that's why the instructions are a little vague.
And so that's why the instructions are a little vague.
And I wanted to see how well they do. Okay, so this is that. And then the pricing, like I said, it's all over the place. You have something that's really expensive and something that's like a fraction of the price kind of. And so the second exercise that I did, so we worked on the scales, made the scales a little better. And then I wanted to understand what actually humans, what kind of dashboards humans are looking for. And so again, this was another double blind test where humans can actually look at the dashboard, and then they can rank it into these tiers, which is what the Gen Z do these days, like S to F tier. And so you will essentially be like, taking like, oh, this is an S tier, or this is a B tier, which is like resonance with Gen Z. So if you have any Gen Z people you're working with, if you mentioned tiers, they'll like, be able to do it. But essentially, this is another double blind test, like I don't know which tier is which had 24 people submitted. Again, this was not judged by the LLM yet. Because right now, the goal of this is to actually understand what kind of dashboards appeal to people, and trying to learn those patterns, and then like formulate it as a skill, so that we can have the LLM judge based on that, like I said, there's a lot of trial and error that will happen.
But one of the things that we started realizing when we are doing these, and these are like different, how people are judging this, like the, the honeybeehive graph or whatever, is you would have like some of the proprietary models that were being rated pretty high. But then you would see some of the models that are being rated like really high in the list, they are like a fraction of the cost of how much the more expensive models are being. And so I think this one now actually, we are trying to think is like, what do we have the more like the cheaper open weights model do compared to what the proprietary models would be doing. And so like, again, this is something that's a work in progress. I am planning to do a lot more local models as well, like something I can run on my machine to see how they generate, how they compare kind of, but one of the things that we do since this is double blind, people don't know if they are creating something with skills or without skills, this at least tells me if the skills are actually making a difference.
So in this case, like, does the skill have an advantage over vanilla, maybe the LLMs are good enough that they don't need skills. And so this is a way we are measuring that same thing, like, are we doing like better with the skills in our, how do we make sure it's like level. And so those are the things that we are evaluating right now, making sure we keep on testing. And so and then hopefully, like I said, like making sure the judges are judging in a way that is similar to how humans would be judging as well. But yeah, I know there's a lot of stuff. But yeah, let me wait here for a second if people have questions. Who's coming out on top so far?
All right, right now it's a GPT 5.5 is the design that most people like.
But yeah, so like I said, that's something and this is not like the only thing because right now I'm going to add local testing and then we are constantly changing our skills to make sure something that did not work. So in this case, anthropic models fail. Like I mean, they did not fail massively. But like one of the things that happened is we found out that in our skills, we had examples that were using the test data sets in the R. And one of the things like, and at the bottom, we mentioned that do not use test data sets when you're creating it. But somehow the one on top, it would always use it. So even though you gave it a thing, it would end up making something with test data sets. So that's why it was ranked like pretty low in the list. It made a good app, but it did not follow the instructions correctly. So hopefully, so we are tweaking the skills to make sure it can be better in the next run.
Very cool. Are these skills available? Like can folks go and try them out now?
So the Python one is still being worked on, but the R one should be, I can send the link, but yeah, there should be in the deposit there. Okay, I can look it up. Thank you.
Q&A: open source, Google models, and evals
Okay, we had a couple more questions that I would love to get in really quickly, because we only have 10 minutes left. Libby McKenna had asked, is there an SOP slash user guide for your whole process? Do you have this open sourced? Yeah, so this is in the Shinygen repo. I have like, Libby or Isabella can share the links later, but it should have all of it and it's like all public.
Okay, great. And then the last one is someone asked, I don't see any Google frontier models on here. Is there a reason for that? No, there's no reason. So we wanted to test Google once again, like it started off like earlier, it was just actually trying to see the difference between OpenAI and Anthropic and then we ended up adding, but yeah, the goal would be to add like the Google one, the one of the reasons we stopped adding because Google, so Google release cycle is like different from other models. So usually OpenAI will do like a six week release, like a new model. So one of the things is Google right now has this thing, like it has 3.5 flash, which is like their latest model, but their earlier model is like pretty old, like three, whatever, the 3.1 probe. I think they are going to plan it by the end of the month, but the delta between their model is like, I think it's like closer to five or six months. And so I did not want to run something that's outdated and then have a new one come up. So hopefully when the new one comes up, we can actually run it at the same time.
If it's all right, I'd love to add on to Libby's question about like, if there's SOP for this, like if we wanted to do something like this, but with something that's not shiny, do you have any recommendations for that?
I mean, depends on what your task is, because like the whole thing about evals, so evals is the underlying concept, which is like, can you have like a quiz that you can give your LLMs and can it like run that quiz every single time and like give results that are in line with expect, because the whole thing with evals is you are essentially generating the targets. You are actually telling what the LLM should be generating. And so I think you get to define which direction you want the LLMs to give you an answer. And if it's not giving the answer, you can make a skill to tweak it, to take it, like take the horse to the water. And so that's what essentially skills are. And evals actually checks, is the horse going close to the water? So that's the whole loop. So you keep on tweaking your scale to see if the horse is going. But like, that's the thing is like, you don't want to just rely on that, because when the new models come, they might be good inherently, they might not even need the skills. And so making sure that's why we have the vanilla and the scale arm to actually see if that is happening or not.
And evals actually checks, is the horse going close to the water? So that's the whole loop. So you keep on tweaking your scale to see if the horse is going.
Very cool. So many considerations. And there's time for one more question. We got a question about the, could you talk a little bit about the dashboard for performance evaluation? Is it a shiny app? And can it be customized for other projects? And then thank you for the explanations. Sure. So performance as in like, I'm just want to clarify. So performance as in like how well it does with like a lot of data or like a lot of different reactivities. Is that what was meant? I think it was just the dashboard of the performance of the LLMs, like the dashboard you showed that have the ranking. Oh, yeah. Oh, yeah. Asking if that was a shiny app and if it could be customized for something different. Oh, okay. So, so I had several dashboards. So this is you did. Yeah. So this is a shiny app. This is not again, like, so this is where, when you are using an LLM is the LLM will use a lot of custom CSS and JavaScript. So it's a you, you can use it, but it's not like a really clean, reusable way because they will like get things in a way how you want it like to get the task achieved. This is not in the best, like reusable format that people should be using kind of. But yeah, ideally, LLMs are pretty good. You can give it like, hey, I want the shiny app in this way. And they could like do a bunch of different things. But this one actually has a lot of custom JavaScript built on top of it. So it's not like it is a shiny app, but like shiny app, you can expand it however you want it. And but this is a lot of custom, like I said, stuff on top of it.
Why LLMs struggle with visual judgment
Okay, awesome. Well, we have five minutes left. Is there one more thing that you wanted to chat about and show? Sure. So one of the things at least like I will like to talk about is, again, like, when I'm trying to find my mirror, but okay, that's okay, let me Okay.
So one of the challenges that people are like, why are LLMs so bad at visualizing or whatever, like, and I think it's just understanding why, how LLMs process like images. So anytime you have an image of a dashboard, because we'll take a screenshot of like something over here and some graphs. So whenever LLMs you feed it to the LLM, which is like, hey, can you tell if this is a good or a bad thing? I think there's a lot of translation that goes on. And I think things get lost in those translations. So anytime it gets over here, what it does is actually it like breaks that image or it downscales the image. So it will like, let me show it like this. So it'll downscale the image like this into a smaller one, because the LLMs can only process like something that's smaller, it'll actually break it into tiles like these, and then convert each tiles into like vectors. It's a bunch of numbers. And each of them will actually be able to like, say like, oh, does this have a text? Is the text? Is there a box? What are the colors or like something? So the vectors will present that. And then there is something called as an adapter that actually takes all this like tiny information and it tries to synthesize that to an LLM. And so then the LLM is sitting over here. And it's like, okay, looking at all of this, let me judge if this is a good dashboard. But you see, in all of these things, something gets lost. And that's why like LLMs are at this point, not that good at it, because it's trying to like downsize it, cut it and then join it again. And that's where I feel like things are not being perfect. When you are looking, you are being able to look at the whole thing, all at once, you are not like looking at tiny keyhole things and trying to figure out what is happening. And so that's like essentially the reason why LLMs right now are struggling with visually judging something. So just wanted to give some context, like I'm not saying like, oh, LLMs are bad at this. Like it's understanding why they are not being able to do it. I think that's equally important.
But you see, in all of these things, something gets lost. And that's why like LLMs are at this point, not that good at it, because it's trying to like downsize it, cut it and then join it again.
Yeah. It's like a computer vision problem, right? Where you're like, let's break this into numbers, basically. And then you sort of lose a lot of the context of the image composition and why it might be good or bad, appealing or not to human eyeballs and the human brain. Well, Karin, this was amazing.
There was a part of our discussion where the chat slowed down so much because we were all just processing what you were saying. We were like, what is happening? I am so glad that there are smart people at Posit working on this problem and thinking about this in a detached and disinterested sort of way and making sure that we're doing good things and not just moving forward blindly. I think that this was amazing. If anybody is curious, they can go check out Shinygen, the repo. But also I wanted to reiterate for everybody, this is like your job, right? You are doing this for us so that we know that our tools that we're using are good at what they are supposed to be doing. It does not fall to us to do this for ourselves, but we can take this forward and adapt this for the things that we are trying to build and make better. And I think that that is what I would like everybody to take away, that you don't have to go replicate what the team is doing, QAing, all of this stuff. But all of these little bits and pieces can be taken forward. Thank you so much. I hope you had a good time.

