Transcript#

This transcript was generated automatically and may contain errors.

Hello, and welcome to the Python Exchange. My name is Cameron Riddell, and I will serve as your host this afternoon. Today, we are joined by a special guest, Rich Ianan, whom you all may know through his various open source projects, Great Docs and Great Tables. If there's a good chance that his project is great, then Rich has had a hand in making it. But before I share more about our guests, let's first introduce our panelists who will help us drive discussion. First up, today, we are joined by, as per usual, James Powell. James, for anyone who has not yet had the opportunity to meet you, can you answer this question for our audience? James, what is one of your favorite Python topics as of today?

Hi, everybody. As Cameron said, my name is James Powell. It's a pleasure to be here. One of my topics that I've been thinking about a lot recently related to Python is, what might come after Python? I know that might be a very controversial thing, but I've been thinking a little bit of what the next language, the next hype, the next thing might be.

Fantastic. Well, James, I definitely want to hear your thoughts on that. However, we have some other hosts who are happy and chomping at the bits to share their favorite Python topics as well. Next up, let's hear from Tom Caswell. Tom, can you answer the same question just so our audience can get to know you a little bit better? And that is, what is one of your favorite Python topics as of today?

I think one of the things I'm most excited about is the new debugging, the new profiling, the sampling profile that is coming in Python 3.15. That is a thing I'm looking forward to being able to use for good.

That's the Tachyon profile, right?

Correct.

Oh, fantastic. I know Pablo spoke a little bit about that at PyCon US, and I missed the talk, so now I have to go watch it. But excellent to have you here today with us, Tom.

Let's also hear from our host, Tani. Tani, same question to you. What is one of your favorite Python topics as of today?

I think it's my background in AI and ML. I think it's definitely related to how can we build Python-based pipelines that can receive the fast-paced advancements of AI and ML and new things coming up. So how to make sure that we can do this transfer learning appropriately across different projects.

That is fantastic. It sounds like a very complex topic of very complex interest, so I love hearing that, Tani. Thanks so much for taking the time today to join us.

Last up for our host panelists, we are also joined today by Dan Allen. Dan, can you also share with us what your current favorite Python topic is?

Sure. I'm excited to see how the free threading work forces me to rethink the assumptions that have been true about Python for as long as I've been doing it. I guess that pairs well with the sampling profile that's going to check my intuition.

The alternate version of that, Dan, is all the bugs that will be found due to the security. We're just waiting for the re-implementation of the GIL then and Python 3.16 so we can have it back.

They need to strip out the JIT work first.

Yeah. Excellence. Well, these are our host panelists who, as I mentioned, are going to help us drive some discussion. And what will we be discussing? Well, we are going to be discussing a topic that our guest today, Rich Iannan, is very passionate about.

But before Rich shares about that, I want you to all get to know Rich just a little bit better.

Rich is a software engineer who spent a long time thinking about how we actually present information in documentation, especially when it comes to tables, summaries, and making complex things easier to scan and understand. He's worked across the open source ecosystem for years, starting primarily in R and recently contributing heavily in Python as well. In fact, there's a good chance that you have heard of or even use some of the tools that he's worked on. Today, Rich will be sharing with us about great-docs. But instead of hearing about great-docs from me anymore, Rich, I would love you to fill in our audience with all of the details. So Rich, take it away.

I will do that. And what I got is actually a bunch of slides and some browser tabs because I figured it's good to show great-docs. Otherwise, it just becomes a theoretical thing. But also, it's kind of funny because I also wanted to just, like, entirely show great-docs and not have any slides. But I thought that was too weird. So we're going to have a mix of both. So I'm going to share my screen.

Introducing great-docs

Okay. So great-docs. It's a documentation framework in Python for Python packages. And we're going to begin with some background. Challenges with Python package documentation. Here's one. It's often the first thing people look at before they commit to actually using the software. So it might be a deciding factor as to whether they use your software.

So documentation, it has to be good to keep existing users engaged. Otherwise, people may turn to alternatives with better docs. More lately, an agent might read the docs first. So we probably should keep in mind that, you know, a doc site might be better suited for agents.

So I got some responses for people that do docs. Three cheese points. One is, like, design for a good initial impression. Organization, presence of reference and guides should be there. And then to keep the users engaged, we need good authoring tools and use those good authoring tools for the reference and guides. And finally, we need things suited for agents, like pages and markdown form, lms.txt, and maybe even documenting skills and having skills available.

So many tools, I'm talking about tools that make documentation for you or help you make documentation. They get you to a pretty decent look and feel, and then they may not go beyond. So let's take a look at a few docs pages. These are not all Python related. I just kind of surveyed, you know, some of the internet and looked for some examples of docs.

Examples of documentation problems

And here's one. It's actually a package I work on. And these docs were made by me. They were made by Cran. And we can learn a lot just by looking at what not to do with docs. If you look on the left-hand side, the contents list is not arranged in any sensible manner. It's alphabetical order, which is good if you know the name of the thing, but otherwise kind of bad. The logo is missing, if you look in the top left. That's not a great good initial impression. The topography is not that great. And the tables, they're hardly recognizable tables. They're not styled. There's no lines. But there are indeed tables in this.

Okay. That's some problems. This is from a Python package. And you may not need me to say it, but if you look from the left side, a lot of those objects are encroaching on the main content area, which is obviously not good. And also the code examples are oddly indented, almost centered. And it's got this problem of many, many methods on a single long page. And maybe that's the fault of the documentation generator. It doesn't give you the option to split those out in some sort of gripped manner. But that's where we are with this.

Here's another one. It's actually for a LaTeX package called long table. And it's a PDF. So you don't really get the benefits of documentation on web pages. You can't get there with a Google search, typically. The command reference, there actually is one on some page, is pretty terse. And it's all stuffed into a summary table. And because this is essentially a guide, there's no expectation of conventional sections. There's a command reference, because the person just deigned to create one.

Here's another one for a project called juice. And I was confused about this one, because I looked for a doc site. But essentially the docs are on the readme, the GitHub readme. So that's a problem. There's no generated API reference. So the issue there is that this could drift from the actual code. It's basically handwritten. And it only ever reflects main. So anyone on an older version can't read docs for that version. So that's what we have here.

So I showed a few egregious examples of maybe not-so-great documentation. But I want to say, a lot of the documentation now is really, really good. Look to the right. FastAPI, Polars, FastMTP. These are great. Like Seaborn, these are fantastic. Authoring features that they use to support teaching. There's a lot of polish there. But still, there's still challenges that remain. Like building docs for AI agent consumption. There's a little bit there. You can sort of see that in FastMTP. But it's kind of like only the beginnings of that.

So a while back, when I was doing this survey, I thought, maybe there's an opportunity here to do something. So enter great-docs. I figured I would throw my hat in the ring with a new Python documentation library. So some of the design goals I had early on is that with it, you can create a beautiful, complete site with almost no configuration. Like, getting close, but you need a little bit. But, you know, we try to gloss over that a bit. It'll have depth features and control when you need that stuff. So we have a lot of features. You don't need it right away, but you might use that later.

And a big thing was, it's got to be ready for human readers. Like, be really, really nice if you're just reading it. And also be useful for AI agents. Because a lot of times, they're just thrown onto some docs. And, you know, it's up to them to make sense of it.

And a big thing was, it's got to be ready for human readers. Like, be really, really nice if you're just reading it. And also be useful for AI agents. Because a lot of times, they're just thrown onto some docs. And, you know, it's up to them to make sense of it.

Inspiration from other documentation tools

Okay. So when I was looking at stuff, I looked for inspiration everywhere. I looked at things like pkgdown, MATLAB. A lot of good ideas, like, wherever you look. So I'll show you what I liked from each of these. pkgdown, that's an R thing. MATLAB, that's just the MATLAB docs. Hexdocs, that's for Elixir. And I think Erlang as well. And Vue.js, which is its own thing.

Okay. So pkgdown is kind of where it started for me. It made good docs to default for R. And I did a lot of R stuff. So basically, I use this a lot. A really cool idea that they had a long, long time ago was this right sidebar. And it just made a lot of sense to me. It just has metadata for the package. It has the authors, some useful pages like the guides, the contributing guide, citation, license. You know, good stuff. So I basically, I stole that. It's in great-docs now. That's like a little screenshot of great-docs. I grabbed that.

Okay. But, you know, pkgdown. Like I said, it's my home turf. I built tons of R packages like these ones here. And they all have a pkgdown site. And I'm pretty familiar with it. So a little bit biased, you might say. What else did I steal? Well, the homepage sidebar, like I showed you. The idea of a zero-config build. Almost zero-config. Conventional reconfiguration. Like if you have certain files in certain spots, the framework will just, like, look for them and then use that in making the docs and the build. Beautiful publishable defaults. And pkgdown's site looks wonderful the instant you make it.

And this wonderful package API index page, which is kind of cool. It kind of gently puts you into, like, the reference section. It gives you an index and some descriptions. And lo and behold, that's in great-docs now. Sort of like a comparison of the two things.

Okay. MATLAB. I gotta admit, I don't really like MATLAB. I used it before in the past. Not really a big user. But I love their docs because they really go so deep in their docs. It's all, like, super accentuated at the top. They tend to overexplain, which I think is kind of a good thing. Like, see, for every sort of, like, overload or use of, like, read table, they have a separate description. So it's like, well, they just went the extra mile there. They have tons of examples. Like, this page is really, really long. Like, they have so much in there. And I kind of appreciate that. And the parameters. Nobody really documents them like MATLAB does. They actually, like, if you look here, they give a name, of course, to each argument, which is wild to me, and pretty useful as well. They give lots of room for examples. They have them in disclosures so you can just fold them up because there's so many of them. And, yeah, again, I love the idea of a short description for each parameter.

Okay. And Elixir. They have, like, this very consistent set of documentation pages. And it just looks good because of the structure, I think. So especially the sidebar. I mean, what I took from this is, like, you know, navigation is very, very important. Like, if we can just simplify it and make it, you know, like, really, really good, I think that's, like, one big sort of, like, piece of documentation that's solved. And they also, like many other frameworks, they put narrative guides as, like, you know, top level with, like, you know, a reference section. So it's, like, super front and center. It's, like, right there on the left. And it's, like, there, the minute you land on this page.

Okay. Another one is the Vue.js docs. They're so easy to navigate, and they're really great at teaching new users especially. They also get navigation right, which is really nice. I like the fact that their sidebar is, like, very flat and easily scannable. They have good use of space. They don't make it too complex. There's no nesting. The header itself is not clickable. It's basically just, like, a group, essentially. The CTAs they put at the end of different sections, they're wonderful. They're basically, like, paths, like, learning paths you can take, and they do that quite a bit. And they have these in-page callouts that are used for tips that you can divert yourself to, like, a different learning path. Like, this one here, like, watch interactive video. I think that's great.

I looked at all these tools. I was really inspired, and some big concepts I used were, like, pkgdown's ease of use, MATLAB's insane depth, Elixir's structure, and Vue.js' clarity.

Great docs. We tried to incorporate all these things along with some well-considered defaults, and what I think we do a bit differently with this project is beautiful out of the box, tries to look good on mobile as it does on desktop widths, and document as many things as possible, especially for LLMs. Like, the API for, of course, CLI, if you have it, MCP servers, and also document agent skills, if you got those as well.

Okay, so I'm going to take you out of these slides, like I said I would, and show you the great docs site. Okay, first I'll just show you, like, the landing page, like, essentially the GitHub repo. It's basically here, under great docs, and there you can find a link to the actual great docs site. Of course, you can search for it, and you'll get there. I'll just show you in this tab.

This is, like, the great docs site. I just want to show you they can have light mode or dark mode. Kind of cool. And here, again, is the right sidebar with different links to go to different places, like you can view the package on PyPI, you have links inside GitHub to, like, report a bug. Cool things, like here, which is essentially the developers, we have, like, links you can send me an email if you want. Kind of nice that it's all here, and we have things like, of course, like affiliations, and also, like, the company as well, showing us, you know, like, who's involved in this whole enterprise.

Another cool thing is, if you have certain files, like, kicking around, great docs will look for that in the directory, or other little places that these files may be. For instance, right here, contributing guide, it looks for contributing.md, it basically publishes it in a separate page. Code of conduct, project roadmap, if you got it, roadmap.md, security.md, your license as well.

I just want to show you one of these. I'll show you license. It's kind of cool. It'll take you to this page right here. We've got this little thing here, and it'll actually tell you little details about licensing things for, you know, because it knows about the MIT license, and, of course, you know, it'll just tell you a little bit more about it. In case you're not, you know, you come up against, like, a license you don't really know, can you use it or not, you know, it basically dispels some uncertainty there.

I also want to show you, like, the user guide right here on this. You can go to this page in the top-level nav, and you get sent to the user guide, and it has a sidebar right here which floats, and you can go to any of these pages, and we have user guide stuff there. And the cool thing is we have things like page tags right here, and lots of authoring tools for things like, you know, bits of, like, code. It could be any type of code. You can specify that. This entire site uses Quarto. It basically builds on top of Quarto, so we get lots of Quarto affordances and authoring tools, and we basically build on top of that and enhance a few things as well.

Reference section and API documentation

So, let me show you a few more things on this site. Another thing is the function, like, basically the reference section, and here we have a reference item table preview inside of great docs, but you can actually go up here. When you start on this page, you start with the index, and you can dig deeper, and you can specify how to organize your left sidebar with different groupings. You can specify, of course, the different methods inside one page, or you can split them out to different pages like this.

And the cool thing about this is it gives your readers a lay of the land on this index page. You can have rich descriptions. You have these short summaries for individual methods or classes or what have you. It's kind of like a good sort of easing in instead of just, like, being hit with the very first thing.

One more thing I want to show you is that we document all sorts of stuff. We document, of course, a Python API. You have that available if you offer one. We will document that as well, and it's pretty good. We basically have a reader and writer for that, which is not that bad. And more lately, we've been doing MCP as well. If you have an MCP server in this project, in any project, we will look at that and write those API pages out.

One more thing I want to show you is the changelog. I find this pretty important. It keeps users up to date. Essentially, what we do here is we look at your releases on GitHub and essentially create a page, and every time you fire off a build, it just looks, again, for any updates to changes, like releases.

Documentation for the AI era

I'm thinking, just because of the age we're in, we now have to think a little bit more about documentation for the AI era because now your documentation is going to be read by agents, whether you like it or not. What we need to have are skills that agents can discover and load. We don't have to have that, but what we do is we offer a basic skill. If you want it, you can opt out of it. It's essentially just a very basic skill that we can write essentially just as a guide to your project.

But if you have custom author skills, like you've made the skills yourself, great docs will find that and actually publish that, and it's quite nice. If you have multiple things, you'll see there's actually some information here, and it'll show you all the components of each skill.

Okay, and I touched upon MCP reference. If you do have that, like an MCP server, it'll actually just go ahead, unless you opt out, and make it MCP reference pages for you.

And this is a thing that you can also opt out of, but it's nice to have. Essentially, it is LLM's text and LLM's full doc text full API. You don't have to write this yourself. great docs will just do this. LLM's doc text, if you don't know it, is not a very big file. It's essentially like a sitemap. It's just like a very simple text file that agents sometimes look for. And LLM's full is essentially the whole API inline, and it's quite a bit bigger, obviously, depending on the size of the package, but LLM's do sometimes look for that as well.

So, I mentioned, I touched upon agents as the first readers. They're increasingly reading your docs before any human does, which changes a lot. If an agent can't parse it, your users will feel it in some small way, maybe a big way. There'll be more turns, longer conversations, maybe some bad decisions. So, in a small way, maybe in a big way, docs are sort of like an API service now, because decisions are made on your docs sometimes. And you're writing for humans and machines alike in this scenario.

In a small way, maybe in a big way, docs are sort of like an API service now, because decisions are made on your docs sometimes. And you're writing for humans and machines alike in this scenario.

So, I came across this really interesting site, this project, whatdoagency.runtype.app. It's a tool for getting an agent or simulating LLM looking at a site. You just pop in a URL, and I did so here with a great docs page. And it shows you whether it can find the things that would help an agent. So, it's really small here, but I zoomed in here. So, it does find LLM's text, it does find LLM's foldout text, and it finds markdown pages, essentially a version of the page which is not .html but .md, so a markdown version of the page, which is useful. So, we can see from this that, yeah, we can also test different agents and see the same results. So, it's being used, and this sort of in a way validates that.

Features: Term Show, Lightbox, and Table tools

Moving aside from AI and agents, I have to confess I love features, and I stuffed a lot of them into great docs and tried to make each one good and useful and something that I would use. So, one of them I'm going to show you is I'll show a few of these live, and I'll just show you a few of these in slides. But a big one is Term Show, and I made that because I was really frustrated with showing or demonstrating CLIs. The tools that were available were kind of good, but it has major shortcomings, I thought. So, I built a thing which I'll show you in more detail to help teach users your CLI or your TUI application.

Another big thing was Lightbox, a full-screen Lightbox with zoom and gallery. Sometimes you have images, it could be diagrams, what have you, and you just want to show things and make it labeled and such. So, I built a thing for that. Another big thing was Table Preview and Table Explorer, which I'll get into. Essentially showing tables. It occurs to me that sometimes we have lots of data either in our package or you transform some data and you want to show the results. That's where this comes in useful.

So, I'm going to show you right now, just give you an inkling of what Term Show is. So, usually, like I said, CLI and TUI demos usually mean a GIF or a video. They can be huge, sometimes blurry because the text is not really well recorded or you're recording at a low bitrate. You often can't select the text and you often can't even pause the playback. So, you want to rewind a bit, but you have to wait until the next go around if it's a GIF. Very frustrating.

So, I built a thing where you just record your terminal activity, you get a file, and then there's also a paired YAML file. This is for a configuration and presentation, essentially, like the way you want to show it. And, as you imagine, it can get pretty crazy editing YAML file to make it just right. So, what I did was also I created an editor for this. It's a browser-based editor, and it allows you... It's almost like a little final cut for terminal presentations, and I'll show you that in the demo pretty soon.

This is great, but what users, like viewers, see is sharp SVG animation playback. It's got its own full-featured player. It's a lot like a video player. And chapters, annotations, you can have those in there, and you can teach with that. You have all the tools to demonstrate what your CLI does and show workflows.

Term Show editor demo

So, I'm going to show that to you right now. So, I've got a tab open right here called Term Show Editor, and here's a video I made a while back. And, essentially, you can scrub around and see almost like the live video of your recording. And you can just add chapter marks. I can just double-click here, add a chapter mark, and I can label it new chapter, for instance. And it's kind of cool. Now I can scrub around and see I'm in new chapter up here. Very nice. You can always just hit play to play back, which is kind of nice. You sort of see it live. This is like an emulation of the final thing.

You can add things like cuts. Say you don't want to see a bunch of the video, just double-click here, change the width of this, and you can cut out some of the extra dead space if you want. So, if I start playing back, it'll just jump over that, which is cool, because sometimes you just have a lot of, you just stutter over certain things, and your terminal, your demonstration is a bit overlong. You might just want to cut bits out.

We also have something called snippets. What that does is it shows you in the top right corner, right here, you can have a little button, and you can have text that's copyable. Say, for instance, the command. You can just have users click that, and they can grab that command and put it in their terminal if they need it. And it's shown at the right time and in the right place up here. So, really cool. And there's also other things I'm not going to show you, like highlights. You can sort of annotate pieces of the actual terminal characters and such. The boxes around things.

The thing I want to show you also is, this is nice, but what if you wanted to just see it, how it would look on a page? There's a preview button here, and what it does is it just goes back, of course, and does a render. And this is how it actually looks inside the player. And this is actually good, because I get to show you the player, which is kind of neat. You can pause it, scrub around, go to the chapter marker here, click here, and you see new chapter I just made. And this is essentially an unsaved state, so before you even commit it to save, you can just look at it right here and go to the end. You can change the player speed in case you like to go really fast. But this is kind of nice. Before you even commit to rendering it, you can make sure it's just right. So that, in a nutshell, is Term Show and the Term Show Editor.

Table Preview, Table Explorer, and Lightbox

Okay, so it doesn't end there in terms of features. There's lots of features. A big one I wanted, I'm really into tables. I have this project called Great Tables, so definitely tables is kind of my bag. But I think it's useful to have that in documentation sometimes, without importing Great Tables or some other table library. So I did something called Table Preview and Table Explorer. Two different things, but they're kind of cut from the same cloth.

Table Preview is kind of like a simple static table. It gives you a small preview of the data, usually like ten rows, or it could be the first, fifth, five rows, and the last five rows. And it shows you things like column types, and it's pretty smart about being wide, but not too wide, in terms of column width controls. And the really good thing about it also is that you can import from many different formats, like we're talking about DataFrames, CSV files, JSONL, Feather files, Parquet. It's really great in terms of importing.

Okay. If you need a table, it's a bit more interactive, full-fledged. Table Explorer might be what you need. So in this case, you have something that's JavaScript-enabled. It has all the things you might expect, like sortable columns. You can sort by multiple columns. It has buttons for downloading the filtered and sorted data, and a means to copy as CSV text. And the filtering interface, I'll show you how it looks when you actually have some filters in place. You just click the Add Filter button, and it presents to you a small interface, which depending on the column type, will present different options for filtering. And then they just appear as these tokens on the field right there, which you can remove, or you can add more. So it's really quite nice.

The lightbox thing is basically born from frustration of showing a few images, and I figure it's a nice thing to have in documentation. You never know when you'll need it until you need it. So what it does, basically, you click on an image, it brings you in this view, and

it just makes it large, essentially. And if you have a gallery, and you can have a gallery, you can just use the arrow keys to page through the gallery.

And you can have dark mode, image swap, so you might have for instance, images for light and for dark mode, like a pair. You can have before and after comparisons, which basically presents to you this slider. And another cool thing, which I'm not really showing here, is annotations. You can have certain regions of an image annotated, having dots and callouts.

It's really small on the top right, but this is zoomed in on the controls. And we offer things like, of course, zoom, the ability to copy image. You can download the image, and the really cool thing is getting a link to the image. So basically the link will just mean, if you provide that link to someone else and they follow it, it'll bring that person to exactly this lightbox view.

Concluding thoughts

Okay, wrapping up. So I want to conclude with a few points. Great docs. It tries to be nice by default, but powerful when you want more. I kind of believe there should be a tradeoff between an easy setup and having depth control. So great docs has sort of been designed to give you both.

Just a thought, I think we should treat documentation as a product, put a lot of weight on it. It's really what people and agents judge your project on before they really commit to using it. So it deserves some design depth and care.

Just a thought, I think we should treat documentation as a product, put a lot of weight on it. It's really what people and agents judge your project on before they really commit to using it. So it deserves some design depth and care.

Okay, this is a bit of a spicy statement, but I'm going to say it anyway. So your documentation is now kind of like an API for agents as much as people. So when an agent is sometimes their first reader, often, you might say, things like skills, mcp docs, lms.txt, they're kind of like table stakes now. They stop being extras and become required, you might say.

Great docs, they teach you. They don't just describe things. So if you have things like runnable examples, callouts, and guides, which are top level with your reference, you might even have terminal presentations, they're all there to actually teach your users.

And being good on mobile, it shouldn't be an afterthought. Docs should be just as good on a phone as they are on a desktop for human readers. This is why things like parameter tables, sometimes they're put in tables and you get a very long parameter on the left side and description that wraps over multiple lines on the right. Probably shouldn't do that on mobile. There's some thought there to make it read better and have better architecture of the information constrained with devices.

So I want to say thanks. Check out Great Docs. It's available in GitHub under positivegreatdocs. And if you do try it out, let me know how it goes. I'm really interested in people that do try it out, and I really want to help people if there's issues at all. Thank you.

Q&A discussion

Thank you so much, Rich, and thank you for the tour of Great Docs. Let's jump into our discussion. I'd love to open things up to our audience as well. So if you're in the audience listening in, then please go ahead. You can submit your questions in the Q&A widget, and we'll have them read aloud for Rich and our panel to discuss. In the meantime, while those questions are coming in, I'd also love to open this up to our host panelists here. So if you have any questions from which Rich has shared, please go ahead. And James, I see that you have your hand up, so if you want to kick things off, go ahead.

Let me first premise this with two pieces of, I think, context. The first piece of context is I have recently been teaching myself the Zig programming language, and their docs are basically the worst you have ever seen. They're out of date, they're all machine-generated, they're impossible to read through, yet I still keep using the language, and the docs don't really slow me down that much, and I just kind of get used to writing code.

The second piece of context is one thing that has always impressed me about the work of two of our colleagues, Tom and Dan, is that even for very new projects, I was always impressed by the volume of documentation that they would be able to put out for things like Kaputo version 0.1 that have docs. My question is, there's some space in between this really good docs and absolutely terrible docs, where the project is actually successful. How does one avoid what appears to me to be the vanity of writing docs, that's something you do for yourself, you make them look really beautiful, but avoid actually getting users, developing features? How do you find yourself not falling into that trap?

People talk about docs. Usually what I see or disparaging posts like these docs are just the worst, but occasionally you see some like, you know, validation, especially if really author, like, use Dr. Craig was like, especially if you're the author, you feel you feel good at, you know that's extra press per, you know, like if it's hard enough to get traction, you know in Python projects or any project. So I'll take a win wherever they can get it.

Absolutely. Now, Dan, I see you here, your hand up, but before we get to you and your question, there's a question from the audience and that is rich. Is there a preferred or supported style for documentation slash doc strings that works the best presumably with great docs?

Hmm. Well, actually it supports three of them, like NumPy doc, um, uh, Google doc, um, is that right? Google form and, um, RST. So you choose one actually during introspection, it actually determines through a simple sort of a heuristic, what you're actually using and it'll just like say it'll set that and then otherwise it kind of gets into like a quirks mode where it tries to like correct, uh, it does, it does a pretty good job. Um, if you have mixed documentation styles. So I say, just choose one that's good. And, uh, another thing that great docs does if this is great doc specific, obviously, uh, it'll actually run like examples. Like if you use sort of like this quarter markdown, um, sort of like notation, basic code fences and such, which is like an extra thing on top. I'm not sure how that reads without great docs in place is if you're just reading doc stream by itself. Uh, it's not worse than RST in terms of like just reading it. Um, I would say choose any style that's comfortable with it. I think the key thing, just write docs, like complete, uh, pick one style and like just go to town on the docs, I would say, yeah.

Fantastic. So it sounds like as long as you're using a common style, NumPy doc and Google docs are pretty standard. I think so. Yeah. Usually one, one, three. Yeah. Some sort of standard. Yeah. Yeah. Perfect. And then Dan, did you have a question on top of that?

Uh, you might be muted on zoom. Sorry. Sorry. I thought I had that. Um, I have a comment and a question. I really like term show. That's super interesting.

We've thanks. Um, we do a lot of internal tutorials at NSLS two for staff, and we've talked about generating videos out of those so people could revisit them. But the problem with video content is it gets stale. I like the idea of a terminal script that is version controlled. So an API or best practice changes and I can just tweak it. I've seen versions of this, I think called ASCII cinema or whatever. And I really liked the full featuredness of term show. It seems like it gets it to a level of like actual usefulness. That's very cool.

Um, the, the two questions. Did you think about synchronizing it with audio ever?

Yes, I did think about that. So I left some space at the bottom of the term show editor for extra sort of like timelines for new things. Yeah. Uh, audio definitely want the extra thing. Basically it'll just be like a little snippets that you can like pull in and, you know, arrange. And of course, you know, it's huge, but I think it's important. Uh, it's a, it's a teachable thing, um, to have audio besides like, just like, you know, text call outs. And I think it might even be in the roadmap, uh, which is, uh, you know, sort of listed in the front page of great docs. So yes, I have thought of it and I think it's a great idea.

And I think we, and I felt the same, um, I looked at the same sort of things like, uh, it was VHS and ASCII cinema. I've actually used VHS on a different site. It's okay, but it takes longer than you expect. It was just like little gifts or videos. Um, cause actually it just runs your terminal with a script and then creates the recording and it takes like a good long while. Uh, but it's got the problem of like playback. It's basically just like running these gifts, which are on repeat. Yeah. I know you've solved a lot of the things that made it like almost good. Yeah. Yeah. I mean, I'm glad those other things exist, but like, I thought, Oh, we're not quite there yet. Maybe this I'll have a stab at it. And yeah.

Okay. So then second part of my question, uh, which is a bit blasphemous, suppose that we're heavily bought into Sphinx or heavily bought into mist MD, but I wanted to embed term show in one of those two sites. Is that doable?

Not yet, but that's a plan as well. Right now. It's like a, it's only for great docs, which is a little unfortunate. Right. I don't know why I did that. I maybe it's just like convenience and such. Uh, but I'm, I'm thinking, um, I'm just going to have that be a separate thing or, cause essentially it's, it almost kind of is right. It just missing the exporting part essentially. And like the embedding part. Um, cause really you're just running great docs, um, my term show and then like create essentially. Yeah. And then the other little tools there. So the last missing pieces like is that, and yes, that'll be a thing pretty soon. Cool. That's my question. Thanks.

The great docs gauntlet

Rich. I had a question as I was looking through great docs. Um, I have not yet used it to myself for a project. We recently moved, uh, our open source project narwhals to use, uh, Zensicle, um, but I did see that, uh, a little mention about, uh, the great docs gauntlet, and I was wondering if you could tell us just a little bit more.

Yes. Yeah. So I take testing pretty seriously and, um, to like maniacal level. So the gauntlet is essentially, um, a lot of small great docs sites. Um, basically rendered, you know, sort of in serial and then like creating their own page with a hub. And, uh, I have a separate repo for that and it builds daily. And, uh, it's a way of sort of like seeing what fails, you know, catching regressions with all sorts of possibilities, cause it's really hard to test like a large framework like this, um, really well, you know, manually you have to something, you know, like basically like a test harness or framework and that's basically what it is. Yeah. Um, and I try to see what kind of, I try to build lots of, you know, like. Dimensions of coverage into that little tool. So anybody can just see this. I have now have a banner or like a little badge on the great docs, uh, repo itself that'll take you there. And, um, it's a great tool for other people to see, like, you know, what can this thing do? Is it, and also it's a good demonstration that this thing is, it's kind of tested. Uh, and, um, uh, but I think, I think that the key thing is like getting enough, like examples, uh, in there. I think, I mean, there's a little bit over 300 now. Um, that covers a lot, but every time there's new features, a few of these have to be added, of course. And, uh, I think it's really useful. It's also a really quick way to sort of see that the new feature added, uh, in a very sort of specific isolated build, uh, actually works like I can quickly iterate on, on the thing. Yeah. So it's just testing. That's very cool. Thank you. Uh, Isabella shared a link in the chat to it. Oh, nice. Nice. Tanya, did you have a question as well?

Um, yeah, I think that, um, this one did go through slightly during the talk. Um, but I was wondering, uh, if you could, um, elaborate a bit more about kind of like having now the commentation moving towards not only being human readable, but also machine readable. Um, and you know, what are different strategies to evaluate both? Um, especially I think something that we will be super interested in will be kind of like that is skill generation feature. Um, in like how agents can make use of it. It's like, how can we evaluate how effective they are beyond them? Maybe just being machine readable.

Yeah. Yeah. So the whole deal with skills being published there is kind of like, so like if you have an agent look at the site, they can discover it on their own. You may not even know that just there's a skill there or a set of skills. And there's actually some, you know, it's version it's essentially it could be versioned, um, but it might be just an extra thing that you have or your, your library itself. And, um, yeah, it's quite nice. So you don't have to like, it's basically self-advertising. It goes into a special directory or a hidden directory called, um, I think well-known and then like skills. And, uh, I've done some like basic evals of this and different agents do tend to find it if you suggest to look for them. Uh, sometimes you, you have to sort of lead it, but it's not quite, uh, great for discovery yet, but I'm trying to be a little bit ahead of the curve in terms of like, you know, having these out there. It's also kind of, it's also, it's actually kind of useful to have as well. Um, just to have transparency on what the package has.

So I'm, I'm of the mind, just publish everything on the doc site. Uh, that's that could possibly be published and it's either going to be useful for, for an agent or for somebody else or for yourself down the line.

Yeah.

Great. Yeah, you're welcome.

Uh, piggybacking off of that, Rich, I was curious. Um, I may, I may have missed this during your, your, uh, presentation earlier, but is that done just by a serving both the markdown and HTML file just separately, and then the AI will get pointed towards the markdown file.

Yeah. Um, you're talking about just like the standard pages, right? Having like a markdown version. Yeah. Essentially we have like a blue, like buttons there to switch to the mode, but essentially each HTML page is paired with an empty page. And so, um, an agent might look for an empty version of the page. It's starting to become more common. Uh, but if that's, if it doesn't do that, you can just point it towards it. Yourself in terms of just going to that, you know, hitting the button where you can copy the text and just fuse the agent. So you have a few avenues to get like a, you know, a page, a page's worth of content into an agent.

Um, so it might be self-discovery or it might just be like, you know, a little bit of feeding. Uh, but it's kind of like all there for all pages and you don't have to really worry about, um, uh, it being there for, because it's automatically there unless you opt out.

That's very nice. Now I noticed, I also noticed that in your examples of good documentation, you put, uh, Matplotlib's documentation. We have Tom Castle from Matplotlib. And so I was curious, um, what, what were some of those things that really make that good documentation stand out that you were striving for to make the defaults in great docs?

What makes great documentation

Yeah, I think a big thing, and this doesn't always get done is examples, like rich examples with pros, you know, explain the example like before and after, like a good setup for an example, like the, the code, the results, and then some additional commentary on that. I think that's like, uh, overlooked and has been for, for so long. Uh, but it's really useful, really very useful, especially when it's as close as possible to the thing and not just like a, you know, like a, like a degraded version of that, that, that, that is valuable to me.

And not only that, like having a few examples, I, I tend to go a little bit maximal on examples myself. You don't have to, but, um, I think it makes, cause a user may just land on that page, you know, like a API reference, API reference page, it shows them the thing, but then it shows a lot of examples and you can get your imagination going on what the thing that's above can actually do. Um, so I think it's, it's kind of central. And you can always stop reading, especially if the examples are at the bottom, you're just like, okay, that's it. That's enough. So it's kind of like, you know, extras and more, I think more people have found them useful than, than, you know, irritating. So, and that's what I think is good in terms of, uh, good documentation.

Organizing large documentation sets

If I can jump in playing devil's advocate bit, um, the problem we have in Matplotlib is that we, we say our docs are both too big and too small and that it does not cover everything, but what is there sufficiently large that it's, it essentially becomes unapproachable. So how do you go about organizing? Well, just considering just math, like something with a scale of methodology gallery, how would you organize that to make it actually discoverable for a user that's not like getting lucky from a search engine?

I tell him that just the gallery itself, just the gallery, just gallery. I think just sectioning is, you know, like essentially I know it can go and it's huge and there's many topics, uh, but there has to be some sort of way to section it such that there's some beginner level things and then you go a bit deeper in the middle and you end off going, you know, with more use cases. Type things. Um, there's gotta be, I think there's gotta be some ordering and sectioning, just make it palatable, those, those, those chunks, but sorry, uh, if you go by complexity, you may find the user needs to go very deep in a very particular functionality, but who's new to the library. And so if you go by functionality, they may spend a long time wandering around with a lot of examples they don't need.

Yeah. I think really just to go like down, down a complexity in this way. So how do you, how do you do recommendations on how to handle the inherent, like cross tapness? Yeah. It's, it's tough with really big APIs and I've, I've, I've got the problem as well. Uh, you can have some things like, um, what I found is, is good. It's having lots of cross-linking, like essentially like interlinks between different topics and have that be prominent, like, you know, if related. Or you can have page tags as well, which I incorporate in great docs. So basically you can go to like a separate page tags index. And then so long as the page tags are sufficiently, you know, like, uh, divided up and specific enough and has enough pages underneath them, you can probably isolate the pages pretty well.

Uh, I think the thing of last resort might be search, uh, which is this keyword search that you actually end up in a pretty good state there, but I think tags like page tags might be, might be the way forward. It's aside from interlinks and just good, good organization. I think if you have all those things, um, even at very large APIs and possibilities and how to use the API, um, you can get further, it's still gonna take more searching, but that's just the nature of the beast, I think, when you have very large, complex software.

Um, and James, I see you had a question, but, uh, before I give that to you, James, Rich, can you explain what you mean by the difference between cross-linking and page tags there? Yeah. Cross-linking would just be essentially interlinks, essentially like little links embedded in the text, like the pros itself, uh, typically it's, it might be in the middle of the page. It might be close to like, you might have them standardized as like, you know, see, you know, see more or, you know, like see also, uh, on page, or you might have tags, which are typically the top of a page and they just like, um, they're just like little things you can click and they take you to like, an index with similar pages, essentially like a, like a sitemap or an index. Um, and that might be good because you might want to go back and say like, oh yeah, I want to see all the pages that have this tag, because this, this, this is what I'm looking at right now. So that's another useful way of organizing things.

Excellent. Thanks. Yeah. And then James, you had a follow-up?

Linear vs. nonlinear documentation consumption

Yeah, I have a follow-up to Tom's question, because I know Tom, I have joked with you more than once that I simply don't ever read documentation, but that's not strictly true. In fact, I end up reading an enormous amount of the units of documentation. I never, unlike I suspect you and your colleagues sit down at the beach with a manual or a man page in hand and read it linearly. I consume these things in an extraordinarily nonlinear fashion, which is why for me as a user, my entry point into the Matplotlib documentation is either one, I'm going from Google or these days I'm asking LLM to find the articles for me and then going through that.

From that perspective, the organization of things like the gallery is honestly completely irrelevant to me because I'm never going to use that organization. I'm going to reconstitute it myself. What also gives me a little bit of a pause is when we're talking about the, the, the LLM versus human use cases here, it seems that we're drawing a very unimaginative contrast between two very strict modalities. A human being is reading these documentation that a human being wrote in the way that we assume that people read these, like printing it out and reading it on the beach, or the LLM is doing it automatically. And I wonder if there's not some middle ground by which the LLM can serve as an automation tool to create a customized nonlinear pathway through the documentation so that the organizational structure is no longer built into the way in which the documentation has been written, but it's constructed by the LLM as part of some kind of learning process.

And I wonder if there's not some middle ground by which the LLM can serve as an automation tool to create a customized nonlinear pathway through the documentation so that the organizational structure is no longer built into the way in which the documentation has been written, but it's constructed by the LLM as part of some kind of learning process.

Okay. Can I respond or you go first? I think one thing about the, that's a fair point about the, you know, search engine or, or the robot to giving you links deep into the docs, but what you lose there is the, um, I don't know how to do anything with the tools. So that means you, you, it to first for search to have worked, which means you typed words at a text prompt, you've actually described that plot with the same words we use to describe that plot. Whereas if you like go to our gallery and just like hit your scroll wheel and let it run, you, you can very quickly see like, Oh, that's the one I want to go back to it.

And the, that serendipity of, I don't know, I don't, I know what I want, but I don't know the name of it. I don't know the words for it, but like, I know it when I see it is something that I think is, is important not to lose, particularly for plotting, plotting tools. Right. I'm very, I'm very biased towards visual things here. I never stumbled across something I didn't know even existed and then learn something I didn't even know I wanted.

Um, and second for the, yeah, and I don't really have a comment on the second one of using the robot to basically build you a custom documentation.

I can probably speak on both of these. Uh, so on earlier packages I made, um, I knew people would land on different pages, like of the, you know, reference API. So my solution there was to, for each of those pages, no matter how small the little thing was, I would have examples there too, because they show you the end product and you can work your way backwards. If you like the end products and this page is the page for you and you land on the right page and you have some working code there as well. Uh, so I resolved to do that. It was very time consuming to write all these examples. Um, but it was worth it because you never know where people land and they may just bounce and give up if they don't find anything right away. So you have to anticipate people will land in the weirdest spots. Uh, we should, you should show something useful, I think, in every spot they land.

And in terms of an LLM, um, you know, helping it through, um, I mean, these days it does a pretty good job, like finding where it has to go. Um, I would have said like, you know, months ago, uh, you know, give it more of a hand, but I think these days, at least when I used it or when I use agents, it does find things pretty fast and it, it does bounce around a little bit, but it does eventually land on land on the right things. And depending on the documentation may have to go into like the source code, but, um, usually the documentation does enough and it's truthful and current. Uh, it's even better. It doesn't have to stumble over the documentation. Um, so I don't know about the second one. Maybe, maybe we don't have to think about that soon. Maybe LLMs, just agents just get better and it's not an issue.

Go ahead, James. Uh, on, on Tom's point that there is some necessary organizational structure to these documentation. I actually agree quite a bit. Cause if you think about the non-linearity, it's not a random order in which I consume the units of documentation. What it is, is it's an order that is subordinated under desiring to build like a world model for how this works. So for Matplotlib, I'll try and figure out what are the boundaries of the core concepts, and I'll look for pieces that then allow me to reconstruct that. Cause usually for tools like pandas and Matplotlib, the ideas of the library are very rarely explicated. Like the pandas documentation doesn't really highlight index alignment. The Matplotlib documentation probably talks about artists and things like that, but not in a way that surfaces it in any, you know, the assumption is nobody actually cares, but in fact, this is what I care about.

As a consequence, when we look at the structure of the docs, the docs are usually structured in a very kind of reference based authoritative. Either, you know, this is, this is easy. This is medium. This is hard. These are aligned in this fashion, but in truth, the non-linearity arises because what I want to do is find disparate examples that then allow me to formulate hypotheses about the underlying model, test those hypotheses and then build that such that I no longer am reliant on the documentation. So that if you have methods A, B and C, I learned method A, I learned method C and I never read the documentation for method B because I can guess how method B works from the model I've created.

From that perspective, I suspect that there might be an organizational structure to these. I don't know that anybody would actually use this or want this. That is very, very different than the traditional organizational structures. And I suspect that it may not be as idiosyncratic as it sounds, but it's still going to be much, much different than what we look like. Still very personalized. And I think that there, I don't know whether you can get an LLM to generate this in any usable fashion, but I think that this would, this is effectively what I'm, what I'm doing, but painstakingly over a million Google searches or a million LLM queries.

I wonder if like a, like a consistent use of C also, we provide, you know, some, a number of like links to like other pages is like the way forward because like, essentially then you have, you know, networking between different parts of the API to each other. And you don't have to limit it to like three things. You can have as many as you want. It's usually the below the example. So it's even further down the page, but you know, less useful for a person, maybe useful for a person, but definitely very useful for an agent that looks for like linkages between things. And if you're just like really rigorous in doing so, and even have an LLM, like create that for you. Like, I mean, I find that LLMs are really good at making documentation these days. And it would be pretty good at just scanning your API and constructing that essentially doing the optimal C also, you know, like chunks that you need in each page.

I don't know that that might do it.

LLM-generated documentation

Excellent. What do you think, Dan? I guess I'll jump in with a quick, I guess we're just talking. So it's a little bit of a subject shift, but you put a thought in my mind. If the, if agents are really good at writing documentation, where do you need to rein them in? Cause I'm, I'm, I've seen it done. We have some LLM written documentation, but I'm more nervous about that than code somehow. And it reminds me of an Onion article headline. I saw AI summary of email longer than original email. Yeah. Yeah. That's, that's, that's a bad case. Um, but if I can get, you can have a skill just for like documentation writing, have your own style, example pages, um, level of rigor, like how you want to support examples, you know, like text before text after, um, how you section

things, like how, how much you nest sections with each other. So your advice is like basically make a style guide or point it at your own writing. Yeah. And, and have examples too, that it can hang by. I think they have that. And like the voice, things like that. Um, yeah, you could do pretty well with just that one skill.

I, I shared Dan's concern about the robot, turning the robot loose on documentation that, um, with code, at least you, they can run the tests and compile and there's, it's very structured. It will, it will happily produce plausible sounding English that uses some of the same words as your project does that are utterly uncorrelated with what it actually does.

I mean, I, I've, I've been trying to, to, um, uh, sorry, I'm freezing. Cause I'm not sure it's an internal project. I'm not sure if I can talk about it, but I was trying to use it to automate writing doc strings for something, you know, many things we had that organically, we were still here to the doc strings and it made some terrible, terrible, terrible errors. Like there's something called lockdown and it picked a plausible version of lockdown. That was not what that was doing.

So, yeah, I've had bad experiences in the past and I've also had bad experiences with, with just tests, but that was a long time ago. And now the test things, uh, the test case has been solved, I would say. And, um, well, to grace, I mean, mostly, I mean, they, they work essentially. They don't like fail immediately, but I say like docs, uh, in terms of agent LLM written docs, they're, they're pretty good. Again, you have to read them. And, um, but if you have a good skill essentially, and you keep refining that skill and keep, you know, I say, keep refining the skill and keep seeing what it turns out, uh, it might be good for like, um, you know, a project that you wish had docs and you don't want to act to have so much activation energy to get to do it yourself. Uh, I would, I would give her, give it a whirl. It's not that bad.

Closing discussion on LLMs and documentation

Awesome. With things in place that I'm going to preface with that. Yeah. And then James, some final thoughts to help close this out here. As a feedback for Dan, might not our friend dietaxis be helpful here. It seems like the LLM for purely reference style documentation is somewhat innocuous in its use. You can check very quickly and see if it's that, but anything that starts to involve human judgment, for example, I was telling some of your DSSI colleagues in my adventurous learning zig. Um, I asked the LLM a bunch of questions and it tells me this is more elegant or this is a better way to do things. I skipped past that.

Anything that comes out of the LLM that is judgment based, even if the judgment is 100% correct, I don't want to read it. I don't want to hear it. You are not a human being with stake in the game. I simply do not care what you have to say. I, the accuracy doesn't matter. Even if it's a hundred percent correct, it doesn't matter. I do not want to hear what a robot has to say. And as a consequence, I think that from the documentation perspective, the very opinionated documentation, that's where I would say, keep the LLM away from it. I really do not want to read that very, very violently. Do not want to read that.

I simply do not care what you have to say. I, the accuracy doesn't matter. Even if it's a hundred percent correct, it doesn't matter. I do not want to hear what a robot has to say.

Right. It's the, I don't want to ask people to read something that I couldn't be bothered to write. I think I'm still there, but I agree. I was saying just for things like, like doc strings, like, uh, no, that's more than fair. Sure. That's, that's fine. But other things which are, you know, like the architecture and like how to use you know, like best practices. Definitely. I would do that. Yeah. By oneself. Yeah.

Fantastic. Well, that takes us to the end of the time that we have here for today for this month's Python exchange. I want to thank all of our host panelists for joining and providing their thoughtful and thought-provoking discussion points. And of course, rich for joining and telling us all about great docs. And finally, a huge thank you to everyone who has joined us today in our audience. Whether you ask questions or are you listening in just to learn about this project next month, we're going to be joined by Dan Yaw from Anaconda who works on the Conda CLI and the broader packaging ecosystem. So if you're interested in learning a little bit more about packaging, then make sure that you check out our Python exchange next month, keep an eye out on our website for more updates and the specific date and time that's meetup.doepy.org. Thank you all for coming. And I hope to see you all next time. Have a great rest of your day.