Sadly Not, Havoc Dinosaur

Auto-Generate Code Comments

In which "AI" suggests comments and explains a function that approximates pi (π)
The digites of Pi arranged in the shape of the Greek Letter π
ascii_pi by Jorel Pi

Headshot of the author, Colarusso. David Colaursso
Co-director, Suffolk's Legal Innovation & Tech Lab

This is the 42nd post in my series 50 Days of LIT Prompts.

Last Thursday on Pi Day, I found myself writing about a program I wrote to approximate pi, as one does, and it occurred to me that I hadn't talked at all about large language models (LLMs) and coding. I haven't written about LLMs and coding here for two main reasons: (1) I suspect as a law school instructor that my readers are more interested in the human language side of things; and (2) it's a pretty crowded space. That's not to say I don't have opinions. Over the past year, I've leaned heavily on LLMs to help get a number of projects off the ground. The LIT Prompts extension itself was written with a good deal of LLM assistance. I also used it to build an open source ago-powered RSS reader—My RSS Algo.

I started using LLMs to help with my coding because an old student project, the Massachusetts Resource Directory, needed a lot of work to stay relevant, and I didn't know when I'd have time to get another student to do the update. So, I budgeted a few hours to see what I could get done with the help of an LLM, and I ended up rebuilding the site from scrtach in less than a day. The experience drove home how useful these tools could be.

I suspect code generation with LLMs is currently stuck in this wierd place. If you can evaluate the output, and know well enough when to ignore what it suggests, they're really quite a time saver. They're especially helpful when navigating unfamiliar syntax, but I fear if you had zero coding experience they might end up being more trouble than they're worth. Similarly, one could be overly credulous and implement everything they suggest absent any proper oversight. Which is sort of a general comment about LLMs right now. What could go wrong? So, as I have with my human language prompts, I'll focus on a use case that strikes me as relatively safe—providing code comments.

Here's the code I mentioned above with zero context or explanations. Don't worry is this doesn't mean anything to you. Feel free to jump to the next bit of text where we'll feed this program into a template and ask it to add comments to the code to help explain what is going on. 

function estimate_pi(points) {
  let start_time = Date.now();
  let inside = 0;
  for (let i = 0; i < points; i++) {
    let x = Math.random()*2-1;
    let y = Math.random()*2-1;
    if (x*x + y*y <= 1) {
      inside++;
    }
  }
  let end_time = Date.now();
  console.log("Elapsed time: "+(end_time-start_time)/1000+" sec");
  return 4 * (inside/points);
}
pi = estimate_pi(1000000);
console.log("π ≈ "+pi);

Below is a transcript of what happened when I highlighted the above code and triggered today's template.

For the most part I'm pretty happy with the output. The leading javascript text seems to be an artifact of how GPT4 deals with code comments. That is, it seems to preface code with a declaration of the code's language. Consequently, you shouldn't include this text in your actual JavaScript as it would throw an error. The comments following //, however, are reasonably good. They do an okay job of describing what the code is doing but stop short of explaining why I might have written the code to take those steps. So, I asked the LLM "why?" and the answer was reasonably cogent. I didn't like the use of (inside/points) in the fifth paragraph. I would have preferred to see (points inside the circle)/(all point), but then I realized that (inside/points) was a direct quote from my code. So, I suppose I should have done a better job of naming my variables. In this way, I'm reminded of our template that anticipates reader questions. That is, I can use this to help make what I've written more accessible and understandable to others by using its output to challenge my assumptions. You may also recognize this as the theme of yesterday's post.

As always, the output isn't perfect, but that doesn't mean it isn't useful. If you do a lot of work in browser-based coding environments like Jupyter Notebooks or Google Colab, having this feature in the browser could be pretty handy. That being said...

Let's build something!

We'll do our building in the LIT Prompts extension. If you aren't familiar with the LIT Prompts extension, don't worry. We'll walk you through setting things up before we start building. If you have used the LIT Prompts extension before, skip to The Prompt Pattern (Template).

Up Next

Questions or comments? I'm on Mastodon @Colarusso@mastodon.social

Setup LIT Prompts

7 min intro video

LIT Prompts is a browser extension built at Suffolk University Law School's Legal Innovation and Technology Lab to help folks explore the use of Large Language Models (LLMs) and prompt engineering. LLMs are sentence completion machines, and prompts are the text upon which they build. Feed an LLM a prompt, and it will return a plausible-sounding follow-up (e.g., "Four score and seven..." might return "years ago our fathers brought forth..."). LIT Prompts lets users create and save prompt templates based on data from an active browser window (e.g., selected text or the whole text of a webpage) along with text from a user. Below we'll walk through a specific example.

To get started, follow the first four minutes of the intro video or the steps outlined below. Note: The video only shows Firefox, but once you've installed the extension, the steps are the same.

Install the extension

Follow the links for your browser.

  • Firefox: (1) visit the extension's add-ons page; (2) click "Add to Firefox;" and (3) grant permissions.
  • Chrome: (1) visit the extension's web store page; (2) click "Add to Chrome;" and (3) review permissions / "Add extension."

If you don't have Firefox, you can download it here. Would you rather use Chrome? Download it here.

Point it at an API

Here we'll walk through how to use an LLM provided by OpenAI, but you don't have to use their offering. If you're interested in alternatives, you can find them here. You can even run your LLM locally, avoiding the need to share your prompts with a third-party. If you need an OpenAI account, you can create one here. Note: when you create a new OpenAI account you are given a limited amount of free API credits. If you created an account some time ago, however, these may have expired. If your credits have expired, you will need to enter a billing method before you can use the API. You can check the state of any credits here.

Login to OpenAI, and navigate to the API documentation.

Screenshot of the OpenAI API Keys page showing where to click to create a new key.

Once you are looking at the API docs, follow the steps outlined in the image above. That is:

  1. Select "API keys" from the left menu
  2. Click "+ Create new secret key"

On LIT Prompt's Templates & Settings screen, set your API Base to https://api.openai.com/v1/chat/completions and your API Key equal to the value you got above after clicking "+ Create new secret key". You get there by clicking the Templates & Settings button in the extension's popup:

  1. open the extension
  2. click on Templates & Settings
  3. enter the API Base and Key (under the section OpenAI-Compatible API Integration)
Screenshot of the LIT Prompts extension popup.

Once those two bits of information (the API Base and Key) are in place, you're good to go. Now you can edit, create, and run prompt templates. Just open the LIT Prompts extension, and click one of the options. I suggest, however, that you read through the Templates and Settings screen to get oriented. You might even try out a few of the preloaded prompt templates. This will let you jump right in and get your hands dirty in the next section.

Screenshot of the LIT Prompts extension popup.

If you receive an error when trying to run a template after entering your Base and Key, and you are using OpenAI, make sure to check the state of any credits here. If you don't have any credits, you will need a billing method on file.

If you found this hard to follow, consider following along with the first four minutes of the video above. It covers the same content. It focuses on Firefox, but once you've installed the extension, the steps are the same.

The Prompt Pattern (Template)

A slide showing the George Box quote: All models are wrong, but some models are useful.
Maps are models; they don't show everything. That's okay as long as you don't confuse the map for the territory.

When crafting a LIT Prompts template, we use a mix of plain language and variable placeholders. Specifically, you can use double curly brackets to encase predefined variables. If the text between the brackets matches one of our predefined variable names, that section of text will be replaced with the variable's value. Today we'll be using {{highlighted}}. See the extension's documentation.

The {{highlighted}} variable contains any text you have highlighted/selected in the active browser tab when you open the extension. To use today's template, select the code you want to comment and trigger the template.

Here's the template's title.

Add code comments

Here's the template's text.

You are reviewing code and providing comments to help future developers understand its operation. In a moment I will give you a bit of code to read over. Your job is to add comments to the code and to return this new version to me. These comments should help explain what the code is doing. If the code includes a function there should be a description of the function's purpose and operation. Alright, here's the code: 

```{{highlighted}}```

Now echo back a version of the code with your comments, place your code in triple tick marks (i.e., ```code```).

And here are the template's parameters:

Working with the above template

To work with the above template, you could copy it and its parameters into LIT Prompts one by one, or you could download a single prompts file and upload it from the extension's Templates & Settings screen. This will replace your existing prompts.

Screenshot of the LIT Prompts Templates and Settings page showing where to upload prompts files.

You can download a prompts file (the above template and its parameters) suitable for upload by clicking this button:

Download prompts file

Kick the Tires

It's one thing to read about something and another to put what you've learned into practice. Let's see how this template performs.

TL;DR References

ICYMI, here are blubs for a selection of works I linked to in this post. If you didn't click through above, you might want to give them a look now.

Mastodon | GitHub | Privacy | RSS Site by David Colarusso