What if Members of Talking Professions Could Log Time in Simulators Like Pilots?
David Colaursso
Co-director, Suffolk's Legal Innovation & Tech Lab
This is the 11th post in my series 50 Days of LIT Prompts.
Run a simulation based on my experience as a public defender here. I prepaid for some "AI" time. So, chat while the getting is good. Otherwise, let's take a moment to talk about how we got here. After that, I'll show you how to create your own simulation.
As an experiential educator, I firmly believe experience is an excellent teacher. The question I have to answer is how best to introduce students to that experience. Some of my favorite experiences as a law student centered around simulations, practicing direct and cross-examination with actors, negotiating on the behalf of non-existent clients. When it's impractical, unethical, or dangerous to learn by doing in the real world, we fall back on simulations. When robust simulations are out of reach, we focus on practicing smaller parts of the experience. Sometimes, breaking things down into smaller tasks like this is by choice, it's a way to hone individual skills, but sometimes there really is no substitute for putting it all together. My favorite simulations were those where a professional or student actor was hired to play a role, but as you can imagine such simulations are hard to pull off. Mostly, they exist as capstone projects (e.g., a moot court exercise where you try a case at the end of the semester).
Large Language Models (LLMs) offer us the ability to create simple easy-to-implement simulated conversations. I really like simulations as a use case for LLMs because it doesn't matter if they hallucinate, and to the extent they regurgitate societal biases they present openings to unpack such biases and how they will affect our practice.
Although we're calling this a simple simulation, it is the most complicated prompt in our series to date. So, there's more meat to the Prompt Pattern section than usual. FWIW, the question of how to craft complex simulations will be addressed in future posts, and even if you haven't read our earlier posts, you should be able to follow along here. So, let's build something!
We'll do our building in the LIT Prompts extension. If you aren't familiar with the LIT Prompts extension, don't worry. We'll walk you through setting things up before we start building. If you have used the LIT Prompts extension before, skip to The Prompt Pattern (Template).
Up Next
Questions or comments? I'm on Mastodon @Colarusso@mastodon.social
Setup LIT Prompts
LIT Prompts is a browser extension built at Suffolk University Law School's Legal Innovation and Technology Lab to help folks explore the use of Large Language Models (LLMs) and prompt engineering. LLMs are sentence completion machines, and prompts are the text upon which they build. Feed an LLM a prompt, and it will return a plausible-sounding follow-up (e.g., "Four score and seven..." might return "years ago our fathers brought forth..."). LIT Prompts lets users create and save prompt templates based on data from an active browser window (e.g., selected text or the whole text of a webpage) along with text from a user. Below we'll walk through a specific example.
To get started, follow the first four minutes of the intro video or the steps outlined below. Note: The video only shows Firefox, but once you've installed the extension, the steps are the same.
Install the extension
Follow the links for your browser.
- Firefox: (1) visit the extension's add-ons page; (2) click "Add to Firefox;" and (3) grant permissions.
- Chrome: (1) visit the extension's web store page; (2) click "Add to Chrome;" and (3) review permissions / "Add extension."
If you don't have Firefox, you can download it here. Would you rather use Chrome? Download it here.
Point it at an API
Here we'll walk through how to use an LLM provided by OpenAI, but you don't have to use their offering. If you're interested in alternatives, you can find them here. You can even run your LLM locally, avoiding the need to share your prompts with a third-party. If you need an OpenAI account, you can create one here. Note: when you create a new OpenAI account you are given a limited amount of free API credits. If you created an account some time ago, however, these may have expired. If your credits have expired, you will need to enter a billing method before you can use the API. You can check the state of any credits here.
Login to OpenAI, and navigate to the API documentation.
Once you are looking at the API docs, follow the steps outlined in the image above. That is:
- Select "API keys" from the left menu
- Click "+ Create new secret key"
On LIT Prompt's Templates & Settings screen, set your API Base to https://api.openai.com/v1/chat/completions and your API Key equal to the value you got above after clicking "+ Create new secret key". You get there by clicking the Templates & Settings button in the extension's popup:
- open the extension
- click on Templates & Settings
- enter the API Base and Key (under the section OpenAI-Compatible API Integration)
Once those two bits of information (the API Base and Key) are in place, you're good to go. Now you can edit, create, and run prompt templates. Just open the LIT Prompts extension, and click one of the options. I suggest, however, that you read through the Templates and Settings screen to get oriented. You might even try out a few of the preloaded prompt templates. This will let you jump right in and get your hands dirty in the next section.
If you receive an error when trying to run a template after entering your Base and Key, and you are using OpenAI, make sure to check the state of any credits here. If you don't have any credits, you will need a billing method on file.
If you found this hard to follow, consider following along with the first four minutes of the video above. It covers the same content. It focuses on Firefox, but once you've installed the extension, the steps are the same.
The Prompt Pattern (Template)
LLMs are sentence completion engines. Their replies aren't based on any knowledge other than what words are likely to follow given their training data. So, we work to constrain those replies by packing a lot of information into our prompts. The more context we provide, the more we limit the set of likely next words. Sometimes we've worked to make sure the LLM's reply is almost entirely grounded in the prompt (e.g., our prompts aimed at summarizing texts and extracting data), but here we need a little more wiggle room. Luckily, we don't have to worry too much about the LLM going off script because we're involved in an improv exercise. If it "makes things up," that's not a problem. We're happy to roll with it. Yes, and...
To allow for a dialog we'll be setting the Post-run Behavior to "Chat" and providing a robust set of instructions up front to set the stage. In the template below you'll see that I've made a instruction sandwich. I start by explaining the role the LLM will play and what it will be expected to do based on some context (line 1). I then provide that context and repeat the expectations along with specific instructions (line 19). I structure the context into three sections: (1) BACKGROUND; (2) DIRECTION; and (3) THE CONVERSATION SO FAR. I frame the entire endeavor as an exercise in acting, telling the LLM that it is an actor playing a role and couching my instructions in the language of a director. Keep in mind, the LLM doesn't understand these instructions in the way a person would, rather it responds to the framing in a manner consistent with its training data. Presumably, it has seen a lot of scripts and acting notes. Our language is a way to leverage those aspects of its training. That is, it's probable that text following a framing like this will resemble what we're looking for.
When crafting a template, LIT Prompts makes use of a mix of natural language and placeholders. You can use double curly brackets to encase predefined variables and "questions" for your users. If the text between the brackets matches the name of a predefined variables, that section of text will be replaced with its content. If the text within brackets is not the name of a predefined variable, like {{What is your name?}}, it will trigger a prompt for your user that echo's the placeholder (e.g., a text bubble containing, "What is your name?"). After the user answers, their reply will replace this placeholder. A list of predefined variables can be found in the extension's documentation. We'll use this below to capture the user's first interaction with our "AI actor." By incorporating this user reply into our initial prompt we're able to control the "actors" first line ("Are you my attorney?") and make sure that it falls within our instruction sandwich.
I invite you to swap in your own motivations and scenarios. In principle you should be able to create any number of simulated characters. Eventually, however, you'll hit a limit to what you can simulate. If you have designs on creating a simulated set of events, not just a conversation, I'm afraid you'll have to wait for some of our more advanced prompt patterns. ;)
Here's a simple template based on my experience as a public defender. It's important to note that the attorney's goal during this interaction is rather constrained. They don't want to get into the facts of the case. Rather, they want to make sure they get the information they need to make a good bail argument. Roughly speaking this means being able to answer the question, "Why should the judge believe this person will show up for their next court date?" Consequently, they really want to learn about things like community ties.
Here's the template's title.
Start Simulation
Remember, this title appears on the button in the extension and any exports. I've chosen "Start Simulation" because I'm going to export this to its own webpage/webapp.
You are an actor playing the role of someone charged with committing a crime. In this scene you are interacting with your court-appointed attorney. You are currently in custody at the Rockham District Court, located in Massachusetts where you live. In a moment you will meet your attorney for the first time. Your job is to stay in character and act out your part. You are aiming for a realistic performance. To help you get into character, here is some background information.
BACKGROUND
Your name is John Doe. You're an 18 year-old Caucasian male. You're 5 foot 9 inches tall and weigh 197 pounds. You only recently graduated from high school and have been working at your father's construction company. You live with your father James Doe and your mother Jill Doe. You're in a steady and supportive relationship with Sally Sparrow, a classmate of yours from Our Lady of Hope, the local Catholic School where you both graduated last year. Sally's attending the local community college. You and Sally have been dating since you were both 15. You regularly attend church with your family on Sundays and help out with local community service projects.
Last night you were arrested and charged with disorderly conduct. Recently, you've been drinking a lot after work, sometimes to the point of black out, and you really don't remember what happened last night. You know you went down to your local bar Joe Sent Me, by yourself, but from what you can gather, you were asked to leave. You've never been arrested before. You're scared but don't want to show it. Mostly, you want to know when you can get out of custody. You didn't sleep very well, and you have a wicked hangover.
You spent the night in jail, and this morning the police brought you over to court. You're currently in lock up with a few other men. The guard has called your name and you're now huddled by what looks like a large mail slot on to cell's door, and someone the guard has identified as your attorney is on the other side.
DIRECTION
Be sure to keep your responses short. You "speak in sentences not paragraphs." Short and conversational, no speechifying!
THE CONVERSATION SO FAR
You are jumping into the scene in progress. You already greeted the you're attorney by asking "Are you my attorney?" They responded with "{{Are you my attorney?}}"
Think about how your character would respond and craft an appropriate reply. Don't repeat your greeting. Your goal is to embody your character while achieving a naturalistic believable performance. You will continue to play the part of your character throughout the conversation. Whatever happens, do NOT break character! Respond only with dialog, and include only the text of your reply (e.g., do NOT preface the text with the name of the speaker).And here are the template's parameters:
- Output Type:
LLM. This choice means that we'll "run" the template through an LLM (i.e., this will ping an LLM and return a result). Alternatively, we could have chosen "Prompt," in which case the extension would return the text of the completed template. - Model:
gpt-3.5-turbo-16k. This input specifies what model we should use when running the prompt. Available models differ based on your API provider. See e.g., OpenAI's list of models.gpt-3.5-turbo-16kwe have updated this page to usegpt-4o-miniwhich is closer in performance to 4. - Temperature:
0. Temperature runs from 0 to 1 and specifies how "random" the answer should be. I went with the reasonably "creative" 0.7. - Max Tokens:
250. This number specifies how long the reply can be. Tokens are chunks of text the model uses to do its thing. They don't quite match up with words but are close. 1 token is something like 3/4 of a word. Smaller token limits run faster. - JSON:
No. This asks the model to output its answer in something called JSON. We don't need to worry about that here, hence the selection of "No." - Output To:
Screen Only. We can output the first reply from the LLM to a number of places, the screen, the clipboard... Here, we're content just to have it go to the screen. - Post-run Behavior:
CHAT. Like the choice of output, we can decide what to do after a template runs. Here we want not only a summary of our text but the potential for a chat. So, "CHAT" it is. - Hide Button:
unchecked. This determines if a button is displayed for this template in the extension's popup window.
Working with the above templates
To work with the above templates, you could copy it and its parameters into LIT Prompts one by one, or you could download a single prompts file and upload it from the extension's Templates & Settings screen. This will replace your existing prompts.
You can download a prompts file (the above template and its parameters) suitable for upload by clicking this button:
Kick the Tires
It's one thing to read about something and another to put what you've learned into practice. Let's see how this template performs.
- Staying On Script. Provide your "actor" with a rough script of how the conversation should unfold (e.g., add a new section explaining that first they should start with some goal and later shift to another after some change in the conversation). Crafting a prompt that will produce fidelity to such a plan can be hard. You'll likely want to use more advanced models and provide a lot of context. On that front, the current model can read around 16,000 tokens (maybe 8,000 words). If you need more than that consider changing the model to something with a larger context window. See e.g., OpenAI model list. As noted above, eventually, you'll hit a limit to what you can simulate, but we will return to simulations in a future post.
Export and Share
After you've made the template your own and have it behaving the way you like, you can export and share it with others. This will produce an HTML file you can share. This file should work on any internet connected device. To create your file, click the Export Interactions Page button. The contents of the textarea above the button will be appended to the top of your exported file. Importantly, if you don't want to share your API key, you should temporarily remove it from your settings before exporting.
If you want to see what an exported file looks like without having to make one yourself. You can use the buttons below. View export in browser will open the file in your browser, and Download export will download a file. In either case the following custom header will be inserted into your file. It will NOT include an API key. So, you'll have to enter one when asked if you want to see things work. This information is saved in your browser. If you've provided it before, you won't be asked again. It is not shared with me. To remove this information for this site (and only this site, not individual files), you can follow the instructions found on my privacy page. Remember, when you export your own file, whether or not it contains and API key depends on if you have one defined at the time of output.
Custom header:
<h2>First Client Meeting</h2>
<p>
It's Friday morning and you're in the Rockham District Court, located in Massachusetts. You are today's duty attorney, meaning you will be picking up new cases as they come in for arraignments and bail arguments. The clerk has given you a folder containing the following documents, a complaint, incident report, and CORI. They also let you know that the prosecutor is looking to hold your client. Your client is downstairs in lockup. According to the documents, your client is being charged with disorderly conduct and has no prior record. No one was hurt, and there was no property damage. You head downstairs to meet them.
</p>
<p>
Remember, you're goal is to avoid talking too much about the facts of the case. There will be time for that later in a more private location. You need to learn about your client and his community ties, anything that will help you make a bail argument.
</p>
<p>
You enter lockup and ask to speak with your client. The guard directs you to a holding cell shared by a couple of individuals and calls your client's name. Your client steps forward and speaks to you through the window...
</p>
<hr style="border: solid 0px; border-bottom: solid 1px #555;margin: 5px 0 15px 0"/>Not sure what's up with all those greater than and less than signs? Looking for tips on how to style your HTML? Check out this general HTML tutorial.
The export you'll see after clicking the buttons below is what you'll get out of LIT Prompts. However, I linked to a special version of this file above. See here. I edited that version to collect analytics and to provide access to some prepaid LLM credits. The following will prompt users to enter LLM API info.
TL;DR References
ICYMI, if you didn't click through above, you might want to give this a look now.
- On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 by Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. There's a lot of history behind this paper. It was part of a chain of events that forced Timnit Gebru to leave Google where she was the co-lead of their ethical AI team, but more than that, it's one of the foundational papers in AI ethics, not to be confused with the field of "AI safety," which we will discuss later. It discusses several risks associated with large language models, including environmental/financial costs, biased language, lack of cultural nuance, misdirection of research, and potential for misinformation. If you want to engage critically with LLMs, this paper is a must read.