Sadly Not, Havoc Dinosaur

If a 3L Can Automate Law Students, I Can Automate Law Profs—Robo Socrates Lives!

A law school instructor shows you how to get AI to brief cases, explain them to a 5yo, play the gunner, and conduct a Socratic dialogue all for FREE!

Headshot of the author, Colarusso. David Colaursso

This is the 16th post in my series 50 Days of LIT Prompts.

Last week I couldn't escape the story of Bradley Neal, a 3L who has built a gen AI tool that creates case briefs and answers questions about cases. I love the story, and I love the tool. So much so I spent the weekend recreating most of its functionality. Like Mr. Neal's tool, mine can create case briefs, explain cases in plain language, and directly answer questions about a case. These are common tasks assigned to law students, but I didn't stop there. I also created a bot capable of acting like a law school professor, a bot that will grill you using the Socratic method. That's right, it's an on-call simulator. I call the suite of tools The Paper Chase 2.0. You can see the results here, or build and customize your own version for FREE by following the instructions below.

I suspect some of my colleagues will be a bit perturbed when they hear about these tools. Maybe we can spark another round of "should we ban laptops in the classroom?" We can add, "an AI will be 'wispering' in their ear" as another in the parade of horribles. It's a conversation that comes up every few years. It always reminds me a bit of Plato's critique of writing as expressed by "Socrates" in Phaedrus. He feared that people would become dependent on writing and wouldn't memorize things anymore. He wasn't wrong, but I kind of like not having to memorize everything. Now lest you get the wrong idea, I took my law school notes on paper and once delivered a soliloquy from memory during a faculty meeting to prove a point. I'm also not convenced Plato's critique of writing was a wholesale admonition of the process. To preview a metaphor you'll see again, the genie is out of the bottle. I'm interested here in figuring out how best to live with this reality, and that involves some nuance.

Effectively, Mr. Neal has created 21st century CliffsNotes. I would caution any law students out there to be careful using these tools for the same reason I might caution against relying too heavily on notes they didn't write themselves. When a teacher asks you to create a case brief, or any distillation of ideas for that matter, the product isn't what's really important. It's the journey that matters. So, the reason I love Mr. Neal's story is not the fact that others can now access his AI-generated case briefs. It's the fact that he built the tool. How cool that experience must have been. Likewise, I love the tool because it's exactly the type of thing I think more law students should be making. Above all, I love that we live at a time where you dear reader could be making tools just like this. Heck, if you finish reading this post, you'll have all the technical know-how you need. Of course, technical know-how isn't really the point either. I joke with my students that you never learn something as well as when you teach it. Then I ask them to imagine how well they'll learn something if they have to teach it to a computer. These tools can be used to great pedagogical purpose if we remember one very important thing. Case briefs and the answers to questions in seminar are to some extent MacGuffins, something needed to drive the plot but not important in themselves. They are a means to an end. Their value comes from their status as artifacts of understanding. Absent that understanding, they are empty words.

If there is any lesson that we should take from stories about genies granting wishes, it's that the desire to get something without effort is the real problem. Think about the story of "The Sorcerer's Apprentice," in which the apprentice casts a spell to make broomsticks carry water but is unable to make them stop. The lesson of that story is not that magic is impossible to control: at the end of the story, the sorcerer comes back and immediately fixes the mess the apprentice made. The lesson is that you can't get out of doing the hard work. The apprentice wanted to avoid his chores, and looking for a shortcut was what got him into trouble.

- Ted Chiang, Will A.I. Become the New McKinsey?

I taught high school physics and astronomy for six years before going to law school. So, I know very well the dangers and promise of cognitive prosthetics. I saw students randomly punch keys on a calculator, never moving beyond the assumption that all I wanted was an answer. Some never understood why I asked them to show their work. However, I also saw students take flight, soaring over the drudgery of common arithmetic, to genuinely explore a world of interconnected relationships described by functions and physical laws. They understood the importance of showing their work because they knew it wasn't only about the answers. I'm a dyslexic attorney, and I firmly believe this profession would have been closed to me had I been born a generation earlier, before spell check and text-to-speech. Cognitive prosthetics increase access. Yes, technological tools can undercut learning, but only if we ignore the lesson of The Sorcerer's Apprentice, only if we use them as "shortcuts." Here, I use "shortcuts" interchangeably with "cheats," and I distinguish both from efficiencies. The first two achieve speed by skipping over necessary steps whereas the latter is best described by the cliché, "work smarter, not harder." The problem is that if one is confused about the MacGuffin's value, it is unlikely they will be able to distinguish between the two. And so, some adopt a paternalistic view and assert that we must protect people from themselves, not by educating them about the difference, but by barring them access to that which they might misuse.

I understand this impulse. It's a lot of work combating the havoc these tools can unleash. Maybe it's just easier if we avoid it all together. Brandolini's Law is often cited in relation to Gen AI. It states, "The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it." The fear is that this asymmetry will grow with Gen AI-fueled BS. Perhaps, we should just nip it all in the bud. Again, I am reminded of Plato's critique of writing. I'm also reminded of the printing press. A lot of blood was shed in the wake of that disruptive technology. When the means of publishing were democratized, literacy spread along with ideas, dangerous ideas. For educators, however, I don't think prohibition is an appropriate answer, and that's because it seems too much like a shortcut. We must do the hard work, and we have to convince our students that it is in their interest to do the hard work as well.

I imagine three students:

I think we can agree that the third student will learn more than the second, and I'm confident there's good reason to think they will be better off than the first student as well. Here are at least three points in favor of this argument.

  1. In the process of creating and refining their tool, they will have to develop a more robust understanding of what it is to brief a case.
  2. They will be in the position when faced with tools like Mr. Neal's in the future to critically evaluate their efficacy. They'll know better than student number one when to call BS.
  3. They are unlikely to stop with case briefs. If they happen upon this post, they'll probably build a robo prof, but I strongly suspect they won't stop there either.

This is where I point out that our discussion of these tools can't remain academic. At some point, our students will find themselves working in a world with them. A part of our job must be helping our students gain the wisdom needed to wield them wisely.

Notes on Case Briefs, Gunners & The Socratic Method

The first scene of The Paper Chase (1973).

For any non-lawyer readers, some context. In US law schools, it's customary to have students read the opinions of courts to learn the law. It's often expected that students will produce short summaries of these cases called case briefs as part of their study. The specifics can differ, but a case brief usually has four main parts:

No one ever really asks to see your case briefs, but you will be questioned about the cases during class. Traditionally, this questioning involved the Socratic method. If you're unfamiliar with the Socratic method, it can be seen throughout the 1973 film the Paper Chase. Today, most law school instructors, however, engage in class discussions that only resemble this method. Ideally, a teacher's questions are an invitation for you to consider and integrate new ideas into your understanding of the world. The questioner is your guide, each question subtly guiding you to take a step this way or that into what for you remains unknown. Cynically, questions act as a motivation for everyone to do the reading. Either way, the hope is that they drive you to do the hard work of figuring out what it all means.

Sometimes teachers will cold call students and others will rely on volunteers. A gunner is a pejorative term used to describe overly eager students who aim to out gun and out shine their peers. Consequently, they always have an answer. Mr. Neal called his Q&A tool Gunnerbot.

Notes on the meaning of FREE

In the subhead above, I claimed that I would show you how to build your own AI tools for free. So, it's worth explaining what I meant by that. I meant that depending on what resources you have available to you it may be possible to build your own version of Paper Chase 2.0 without incurring any new costs aside from those associated with running your computer. We'll be building our tools with the LIT Prompts extension which is free to download and use. However, it will require you to make use of a Large Language Model (LLM). If you're computer is of the right make and model, you can run LM Studio locally and make use of an open source LLM for free. If you're new to OpenAI, you can make use of a small set of free credits given to you when you create a new account. In practice, however, it's probably easiest to pay for your LLM. It is, however, possible to avoid doing so.

The fact that you could build these tools for free, or very little money, raises some interesting questions about the future of products like Mr. Neal's. What is the right price for such a tool? What is it customers are paying for? It might also present some interesting implications for the delivery of legal services, but we'll leave that discussion for another time.

Disclaimer

I cobbled these tools together over the weekend. Consequently, I have not subjected them to rigorous testing. Before using them, you should examine the text of their prompt templates. Also, it's worth noting for those who haven't read the other posts in this series, a major point of these posts is to encourage experimentation with imperfect tools. LLMs hallucinate, and those hallucinations reflect the biases found in their training data. What I'm saying is, DON'T RELY ON THE OUTPUT OF THESE TOOLS!

You should treat an LLM's output like you would any secondary source. When it really matters, you need to track down and check the primary sources. That's why my two main use cases for summarization templates are skimming and refreshing my recollection of something I have already read.

With that out of the way...

Let's build something!

We'll do our building in the LIT Prompts extension. If you aren't familiar with the LIT Prompts extension, don't worry. We'll walk you through setting things up before we start building. If you have used the LIT Prompts extension before, skip to The Prompt Pattern (Template).

Up Next

Questions or comments? I'm on Mastodon @Colarusso@mastodon.social


Setup LIT Prompts

7 min intro video

LIT Prompts is a browser extension built at Suffolk University Law School's Legal Innovation and Technology Lab to help folks explore the use of Large Language Models (LLMs) and prompt engineering. LLMs are sentence completion machines, and prompts are the text upon which they build. Feed an LLM a prompt, and it will return a plausible-sounding follow-up (e.g., "Four score and seven..." might return "years ago our fathers brought forth..."). LIT Prompts lets users create and save prompt templates based on data from an active browser window (e.g., selected text or the whole text of a webpage) along with text from a user. Below we'll walk through a specific example.

To get started, follow the first four minutes of the intro video or the steps outlined below. Note: The video only shows Firefox, but once you've installed the extension, the steps are the same.

Install the extension

Follow the links for your browser.

  • Firefox: (1) visit the extension's add-ons page; (2) click "Add to Firefox;" and (3) grant permissions.
  • Chrome: (1) visit the extension's web store page; (2) click "Add to Chrome;" and (3) review permissions / "Add extension."

If you don't have Firefox, you can download it here. Would you rather use Chrome? Download it here.

Point it at an API

Here we'll walk through how to use an LLM provided by OpenAI, but you don't have to use their offering. If you're interested in alternatives, you can find them here. You can even run your LLM locally, avoiding the need to share your prompts with a third-party. If you need an OpenAI account, you can create one here. Note: when you create a new OpenAI account you are given a limited amount of free API credits. If you created an account some time ago, however, these may have expired. If your credits have expired, you will need to enter a billing method before you can use the API. You can check the state of any credits here.

Login to OpenAI, and navigate to the API documentation.

Once you are looking at the API docs, follow the steps outlined in the image above. That is:

  1. Select "API keys" from the left menu
  2. Click "+ Create new secret key"

On LIT Prompt's Templates & Settings screen, set your API Base to https://api.openai.com/v1/chat/completions and your API Key equal to the value you got above after clicking "+ Create new secret key". You get there by clicking the Templates & Settings button in the extension's popup:

  1. open the extension
  2. click on Templates & Settings
  3. enter the API Base and Key (under the section OpenAI-Compatible API Integration)

Once those two bits of information (the API Base and Key) are in place, you're good to go. Now you can edit, create, and run prompt templates. Just open the LIT Prompts extension, and click one of the options. I suggest, however, that you read through the Templates and Settings screen to get oriented. You might even try out a few of the preloaded prompt templates. This will let you jump right in and get your hands dirty in the next section.

If you receive an error when trying to run a template after entering your Base and Key, and you are using OpenAI, make sure to check the state of any credits here. If you don't have any credits, you will need a billing method on file.

If you found this hard to follow, consider following along with the first four minutes of the video above. It covers the same content. It focuses on Firefox, but once you've installed the extension, the steps are the same.


The Prompt Patterns (Templates)

I was able to put these tools together over the weekend because they're mostly a rehash of prior work. If you're looking for more context, I suggest you check out the following two posts: (1) Using AI to Distill and Question Texts; and (2) What if Members of Talking Professions Could Log Time in Simulators Like Pilots?

When crafting a LIT Prompts template, we use a mix of plain language and variable placeholders. Specifically, you can use double curly brackets to encase predefined variables. If the text between the brackets matches one of our predefined variable names, that section of text will be replaced with the variable's value. Today we'll be using the {{scratch}} variable. See the extension's documentation.

The {{scratch}} variable contains the text in your Scratch Pad which is accessible from the extension's popup window. The button is to the right of the Settings & Templates button that you have used before. That being said, we're actually introducing four templates today. They all work on the assumption that you have place the text of a legal opinion in the Scratch Pad.

The first two templates, will summarize the opinion, creating case briefs. They will also stay open for chat, allowing you to ask follow up questions. The second template makes use of the ELI5 (Explain like I'm 5) format to produce a plain language version of the case brief. Play the gunner, jumps right to a Q&A, and Play the prof launces an simulated law prof who will engage you in a Socratic dialogue.

FWIW, I've used OpenAI's gpt-3.5-turbo-16k to power the below templates. Mr. Neal has said that he's using gpt-4 though I'm not sure exactly what variant. Also, the description of case briefs found in the templates below was drafted with the aid of ChatGPT.

Brief case & answer questions

Here's the template's text.

In a moment I will give you the text of a legal opinion, also know as a case. I will ask you to pull out the important bits and use them to write a Case Brief. Keep the following in mind when writing your brief. 

---

Guide to Writing a Case Brief for Law School

1. Purpose of a Case Brief: Case briefs are essential study tools in law school, aiding in the analysis and recall of complex legal materials. They serve as a concise summary after thoroughly reading, dissecting, and understanding a case. Primarily for personal use, they also act as quick references for class discussions and exam preparations.

2. Audience: Primarily, you are the audience for your case briefs. Professors value the process but typically won't review your briefs. In professional practice, the focus is on effective legal representation rather than the briefing process itself.

3. Essential Elements: Your brief should include the following core components for it to be effective:

    Facts: Case name, parties involved, factual background, procedural history, and the judgment.
    Issues: The legal questions or disputes at the heart of the case.
    Holding: The court's legal decision or rule applied in the case.
    Rationale: The reasoning behind the court's decision.

4. Optional Elements: Depending on the case, consider adding:

    Dicta: Important but non-decisive commentary.
    Dissent: Valuable opposing opinions.
    Party’s Arguments: Each side's main arguments.
    Comments: Personal insights or notes.

5. Organization Tips: You might find it helpful to further break down the Facts into:

    Specific case facts
    Procedural history
    Judgment outcome

Remember, the Holding differs from the Judgment as it reflects the legal principle, whereas the Judgment is the case's final decision.

6. Relevance and Brevity: Focus on including relevant information that aids memory and understanding:

    Include crucial facts, especially those pivotal to the case's outcome.
    Highlight the main legal issue(s) and the court's final ruling.
    Summarize the rationale, focusing on key reasoning and factors influencing the decision.

7. Length: Aim for your briefs to be concise, ideally one page, to facilitate easy review and reference.

--- 

Okay, now here is the text of the case. 

{{scratch}}

---

When you write your brief, be sure that it includes the following sections: 

- Case Name: 
- Court: 
- Date Published: 
- Citations: 
- Parties: 
- Facts:
- Procedural History:
- Issues:
- Holding:
- Rationale:

If asked any follow-up questions, use the above text of the case, and ONLY the above text, to answer them. If you can't find an answer in the above text, politely decline to answer explaining that you can't find the information. You can, however, finish a thought you started above if asked to continue, but don't write anything that isn't supported by the above text. And keep all of your replies short! But first, give me your case brief for the above case.       
      

And here are the template's parameters:

ELI5 case brief & answer questions

Here's the template's text.

In a moment I will give you the text of a legal opinion, also know as a case. I will ask you to pull out the important bits and use them to write a Case Brief. Keep the following in mind when writing your brief. 

---

Guide to Writing a Case Brief for Law School

1. Purpose of a Case Brief: Case briefs are essential study tools in law school, aiding in the analysis and recall of complex legal materials. They serve as a concise summary after thoroughly reading, dissecting, and understanding a case. Primarily for personal use, they also act as quick references for class discussions and exam preparations.

2. Audience: Primarily, you are the audience for your case briefs. Professors value the process but typically won't review your briefs. In professional practice, the focus is on effective legal representation rather than the briefing process itself.

3. Essential Elements: Your brief should include the following core components for it to be effective:

    Facts: Case name, parties involved, factual background, procedural history, and the judgment.
    Issues: The legal questions or disputes at the heart of the case.
    Holding: The court's legal decision or rule applied in the case.
    Rationale: The reasoning behind the court's decision.

4. Optional Elements: Depending on the case, consider adding:

    Dicta: Important but non-decisive commentary.
    Dissent: Valuable opposing opinions.
    Party’s Arguments: Each side's main arguments.
    Comments: Personal insights or notes.

5. Organization Tips: You might find it helpful to further break down the Facts into:

    Specific case facts
    Procedural history
    Judgment outcome

Remember, the Holding differs from the Judgment as it reflects the legal principle, whereas the Judgment is the case's final decision.

6. Relevance and Brevity: Focus on including relevant information that aids memory and understanding:

    Include crucial facts, especially those pivotal to the case's outcome.
    Highlight the main legal issue(s) and the court's final ruling.
    Summarize the rationale, focusing on key reasoning and factors influencing the decision.

7. Length: Aim for your briefs to be concise, ideally one page, to facilitate easy review and reference.

--- 

Okay, now here is the text of the case. 

{{scratch}}

---

When you write your brief, be sure that it includes the following sections: 

- Case Name: 
- Court: 
- Date Published: 
- Citations: 
- Parties: 
- Facts:
- Procedural History:
- Issues:
- Holding:
- Rationale:

If asked any follow-up questions, use the above text of the case, and ONLY the above text, to answer them. If you can't find an answer in the above text, politely decline to answer explaining that you can't find the information. You can, however, finish a thought you started above if asked to continue, but don't write anything that isn't supported by the above text. And keep all of your replies short! But first, give me your case brief for the above case, but explain it to me in very simple terms such that anyone could understand it. Avoid all legalize and big words, shoot for a sixth grade reading level. As they might say on Reddit, ELI5. Continue to answer in the ELI5 format for any follow-up questions.  

And here are the template's parameters:

Play the gunner (answer questions)

Here's the template's text.

You are a law school student who is "on call" for the following case. That is, you are expected to read this case carefully and to be ready to answer questions about it in a thoughtful and intelligent manner. In a moment I will give you the text of the case. After that I will ask you questions about the case. Use the text of the case, and ONLY the above text, to answer them. If you can't find an answer in the case, politely decline to answer, explaining that you couldn't find the information in the case. You can, however, finish a thought you started if asked to continue, but don't write anything that isn't supported by the text of the case. And keep all of your replies short! Here's the case:

{{scratch}}

---

Now that you've read the case, answer the following questions. Remember, keep your replies short. 

{{Okay, I've read the case. What do you want to know?}} 
      

And here are the template's parameters:

Play the prof (Socrates lives!)

Here's the template's text.

You are an actor playing of a law school professor conducting a Socratic dialogue. In this scene you are interacting with a student, asking them questions about a case. In a moment I will show you the case so you can prepare your questions. Your job is to stay in character and act out your part. You are aiming for a realistic performance. To help you get into character, here is some background information.

BACKGROUND

You're character is very similar in demeanor to Professor Charles W. Kingsfield Jr. in the Paper Chase. You are hard but fair, and see your role in class as that of Socrates. You're job is to have students come to understand the material through guided questions. The following is how you have described what you try to do for each dialogue. 

Preparation

    - Choose a Case: Select a legal case that presents complex issues.
    - Identify Issues: Pinpoint the main legal issues and principles in the case.
    - Develop Questions: Create a sequence of questions ranging from broad to specific to guide the discussion.

Conducting the Dialogue

    - Start Broad: Initiate with questions that summarize the case and identify issues.
    - Probe Deeper: Continue with questions that analyze arguments, evaluate reasoning, and apply principles.
    - Encourage Participation: Foster an inclusive atmosphere for all students to contribute.
    - Guide Discussion: Facilitate the dialogue, connecting ideas and challenging assumptions.
    - Clarify and Summarize: Regularly clarify points and summarize key takeaways.

Deepening Understanding

    - Reflect on Discussion: Encourage students to consider the broader implications and applications.
    - Integrate Perspectives: Where relevant, bring in ethical, historical, or societal contexts.
    - Provide Feedback: Offer constructive feedback on students' analyses.

Concluding the Dialogue

    - Wrap-Up: Summarize the discussion's main insights and their relevance.
    - Connect Themes: Link the dialogue to overarching course themes.

Here's the text of the case you will be asking students about.

THE CASE

{{scratch}}

DIRECTION

Be sure to keep your questions and responses short. You "speak in sentences not paragraphs." Short and conversational, no speechifying!

THE CONVERSATION SO FAR

You are jumping into the scene in progress. You already opened the discussion by calling on a student and asking them, "Please describe the facts of the case." They responded with "{{Please describe the facts of the case.}}"

Think about how your character would respond and craft an appropriate reply. Remember, you are a law professor conducting a Socratic dialogue. Your goal is to embody your character while achieving a naturalistic believable performance. You will continue to play the part of your character throughout the conversation. Whatever happens, do NOT break character! Respond only with dialog, and include only the text of your reply (e.g., do NOT preface the text with the name of the speaker). What do you say?
        
  

And here are the template's parameters:

Working with the above templates

To work with the above templates, you could copy it and its parameters into LIT Prompts one by one, or you could download a single prompts file and upload it from the extension's Templates & Settings screen. This will replace your existing prompts.

You can download a prompts file (the above templates and their parameters) suitable for upload by clicking this button:


Kick the Tires

It's one thing to read about something and another to put what you've learned into practice. Let's see how this template performs.


Export and Share

After you've made the templates your own and have them behaving the way you like, you can export and share them with others. This will produce an HTML file you can share. This file should work on any internet connected device. To create your file, click the Export Scratch Page & Interactions Page button. The contents of the textarea above the button will be appended to the top of your exported file. Importantly, if you don't want to share your API key, you should temporarily remove it from your settings before exporting.

If you want to see what an exported file looks like without having to make one yourself. You can use the buttons below. View export in browser will open the file in your browser, and Download export will download a file. In either case the following custom header will be inserted into your file. It will NOT include an API key. So, you'll have to enter one when asked if you want to see things work. This information is saved in your browser. If you've provided it before, you won't be asked again. It is not shared with me. To remove this information for this site (and only this site, not individual files), you can follow the instructions found on my privacy page. Remember, when you export your own file, whether or not it contains and API key depends on if you have one defined at the time of output.

Custom header:

<h2>The Paper Chase 2.0</h2>
<p>
  Place the text of a court opinion to the left, then choose one of the options below. If you want to create a case brief, you can select either the first or second option. The second option, ELI5 (Explain it like I'm 5), will attempt to make the brief more accessible. If you have questions about the case, you can use any of the first three options. The third, "Play the gunner," will jump right to a Q&A. To test your own understanding, choose "Play the prof (Socrates lives!)" This will launch a simulated law prof who will engage you in a Socratic dialogue.
</p>
<p>
  To keep costs down, this tool is using OpenAI's <i>gpt-3.5-turbo-16k</i>. This limits the size of your case texts to something like half a dozen pages. FWIW, here's a case that fits in the context window: <a href="https://www.courtlistener.com/opinion/3574015/hawkins-v-mcgee/?type=o&q=HAWKINS%20v.%20McGEE&type=o&order_by=score%20desc&stat_Precedential=on" target="_blank">Hawkins v. McGee</a>. If you want a larger input size or "better" briefs, consider making your own version of this tool and choosing a different model. Here's a <a href="https://sadlynothavocdinosaur.com//posts/socrates-lives/">How To</a> you can follow.   
</p>
<p>
  If you're in the middle of a chat below, and you want to restart, just refresh this page. Your text to the left will be saved. <a href="https://sadlynothavocdinosaur.com//posts/socrates-lives/#disclaimer">DISCLAIMER</a>
</p>
<hr style="border: solid 0px; border-bottom: solid 1px #555;margin: 5px 0 15px 0"/>

Not sure what's up with all those greater than and less than signs? Looking for tips on how to style your HTML? Check out this general HTML tutorial.


TL;DR References

ICYMI, here are blubs for a selection of works I linked to in this post. If you didn't click through above, you might want to give them a look now.