Making Expert Systems with LLMs
David Colaursso
Co-director, Suffolk's Legal Innovation & Tech Lab
This is the 31st post in my series 50 Days of AI. Previous Post | Next Post
A few week's back we talked about expert systems as one form of AI. They were the prefered methdod for the symbols folks. Attempst to codify expert knowledge into a set of rules.
After the term artificial intelligence was coined, two main camps emerged, the symbolicists and the connectionist. The first assumed that it would be necessary for people to program rules into computers if they wanted them to perform "human" tasks like identifying images. The latter reasoned it was better to have computers "discover" the rules in data. For a time, the focus was on the symbolic approach. Results, however, didn't scale and a winter fell upon the land. With the increased availability of data and computation, the connectionist regained the spotlight thanks to a bitter lesson, more data and inferred rules tend to out perform expert-drafted rules. Now folks argue about if more data is really all one needs and many assume the future lays in a combination of the two approaches, and today we'll play a little at their intersection.
Well, back in 2014 I wrote a simple programming language for lawyers called Qn AMarkup. You can learn everything you need to know to write QnA by watching this 10 min video on how to write QnA or reading this How To page. However, today, we're going to see if we can skip over that bit and get our prompt to write one.
The video up above is a roughly 30 min lesson on how to create flowcharts from a deep reading of a text. It's from an old class of mine. After watching it, I invite you to create a QnA Markup. However, unlike my former students, you can use the template I've created below. Importnatly, your mission can start now as we will not work on it together. You'll find more info under Kick the Tires.
Let's build something!
We'll do our building in the LIT Prompts extension. If you aren't familiar with the LIT Prompts extension, don't worry. We'll walk you through setting things up before we start building. If you have used the LIT Prompts extension before, skip to The Prompt Pattern (Template).
Up Next
Setup LIT Prompts
LIT Prompts is a browser extension built at Suffolk University Law School's Legal Innovation and Technology Lab to help folks explore the use of Large Language Models (LLMs) and prompt engineering. LLMs are sentence completion machines, and prompts are the text upon which they build. Feed an LLM a prompt, and it will return a plausible-sounding follow-up (e.g., "Four score and seven..." might return "years ago our fathers brought forth..."). LIT Prompts lets users create and save prompt templates based on data from an active browser window (e.g., selected text or the whole text of a webpage) along with text from a user. Below we'll walk through a specific example.
To get started, follow the first four minutes of the intro video or the steps outlined below. Note: The video only shows Firefox, but once you've installed the extension, the steps are the same.
Install the extension
Follow the links for your browser.
- Firefox: (1) visit the extension's add-ons page; (2) click "Add to Firefox;" and (3) grant permissions.
- Chrome: (1) visit the extension's web store page; (2) click "Add to Chrome;" and (3) review permissions / "Add extension."
If you don't have Firefox, you can download it here. Would you rather use Chrome? Download it here.
Point it at an API
Here we'll walk through how to use an LLM provided by OpenAI, but you don't have to use their offering. If you're interested in alternatives, you can find them here. You can even run your LLM locally, avoiding the need to share your prompts with a third-party. If you need an OpenAI account, you can create one here. Note: when you create a new OpenAI account you are given a limited amount of free API credits. If you created an account some time ago, however, these may have expired. If your credits have expired, you will need to enter a billing method before you can use the API. You can check the state of any credits here.
Login to OpenAI, and navigate to the API documentation.
Once you are looking at the API docs, follow the steps outlined in the image above. That is:
- Select "API keys" from the left menu
- Click "+ Create new secret key"
On LIT Prompt's Templates & Settings screen, set your API Base to https://api.openai.com/v1/chat/completions and your API Key equal to the value you got above after clicking "+ Create new secret key". You get there by clicking the Templates & Settings button in the extension's popup:
- open the extension
- click on Templates & Settings
- enter the API Base and Key (under the section OpenAI-Compatible API Integration)
Once those two bits of information (the API Base and Key) are in place, you're good to go. Now you can edit, create, and run prompt templates. Just open the LIT Prompts extension, and click one of the options. I suggest, however, that you read through the Templates and Settings screen to get oriented. You might even try out a few of the preloaded prompt templates. This will let you jump right in and get your hands dirty in the next section.
If you receive an error when trying to run a template after entering your Base and Key, and you are using OpenAI, make sure to check the state of any credits here. If you don't have any credits, you will need a billing method on file.
If you found this hard to follow, consider following along with the first four minutes of the video above. It covers the same content. It focuses on Firefox, but once you've installed the extension, the steps are the same.
The Prompt Pattern (Template)
When crafting a LIT Prompts template, we use a mix of plain language and variable placeholders. Specifically, you can use double curly brackets to encase predefined variables. If the text between the brackets matches one of our predefined variable names, that section of text will be replaced with the variable's value. Today we'll be using two variables we've seen before: {{highlighted}} and {{passThrough}}. See the extension's documentation.
The {{highlighted}} variable contains any text you have highlighted/selected in the active browser tab when you open the extension. To use {{passThrough}}, however, we make use of the Post-run Behavior parameter to pass data from one template to another. If you use Post-run Behavior to send one template's output to another template, the first template's output can be read by the second template via the {{passThrough}} variable. Here we have two templates, one sending data to the other. The first "trick" is that the Post-run Behavior of the second prompt being set to "createMarkup," which is the name of our first template. This means that after the second prompt is run it will trigger "createMarkup," and pass its. Because the Output To is set to Hidden, however, the user will not see this data transfer and because the first prompt below has Hide Button set to true the user will only see the second button.
To use this prompt, find text with rules in it, highlight them and run. It should provide you with QnA Markup that you can paste into the editor at QnAMarkup.org.
Here's the first template's title.
createMarkup
Here's the template's text.
You an attorney helping to create a decision tree for the rules found in a document. You are to read the document and then provide code for the decision tree/flowchart in QnA Markup. First I'll show you the syntax for QnA Markup, then I'll show you the document.
Here's a description of QnA Markup.
===== START QNA MARKUP =====
{{passThrough}}
===== END QNA MARKUP =====
Noe here's the document with rules for you to encode.
===== START DOC =====
{{innerText}}
===== END DOC =====
Now given use what you know about QnA Markup to encode the rules set out in the document into properly formatted QnA Markup.
And here are the template's parameters:
- Output Type:
LLM. This choice means that we'll "run" the template through an LLM (i.e., this will ping an LLM and return a result). Alternatively, we could have chosen "Prompt," in which case the extension would return the text of the completed template. - Model:
gpt-4o-mini. This input specifies what model we should use when running the prompt. Available models differ based on your API provider. See e.g., OpenAI's list of models. I activly choose this model because the lates model from OpenAI didn't work as well. Model choice matters. - JSON:
No. This asks the model to output its answer in something called JSON. We don't need to worry about that here, hence the selection of "No." - Output To:
Screen Only. We can output the first reply from the LLM to a number of places, the screen, the clipboard... Here, we're content just to have it go to the screen. - Post-run Behavior:
FULL STOP. Like the choice of output, we can decide what to do after a template runs. To keep things simple, I went with "FULL STOP." - Hide Button:
checked. This determines if a button is displayed for this template in the extension's popup window. Here we want to hide this button as it wouldn't work properly if triggered before the prior prompt because it is counting on the passThrough variable.
Of course for that prompt to work it needs to be fed the rules for making QnA Markup. The following template does that. Here's its title.
Encode into QnA Markup
Here's the template's text.
---
name: qna-markup
description: Write, edit, review or convert content into QnA Markup, the plain-text language (Q:, A:, X:, DOC:, GOTO:) that www.qnamarkup.org turns into interactive question-and-answer interviews, guided forms, decision trees and document assemblers. Use whenever someone asks for "a QnA", QnA Markup, an interactive interview or decision tree they can run in a browser, or asks to encode a process, checklist, policy, FAQ or document as questions and answers.
---
# QnA Markup
QnA Markup is a plain-text language. A `.txt` file of it, pasted into the editor at
https://www.qnamarkup.org/, becomes a chat-style interview: the visitor is shown one question at a
time, answers by clicking a button or typing, and is led down a branch you wrote. It was made for
legal-aid style guided interviews, but it is a general decision-tree / guided-form language.
## What to produce
Produce **plain text**, indented with **tabs**, saved or offered as a `.txt` file (or shown in a
code block). Never a table, never Markdown bullets, never HTML around it. A minimal QnA:
```
Title: Is it a weekend?
Q: What day is it?
A: Saturday
Q: Then it's the weekend.
A: Sunday
Q: Then it's the weekend.
A: Something else
Q: Then it's a weekday.
```
## Workflow
1. **Understand the process** before writing. Find the decision points (each becomes a `Q` with `A`
buttons), the facts to collect (each an `X` text field), the outcomes (leaf questions with no
answers), and any text to generate (`DOC:`). Ask the person, briefly, if something essential is
unclear; otherwise make sensible assumptions and say what they were.
2. **Draft the tree.** One question per `Q`. Answer buttons phrased as answers ("Yes", "Under 18",
"I rent"), not as questions. Every path must end somewhere sensible (a closing `Q` with no
answers, or a `GOTO` to one). Give every question that anything jumps to a name: `Q(rent):`.
3. **Check it against the rules below** (indentation, alternation, GOTO placement, unique names).
If `scripts/check.js` is available, run `node scripts/check.js file.txt`; it prints the
interpreter's own error messages with line numbers, or `OK`.
4. **Hand it over** with one line on how to run it: paste into https://www.qnamarkup.org/ (or open
`https://www.qnamarkup.org/?source=URL` when the file is on the web with CORS enabled), then use
*Update Outputs*, *Link* or *HTML* to share it.
## The rules that matter (the interpreter enforces them)
**Tags** start a line (after indentation) and end with a colon:
| tag | meaning |
|---|---|
| `Title:` `Author:` `Description:` | header, before the first `Q`; shown under *credits* |
| `Before:` `After:` | HTML (and `<script>`) placed above / below the conversation |
| `Q:` or `Q(name):` | a question; text follows, may span lines, may contain HTML |
| `A:` or `A(value):` | a button; `value` is what the variable stores (default: the button text) |
| `A[href]:` / `A:[href]` | a button that is a link (same window / new window); `javascript:` runs code |
| `X:` | a text field instead of buttons; `X:number` for a number field; nothing else may follow the colon; one per question |
| HTML form controls in a `Q` | `<input name="dob" type="date">`, checkboxes, `<select>`, `<textarea>` written in the question text become variables named by their `name`; the `A` button submits them (see reference §Form fields) |
| `DOC:` | text collected for a document when the `Q` right under it is shown |
| `GOTO:name` | the last thing in a `Q`'s text: continue at that question |
**Structure**
* Indentation is nesting: the answers to a question sit at the **same** indentation as the question;
the question that follows an answer sits **one level deeper** (one tab more). Use tabs (or a
consistent number of spaces); mixing widths is an error.
* Tags alternate: a `Q` is followed by its `A`/`X` tags, each `A`/`X` by at most one `Q` beneath it.
Two `Q`s in a row at the same level, or an `A` deeper than its `Q`, are errors.
* Every QnA needs at least one `Q` and one `A` or `X`.
* The conversation starts at the first top-level `Q`. Other top-level `Q`s are reachable only by
`GOTO` (they are where shared endings and loops live).
* An answer with nothing beneath it ends the conversation there (fine for a closing statement:
put the statement in a `Q` with no answers instead, so the visitor reads it).
**Names and variables**
* `Q(name):` names the question and the variable it stores. Names: letters, digits, `.`, `_`, `-`;
unique within the QnA. Leave the parentheses out and the editor numbers the question (`Q(1.2.1):`);
those numbers are machine names and shift when the tree changes, so use words for anything you
refer to.
* The visitor's answer is stored under the question's name: the `A(value)` (or button text), or the
typed text of an `X`. Show it later with `<x>name</x>` anywhere in a `Q`, `A` or `DOC`.
**GOTO**
* `GOTO:target` must be the last thing in the `Q`'s text, one per `Q`, never in an `A`. The
target is a name (or an id). A `Q` that is only `GOTO:x` is a silent jump; text before it is
shown first.
* Loops (a `GOTO` back to an earlier question) are how you retry or repeat.
**Text**
* Question and answer text is HTML: `<b>`, links, images, `<br>` all work. Two `<br><br>` split a
question into two bubbles. Plain text is fine and usually best.
* Keep questions short; one thing per question. Buttons are short phrases.
## Templates for common asks
* **"Encode this process / policy / checklist as a QnA."** Walk the source in order. Each condition
("if the tenant received notice…") becomes a `Q` with the possible cases as `A`s; each step that
needs a fact becomes an `X`; each outcome becomes a closing `Q` (or a `GOTO` to a shared one).
Quote the source's wording in the outcomes so the result stays faithful. Name the questions after
the concepts (`Q(notice):`), not the numbering of the source.
* **"Help me write a QnA about X."** Ask what the visitor should end up with (an answer, a document,
a referral), then design from the outcomes back to the first question.
* **"Turn this form / letter into a QnA that fills it in."** Collect each field with an `X`, then a
`DOC:` tag above a final `Q` assembles the text with `<x>field</x>`, and a button shows it:
`A[javascript:showdoc('Review, then print or save.')]: Show my letter`. See
`examples/letter.txt`.
* **Adding logic** (compare a number, pick a random branch, compute something): a script in an
`X[javascript:…]:` or `A[javascript:…]:` bracket can call `goto('name')` and read answers with
`getvar('name')`. See `examples/quiz.txt`. Loading another QnA file into the conversation:
`loadQnA('url', 'done', 'next')`. Details in `reference.md`.
## Before you hand it over
* [ ] Tabs, consistent; answers level with their question; next question one level deeper.
* [ ] Every `GOTO` names a question that exists exactly once, and is the last thing in its `Q`.
* [ ] No `GOTO` in an `A`; nothing after `X:` on its line (except `number`); one `X` per question.
* [ ] Every form control written in a `Q` has a `name`, and no name repeats a question's name or id.
* [ ] Names unique, made of letters/digits/`.`/`_`/`-`; the ones referenced are words.
* [ ] Every branch ends in a readable closing question (not in a dangling answer) unless it loops.
* [ ] `<x>name</x>` only for names that will have a value by then.
* [ ] `Title:` (and `Description:`) filled in; no `Settings:` line (the editor manages that).
`reference.md` has the full syntax, the predefined JavaScript functions, the error messages and
what causes them. `examples/` has complete, checked QnAs to pattern from.
And here are the template's parameters:
- Output Type:
Prompt. TBy choosing "Prompt" the template runs without being submitted to an LLM. It's output is just the template after slotting in variable values. - JSON:
No. This asks the model to output its answer in something called JSON. We don't need to worry about that here, hence the selection of "No." - Output To:
Hidden. We can output the first reply from the LLM to a number of places, the screen, the clipboard... Here, we're going to hide the conetnt as it is intended for the next template, not the user to see. - Post-run Behavior:
createMarkup. Like the choice of output, we can decide what to do after a template runs. Here we have chosen the name of another template. So, after this template runs it will trigger that template and pass through its output. - Hide Button:
unchecked. This determines if a button is displayed for this template in the extension's popup window.
Working with the above template
To work with the above template, you could copy it and its parameters into LIT Prompts one by one, or you could download a single prompts file and upload it from the extension's Templates & Settings screen.
You can download a prompts file (the above template and its parameters) suitable for upload by clicking this button:
Kick the Tires
It's one thing to read about something and another to put what you've learned into practice. Let's see how this template performs.
- See how well it handels DACA. Skim (or watch at 3x speed) the Youtube Video above. Then open up the original DACA Memo and see how our template does. Here's an HTML version of the memo. Remember LIT Prompts won't work on a PDF unless you copy and past the text into something like the Scratch Pad. So, I'd suggest opening the HTML version, highlighting the page and runing the prompt. It should provide you with QnA Markup that you can paste into the editor at QnAMarkup.org.
- Make your own QnA. Find some text with an embedded rule and see if you can get this prompt to turn it into a QnA Markup interview.





