Sadly Not, Havoc Dinosaur

TL;DR References (A.I. & Large Language Models)

A reading list for folks interested in understanding large language models (LLMs)
AI-Generated image of the New York Public Library Reading Room
Reading Room, latent space "photography" by Colarusso

Headshot of the author, Colarusso. David Colaursso
Co-director, Suffolk's Legal Innovation & Tech Lab

ICYMI, here are blubs for a selection of works referenced as part of my 50 Days of LIT Prompts series. Most of them are about LLMs and AI, but I have slipped in some random Shakespeare. 😉 If you want to understand the context in which I referenced them, there are links to the posts from which they came. I've aimed, however, to order them here such that they would made sense as readings absent that context. That being said, the serise is still unfolding. So, expect this list to grow. Updated regularly.

Key: 🗞️ Popular Press/General Audaince | ⚖️ Law Journalish | 📖 Bookish | 🤖 Technical or Social Sci Paperish | 🌎 Blog Post | 🎭 Play or Movie. Things don't always fit in nice buckets, hence the use of "ish."

1. ChatGPT Is a Blurry JPEG of the Web

By Ted Chiang. Writing at the beginning of ChatGPT's rise to prominence, this article discusses the analogy between language models like ChatGPT and lossy compression algorithms. Chiang argues that while models can repackage/compress web information, they lack true understanding. Ultimately, Chiang concludes that starting with a blurry copy is not ideal when creating original content and that the struggling to express thoughts is an essential element of the writing process.

🗞️ Referenced in:

2. We are an information revolution species

By Ada Palmer. Palmer discusses the ongoing information revolution and the impact of AI on society. She emphasizes that information revolutions have been a normal part of human life for centuries, and AI is just the latest iteration of this trend. Palmer argues that AI has the potential to democratize the power to create media, such as video games and movies, and enable more people to express themselves artistically. She acknowledges that AI may threaten certain livelihoods, but believes that thoughtful transitions and safety nets can help mitigate these challenges. Palmer also addresses concerns about fake news and propaganda, noting that society has always learned to combat the dangers of new media. She concludes by emphasizing the importance of policy and planning to ensure that the rollout of AI is beneficial for all. Summary based on a draft from our day one template.

🗞️ 🌎 Referenced in:

3. Will A.I. Become the New McKinsey?

By Ted Chiang. This article explores the potential risks and consequences of artificial intelligence (A.I.) in relation to capitalism. Chiang suggests that A.I. can be seen as a management-consulting firm, similar to McKinsey & Company, which concentrates wealth and disempowers workers. He argues that A.I. currently assists capital at the expense of labor, and questions whether there is a way for A.I. to assist workers instead of management. Chiang also discusses the need for economic policies to distribute the benefits of technology appropriately, as well as the importance of critical self-examination by those building world-shaking technologies. He concludes by emphasizing the need to question the assumption that more technology is always better and to engage in the hard work of building a better world. Summary based on a draft from our day one template.

🗞️ Referenced in:

4. Prediction Machines - The Simple Economics of Artificial Intelligence

By Professors Ajay Agrawal, Joshua Gans, and Avi Goldfarb. I find the framing of AI tools as "prediction machines" to be both accurate and concise. The first edition of this book was a very good framing of AI as prediction. Apparently, there is a new edition of the book though I've only read the original. That version was written well before the current shift in the meaning of "AI." When the first edition was published, the vernacular use of AI was most often attached to machine learning; now it attaches to LLMs.

🗞️ 📖 Referenced in:

5. Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence

By Shakir Mohamed, Marie-Therese Png & William Isaac. The article discusses the integration of decolonial theory into artificial intelligence (AI) to address ethical and societal impacts. It highlights the importance of critical science and post-colonial theories in understanding AI's role in modern societies, emphasizing the need for a decolonial approach to prevent harm to vulnerable populations. The paper proposes tactics for developing a decolonial AI, including creating a critical technical practice, seeking reverse tutelage, and renewing affective and political communities. These strategies aim to align AI research and technology development with ethical principles, centering on the well-being of all individuals, especially those most affected by technological advancements. Summary based on a draft from our day one template.

🤖 Referenced in:

6. GPT-4 Passes the Bar Exam

By Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. When this paper came out it caused quite a stir in legal academia. As the title suggests, it demonstrated that an LLM could pass the Multi State Bar Exam. Don't confuse this with the arrival of AI lawyers. What's undeniable is that such an accomplishment says something interesting. I tend to think it says more about the way we test lawyers than most commentary on it would suggest, but like the next link, it's the source of something you may have heard somewhere else, "AI Passes the Bar!!!" ⚖️

⚖️ 🤖 Referenced in:

7. Prompt injection explained, November 2023 edition

By the person who coined the term—Simon Willison. TL;DR: Prompt injection is a security vulnerability where users can override intended instructions in a language model, by "hiding" instructions in texts, potentially causing harm or unauthorized access, and we don't have a 100% solution to this. So, there a lot of things folks want to build with these tools that the shouldn't.

🌎 Referenced in:

8. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

By Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. There's a lot of history behind this paper. It was part of a chain of events that forced Timnit Gebru to leave Google where she was the co-lead of their ethical AI team, but more than that, it's one of the foundational papers in AI ethics, not to be confused with the field of "AI safety," which we will discuss later. It discusses several risks associated with large language models, including environmental/financial costs, biased language, lack of cultural nuance, misdirection of research, and potential for misinformation. If you want to engage critically with LLMs, this paper is a must read.

🤖 Referenced in:

9. Some quick thoughts about integrating AI with law school clinical practice

By Quinten Steenhuis. I co-direct the LIT Lab with Quinten and really appreciate his take on the use of AI in law school clinics. He believes that law school clinics should be using generative AI tools, but acknowledges that it requires careful thought and planning. Steenhuis suggests several safe uses for AI in clinical education, such as solving the blank page problem, brainstorming, extracting information, classifying, editing, translating, and simplifying. He also addresses concerns about teaching generative AI, including the risk of automation bias and perpetuating biases. Steenhuis emphasizes the importance of teaching students how to critically evaluate AI output and suggests integrating AI lessons into existing curriculum. He concludes by stating that generative AI has practical uses and ignoring it in clinical practice will put law students at a disadvantage. Summary based on a draft from our day one template.

⚖️ 🌎 Referenced in:

10. Any sufficiently transparent magic . . .

By Damien Patrick Williams. The article explores the connections between religious perspectives, myth, and magic with the development of algorithms and artificial intelligence (AI). The author argues that these elements are not only lenses to understand AI but are also foundational to its development. The article highlights the need to consider social and experiential knowledge in AI research and emphasizes the importance of engaging with marginalized voices to better understand and mitigate the harms of AI. The author also draws parallels between AI and magical beings, such as djinn, suggesting that AI systems may fulfill desires as thoroughly as they would for themselves. The article critiques the terminology and hype surrounding AI, calling for a more intentional examination of the religious and magical aspects of AI. Summary based on a draft from our day one template. 🤖

🤖 Referenced in:

11. Efficient Estimation of Word Representations in Vector Space

By Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. This is the paper that introduced word2vec to the world. It's a technical paper. If that scares you, consider looping back and looking at it after we finish this week's micro-lessons. It might make more sense given that perspective as we'll spend the next several posts unpacking what word2vec does and why it matters.

🤖 Referenced in:

12. GOAT: Who is the Greatest Economist of all Time and Why Does it Matter?

A generative book by Tyler Cowen. From the site, "Do you yearn for something more than a book? And yet still love books? How about a book you can query, and it will answer away to your heart's content? How about a book that will create its own content, on demand, or allow you to rewrite it? A book that will tell you why it is (sometimes) wrong? That is what I have tried to build with my latest work. It's called GOAT: Who is the Greatest Economist of all Time and Why Does it Matter?"

📖 Referenced in:

13. Romeo and Juliet

By William Shakespeare. Technically, I didn't link to this in my post, but I did allude to it a couple of times. Either way, I'll take any chance I can to share the fact that Project Gutenberg has a great selection of public domain works available to read on the web or with your e-reader. The above link will get you the whole play.

🎭 Referenced in:

14. The Dictionary Is Not a Fortress: Definitional Fallacies and a Corpus-Based Approach to Plain Meaning

By Stephen C. Mouritsen. This paper discusses the limitations of relying solely on dictionaries for determining the meaning of statutory terms in legal interpretation. The author argues that dictionaries are not infallible and can be subject to definitional fallacies. Instead, the author proposes a corpus-based approach to determining the ordinary meaning of statutory terms, concluding that Corpus Linguistics has the potential to provide a more objective and empirical approach to statutory interpretation. As you might imagine the devil is in the details, and though many have taken up the charge of legal corpus linguistics others have questioned its application, hence, my next reference.

⚖️ Referenced in:

15. Legal Corpus Linguistics and the Half-Empirical Attitude

By Anya Bernstein. Its proponents argue that corpus linguistics provides empirical grounding to claims about ordinary language. However, the paper argues that legal corpus linguistics falls short in delivering on this promise because it ignores the relevant legal and institutional contexts in which legal language is produced and interpreted. The author refers to this approach as a "half-empirical attitude" because it treats normative claims as empirical findings. The paper suggests that legal corpus linguistics could be useful to legal theory if it embraces a more comprehensive empirical attitude.

⚖️ Referenced in:

16. Wherein The Copia Institute Tells The Copyright Office There's No Place For Copyright Law In AI Training

By Cathy Gellis. This article outlines a comment filed by the Copia Institute with the US Copyright Office, arguing that copyright law should not apply to AI training. The comment states that copyright law should not interfere with AI training because it would impede the public's right to consume works. They argue that AI training is an extension of the public's right to use tools, including software tools, to help them consume works. The comment also notes that AI training is not the same as copying or distributing copyrighted works, as it involves the analysis and processing of information rather than the creation of new works. They conclude that copyright law should not have a role in AI training and that AI training should be considered fair use or exempt from copyright altogether.

⚖️ 🌎 Referenced in:

17. Copyright Liability On LLMs Should Mostly Fall On The Prompter, Not The Service

By Ira Rothken. The use of large language models (LLMs) like ChatGPT has raised questions about the bounds of fair use and the responsibilities of AI developers and users in relation to copyright law. In this article Rothken proposes the "Training and Output" (TAO) Doctrine as a way to address these issues. The TAO Doctrine suggests that if an AI LLM engine is trained using copyrighted works and the outputs generated are based on user prompts, the responsibility for any potential copyright infringement should lie with the user, not the AI system. This approach recognizes the dual-use nature of AI technologies and emphasizes the importance of user intent and inputs in determining the nature of the output and any downstream usage. The TAO Doctrine aims to strike a balance between fostering innovation and respecting copyright laws.

⚖️ 🌎 Referenced in:

18. Distributed Representations of Words and Phrases and their Compositionality

By Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. This is the second word2vec paper, and it is here you can find a discussion of the country-capital directionality we noted in this mico-lesson.

🤖 Referenced in:

19. The Paper Chase (1973 Film)

Directed by James Bridges. "The Paper Chase" is a 1973 American comedy-drama. It is based on John Jay Osborn Jr.'s 1971 novel of the same name. The film follows James Hart, a first-year law student at Harvard Law School, as he navigates his studies and his complicated relationship with Professor Charles Kingsfield, a demanding contract law instructor. John Houseman won an Academy Award for his performance as Professor Kingsfield. The film received positive reviews for its portrayal of the intense and competitive environment of law school. It was followed by a television series that ran for four seasons, continuing the story of James Hart's law school journey. Summary based on a draft from our day one template.

🎭 Referenced in:

20. Unsupervised Machine Scoring of Free Response Answers—Validated Against Law School Final Exams

By David Colarusso. This paper presents a novel method for unsupervised machine scoring of short answer and essay question responses, relying solely on a sufficiently large set of responses to a common prompt, absent the need for pre-labeled sample answers—given said prompt is of a particular character. That is, for questions where “good” answers look similar, “wrong” answers are likely to be “wrong” in different ways. Consequently, when a collection of text embeddings for responses to a common prompt are placed in an appropriate feature space, the centroid of their placements can stand in for a model answer, providing a lodestar against which to measure individual responses. This paper examines the efficacy of this method and discusses potential applications.

🤖 Referenced in:

21. Beyond Readability with RateMyPDF: A Combined Rule-based and Machine Learning Approach to Improving Court Forms

By Quinten Steenhuis, Bryce Willey, and David Colarusso. In this paper, we describe RateMyPDF, a web application that helps authors measure and improve the usability of court forms. It offers a score together with automated suggestions to improve the form drawn from both traditional machine learning approaches and the general purpose GPT-3 large language model. We worked with form authors and usability experts to determine the set of features we measure and validated them by gathering a dataset of approximately 24,000 PDF forms from 46 U.S. States and the District of Columbia. Our tool and automated measures allow a form author or court tasked with improving a large library of forms to work at scale. This paper describes the features that we find improve form usability, the results from our analysis of the large form dataset, details of the tool, and the implications of our tool on access to justice for self-represented litigants. We found that the RateMyPDF score significantly correlates to the score of expert reviewers. While the current version of the tool allows automated analysis of Microsoft Word and PDF court forms, the findings of our research apply equally to the growing number of automated wizard-driven interactive legal applications that replace paper forms with interactive websites.

🤖 Referenced in:

22. Hamlet, Prince of Denmark

By William Shakespeare. Technically, I didn't link to this in any of my posts, but I did allude to it a couple of times. Either way, I'll take any chance I can to share the fact that Project Gutenberg has a great selection of public domain works available to read on the web or with your e-reader. The above link will get you the whole play.

🎭 Referenced in:

23. Silicon Valley Confronts a Grim New A.I. Metric

By Kevin Roose. P(doom), or the probability of doom, is a statistic that some artificial intelligence researchers use to assess the likelihood of an AI apocalypse or other catastrophic event caused by AI. It has become a popular topic of discussion in Silicon Valley, with techies casually asking each other about their p(doom) as a way to gauge their views on the potential risks of AI. The term originated on an online message board called LessWrong and has since been adopted by members of the Effective Altruism movement. However, p(doom) is not a precise measurement and is more about where someone stands on the spectrum of utopia to dystopia. It reflects their thoughts on the potential impact of AI and its regulation. Summary based on a draft from our day one template.

🗞️ Referenced in:

24. Moderator Mayhem: A Mobile Game To See How Well YOU Can Handle Content Moderation

By Mike Masnick. Moderator Mayhem is a mobile browser-based game developed in partnership with Engine that allows players to experience the challenges of content moderation. In the game, players act as front-line content moderators for a fictional review website called TrustHive. They must make quick decisions about whether to keep up or take down user-generated content based on the company's policies. The game aims to provide a realistic understanding of the complex scenarios and competing pressures faced by content moderators. Players receive feedback on their performance and can see how their decisions are perceived by the public and their superiors. The game does not provide a "correct" answer, as content moderation often involves subjective judgment. Summary based on a draft from our day one template.

🌎 Referenced in:

25. Thou shalt not commit logical fallacies

By School of Thought. FWIW, this poster hangs in my office at the law school. From the site's description, "A logical fallacy is a flaw in reasoning. Logical fallacies are like tricks or illusions of thought, and they're often very sneakily used by politicians and the media to fool people. Don't be fooled! This website has been designed to help you identify and call out dodgy logic wherever it may raise its ugly, incoherent head."

🌎 Referenced in:

26. Planet Money Episode 763: BOTUS

Wall Street is increasingly being taken over by computers and bots, even in the realm of stock-picking. Bots are cheaper, less emotional, and more disciplined than human stock-pickers, and they can process large amounts of information at once. To understand how these stock-picking bots work, the Planet Money podcast built their own bot called @BOTUS. This bot looks at President Trump's Twitter feed and trades stocks based on his tweets, using real money. If Trump tweets positively about a company, the bot buys its stock, and if he tweets negatively, the bot sells it short. The bot holds the position for 30 minutes before getting out. The Planet Money staff members have invested $1,000 of their personal funds in this experiment to see if the bot can make money. Summary based on a draft from our day one template.

🗞️ Referenced in:

27. The Elements of Style

By William Strunk Jr. The Elements of Style is a style guide for writing American English. It was originally written by William Strunk Jr. in 1918 and published in 1920. The book includes eight rules of usage, ten principles of composition, some matters of form, a list of commonly misused words and expressions, and a list of often misspelled words. Summary based on a draft from our day one template.

📖 Referenced in:

28. This is how AI image generators see the world

By Jeremy B. Merrill. Artificial intelligence image generators, such as Stable Diffusion and DALL-E, have been found to amplify bias in gender and race, despite efforts to reduce bias in the data used to train these models. The data used to train AI image tools often contains toxic content, including pornography, misogyny, violence, and bigotry, which leads to the generation of stereotypes in the AI-generated images. For example, AI image generators tend to depict Asian women as hypersexual, Africans as primitive, and Europeans as worldly. Efforts to detoxify the data have focused on filtering out problematic content, but this approach is not a comprehensive solution and can even exacerbate cultural bias. The AI field is divided on how to address bias, with some experts believing that computational solutions are limited and that understanding the limitations of AI models is crucial. Summary based on a draft from our day one template.

🗞️ Referenced in:

29. Dungeon Master's Helper

By David Colarusso. This webapp is meant as a handy tool for game masters (Dungeon Masters in the D&D world). It has tools to help folks run ability checks and the like. It has a number of beginners resources, including a glossary where much of the game play and many concepts are explained.

🌎 Referenced in:

30. A Clockwork Miracle

Radiolab Podcast. In this episode of the Radiolab podcast, Jad and Latif explore the legend of a clockwork miracle that took place in 1562. When the crown prince of Spain fell down a set of stairs and suffered a severe head wound, his father, King Philip II, turned to a relic and made a deal with God. If his son was saved, he promised to create a miracle of his own. With the help of a renowned clockmaker, the king fulfilled his promise by creating an intricate mechanical invention known as the monkbot. Jad and Latif visit the Smithsonian to learn more about this nearly 450-year-old creation. The episode was reported by Latif Nasser and features insights from Elizabeth King, a professor emerita at Virginia Commonwealth University. Summary based on a draft from our day one template.

🌎 Referenced in:
Mastodon | GitHub | Privacy | RSS Site by David Colarusso