Judging Wikipedia’s content

In an idealized vision of the world, judges in common law countries are unbiased actors who rely on expert knowledge and detailed research to craft rulings which accurately reflect both the substance of the written law, and the body of precedent that applies to their jurisdiction. A judge must be proficient at doing their research and must consult all relevant ruling. Ideals aside, judges are human, and generally have heavy caseloads. Like the rest of us, they’re likely to rely on tools to ease their way through the research process. We live in a world where everyone relies on Wikipedia, regardless of whether they admit it. And according to research recently published by Neil Thompson and colleagues, judges are just like everyone else in this regard.

Back in 2019 I wrote about a study by Neil Thompson and Douglas Hanley which suggested that Wikipedia content helped shape the scientists’ understanding of their own field of study. They showed that the language used in the technical literature in chemistry converged with the wording used in Wikipedia articles about a given topic. Through a fascinating study, they were able to demonstrate experimentally that this wasn’t coincidental — the way topics are discussed on Wikipedia influences the way they’re discussed in the literature. The fact that Wikipedia articles influence the way people understand their own fields highlights the importance of experts getting involved in the process of editing articles.

In a new study of Irish legal cases, Neil Thompson and colleagues were able to show that judges rely on Wikipedia articles to inform them about settled cases and precedents, and concluded that judges are relying on Wikipedia as a replacement for their own reading of Supreme Court rulings.

Much like in the previous study, the researchers created 154 new articles about Irish Supreme Court cases, and uploaded half of them to Wikipedia, while keeping the other half as a control set. Most of these articles were created by law students with the support and supervision of faculty (using a methodology based on Wiki Education’s Student Program). Wikipedia’s coverage of Irish Supreme Court cases was very incomplete, which meant that it was easy to create new articles about cases where none existed previously.

What happened next was probably not a huge surprise — creating a Wikipedia article about a case increased its rate of citation in rulings by almost 22%. While this showed that judges are relying on internet searches to locate relevant cases, it said little about how they are using the information on Wikipedia. But the second part of the study looked at the textual similarity between rulings and the Wikipedia articles. Here again, they found a statistically significant effect. In other words, judges (or, perhaps, their law clerks) were paraphrasing Wikipedia articles as they drafted their rulings.

The implications of this study are pretty major. While the best Wikipedia articles provide accurate, comprehensive, unbiased coverage of a topic, most fall short in one area or another. This is rarely intentional — while Wikipedia’s contributors are usually dedicated to producing high-quality articles, they’re mostly volunteers who face constraints of time, access to sources, and sometimes subject-matter expertise. But Wikipedia’s open nature also means that people with vested interests in the outcomes of a case have the ability to manipulate articles about important precedents.

To avoid these sorts of problems, Thompson and colleagues suggest ways to improve the quality of Wikipedia articles: “Policy-wise, this could be addressed by buttressing the reliability and review of Wikipedia content by including legal professionals as supervising editors to certify page quality, or by augmenting the content of authoritative but less-broad sources, and using those for the provision of legal information about particular jurisdictions.”

These are reasonable suggestions, but they’re also ones that the community has tried without much success throughout Wikipedia’s existence. It’s hard to convince experts to dedicate their limited time to reviewing Wikipedia articles. It can also be difficult for experts to work with the Wikipedia editing community, especially when outside experts don’t have a good sense of the community. (Despite the frequent assumption that Wikipedians are just random amateurs, many have advanced degrees in the subject areas where they contribute, while others have become experts while contributing over the last two decades.)

Short of convincing judges not to use Wikipedia, there are other ways to mitigate some of these problems. The more active editors there are in a subject area, the harder it is to insert bias. People pay more attention to changes to existing articles, especially if they are actively being edited. It’s much harder to insert bias into existing articles than it is to do it when you’re creating a brand new article. Programs that bring more contributors to Wikipedia — like Wiki Education’s Student Program — not only can fill content gaps in legal topic areas, they also bring more traffic and more editorial attention to these articles (and to articles that are downstream from them). After all, the articles that Thompson and colleagues used for this study were mostly created by student editors.

The other way to mitigate potential harm is to make people better consumers of information from Wikipedia. Few people who consult Wikipedia articles ever look at the history tab or the talk page, despite the fact that they can provide crucial information about the state of the article. Even fewer know about plug-ins like “Who Wrote That?“ that supply information about when individual “facts” were added (and by whom). Training judges (or the pool of legal professionals from which judges are appointed) would make them better consumers of Wikipedia. This isn’t a far-fetched idea — the model for this kind of thing exists in our Scholars & Scientists Program, where participants gain these kinds of skills (among others).

As Thompson’s research has shown, Wikipedia is influential on multiple disciplines. If you’re interested in influencing the public’s understanding of your topic area, as well as future ways of writing about your subject area, adding neutral, fact-based information to Wikipedia is the way to go. Instructors who are interested in teaching with Wikipedia, visit teach.wikiedu.org for more information on Wiki Education’s support for assignments. Knowledge or disciplinary organizations, empower your staff or members to improve Wikipedia’s coverage of your topic by hosting a Wikipedia editing course.

Thumbnail image by Blogtrepreneur, (CC BY 2.0) via Wikimedia Commons