What we do when cleaning court decision data, and why

Preprocessing and data cleaning

Following the previous blog post, we were able to turn images into text (OCR), including the documents that had no text layer. Now nothing can stop us from putting the huge dataset to use, right?

We could start searching the data right away, and in most cases we probably would not even notice the difference: there are so many decisions that we will immediately get some kind of seemingly usable result. But that is exactly the trap. If we look more closely into a great many decisions, or simply run various analyses on them with due care, we will see that many of the downloaded datasets are teeming with errors, and if we start using our system like this, sooner or later it will affect the results too.

That is why we need to talk about data cleaning, preprocessing, testing and evaluation. This blog post focuses in detail on data cleaning, and I also write briefly about the meaning of preprocessing (which is broader than data cleaning). It is certainly not an exciting topic, but without it we have no chance of understanding what data-driven legal work actually requires. Let us not forget that this is one of the most serious tasks of the publishers of legal software and legal databases; here I can only try to give a taste of it.

Let us treat preprocessing as the broader concept: its aim is to bring an available dataset, such as the text of decisions, into a form that suits a specific purpose as well as possible, e.g. searching for everyday legal work, or some kind of language modelling or analysis (e.g. analysing the case law on the amounts of damages or on the award of legal costs).¹ If my aim is to analyse the amounts of damages awarded, it does not matter if, say, Cyrillic or Greek text in Hungarian judgments is unreadable, but it does matter that numbers are written uniformly, with uniform decimal separators and uniform grouping of digits into thousands. On the other hand, if we want to reproduce exactly the “thoughts” of defendants quoted in judgments, we make a mistake if we correct even their obvious spelling errors. If we want to quote a decision with a reference to its page numbers, we cannot change its page layout either; but for analyses, page numbers and even line breaks are only a nuisance and reduce accuracy, because, for example, the machine will not recognise in a word broken across lines, such as “vád- | lott” (“defen- | dant”), that the sentence is actually about the defendant.

Data cleaning in general

If we want to keep the downloaded datasets as versatile as possible, we should not narrow our text down to one specific use. By contrast, the aim of data cleaning is simply to filter out the data errors, unjustified variations and missing data that would spoil the expected result. Data cleaning is therefore regarded as a separate step within the broader concept of preprocessing (alongside conversion, the integration of data from different sources, etc.), and its main aim is to eliminate the differences between data that are considered “unjustified” and the phenomena that are considered errors.

The spelling example above shows that even data cleaning cannot be done “just like that”, regardless of what we want to use the cleaned data for. For example, if we drop the original page layout of the text and delete the page numbers written into it, we will no longer be able to tell on which page of the original decision a provision we want to quote appeared. From another point of view, however, such page numbers, hard line breaks and page breaks are very disturbing when reading, whether the reader is a machine or a human.

In other cases, the data analysis may go beyond the typical questions lawyers ask, and the purpose of data cleaning has to take this into account as well. With the amount of data available, it is quite possible to mine from the BHGY decisions which templates individual courts shared with each other, or how the body publishing the BHGY changed the settings of its conversion or anonymisation software over time. We can gain such enormously useful insights as the fact that the number of hand-made decorative lines separating the structural parts of a judgment (e.g. the operative part, the ruling on legal costs) has fallen markedly in recent years. Earlier, ,-,-, and ~.~. were popular in Székesfehérvár, while */*/ was used in Tatabánya and -/-/ in Kaposvár. The most striking decorative line was used by a judge of the Miskolc District Court in two decisions in 2014 and 2015. In these, the Gujarati vowel “ru” (ઋ, Unicode U+0A8B) is lined up one after another, purely for decorative purposes, of course. We will probably never find out why, but it may well be nothing more than a chain of technical coincidences. (We also chose the courts’ decorative lines as the cover image of this post.)

When checking the data, we have to keep in mind that in a smaller dataset many errors never come to light: checking ten decisions naturally calls for entirely different methods and emphases than checking ten thousand or a hundred thousand.

Why is data cleaning necessary?

In the previous blog post we examined how we could extract text from decisions without a text layer. This mainly affects the dataset of contracts subject to notice (20% have no such layer), a few decisions of the NKFH (0.8%) and 5% of the expert opinions published by the SZTNH. With any OCRed text we have to expect character errors too, whether we ourselves or the publisher did the OCR (the BHGY decisions, for example, also contain OCRed characters, not only text extracted from the original source file). There are also errors typical of a given scanner, OCR program or scanning habit, e.g. spaces inside words or several words merged into one, confusion of the letters n/r, ö/ő and i/r, or of the “li” ligature and t, or the swap of ó and ő that is common with Canon copiers (“szerzódés” for “szerződés” (contract), “eladő” for “eladó” (seller)), and so on.

Conversions between file formats can also cause many serious errors. For example, until 2019 almost every decision in the BHGY was in the so-called RTF (Rich Text Format)², and since 2021 almost all of them have been in the .docx format used by Word,³ that is, OOXML. As a result, more than 68% of the decisions published so far have to be converted to text in a way that avoids serious data loss or distortion, which is not a trivial task even for the latest agentic frameworks. On top of this, the anonymisation tools⁴ also regularly caused minor data errors. György Görög and Péter Weisz already drew attention to this in 2019: the legal text sources they processed also suffered from scanning and OCR errors, and from serious data loss caused by overenthusiastic anonymisation.⁵

When converting the RTFs, we can draw the following lessons:

a) With the tool called pandoc, the contents of tables are lost, and it is also very slow, so for fast conversion of this kind it is worth building custom converters (which also take care of the individual errors mentioned below).

b) The decisions use many different characters, whether the accents typical of Hungarian or Greek and Cyrillic characters inserted into Hungarian text (we can even find, for example, a bilingual Ukrainian–Hungarian court decision in an immigration matter). In RTF, the encoding of characters is not as unambiguous as in .docx or in text files. Here, the characters used are marked according to various code pages, not according to the uniform Unicode standard, adopted in 1991 but spreading only gradually (which forms a single character table independent of the writing system). The code page can be specified for the whole text, for the fonts used and for individual characters, and the differences between these methods often cause problems. For example, if the whole text is marked as using the Western European character set (cp1252) while the font is set to Central European, but this latter setting is not recorded, the text will show õ with a tilde and û with a circumflex. Or the other way round: in Central European text, the Western European single character for one quarter (¼) becomes the Slovak L with a caron (Ľ), and only the context makes it clear which character should actually be used there. In more than 600 decisions, the § sign and the one-character “…” used for anonymisation were recorded in Japanese (Shift-JIS) character encoding. It can also happen that the typist pastes in a text that, invisibly to them, contains a character that marks Arabic text (together with a change of writing direction); this becomes visible only during conversion, and the accented letter turns into an Arabic letter (“… fizessen meg az لllamnak külِn …”). Finally, there are cases where only the shape of the printed character would make it clear that it was not that letter but some kind of graphic symbol, because, for example, the Wingdings font was used, and within it the right-pointing arrow became an R with a caron (Ř). Errors of this kind affected a few thousand decisions.

c) It happened that the original conversion program used by the OBH did not separate so-called control characters similar to the tab character (\tab, or \qj, the latter meaning justified paragraph alignment) from the following word, which produced text such as \qjDr, \qjBudapest and \tabSzentendrei. Neither Word nor any other application displays these, since no such control character exists. Such errors have to be taken into account during conversion. This affected 28% of the decisions, but in most cases it remained invisible (because, for example, nobody has so far asked whether the judge was a doctor or not).

By contrast, with PDFs⁶ we meet different kinds of errors: characters accidentally separated from each other inside words (“rendelk ezése” for “rendelkezése” (its provision)), because a PDF often “draws” even the text, and the positioning of the letters relative to each other can cause such errors.

Of course, we find not only conversion errors: the source text produced by the court can itself contain errors. For example, among all published decisions, the word “felepres” (for “felperes”, plaintiff) appears 2,381 times in 1,546 decisions, “konjuktív” (for “konjunktív”, conjunctive) 1,520 times in 1,251 files, and various derivatives of “felbújt” (for “felbujt”, to incite) in 923 decisions… It is an interesting question what we should consider, on the basis of the BHGY decisions, the established name of the drug methylenedioxymethamphetamine, when “extasy” appears 5,512 times and the otherwise correctly spelt “ecstasy” only 3,212 times. It is also important to bear in mind that there are many legal terms of art that a conventional spell checker marks as errors, such as “védirat” (statement of defence), “szúrcsatorna” (stab wound track) or “érintetti” (relating to the data subject).

Data errors in the individual datasets

Let us review the individual datasets in terms of the errors above (for the abbreviations, see the earlier blog post).

Page numbers, letter-spaced words and hard line breaks are discussed below, among the questions of normalisation.

How do we notice data errors?

When we come across a new text dataset, an important question is what methods we use to assess the errors of the text in front of us, whether or not we have to convert it. We can start looking for errors at the level of characters, words, lines or whole files (decisions), or we can group the text to be examined in advance by some known source or method, since similar text sources and technical operations tend to produce similar typical errors.

Ideally, we would have a reference text showing what a “correct” text should look like. Depending on the size and variety of the reference text, we can look for its characteristic features and try to derive general rules and features from them. At the character level, for example, this can be which characters occur in Hungarian texts, in what proportion and how often. Or what the typical word length and line length are. If a few texts in the dataset differ greatly from this, we take a closer look at them. We can then also build a large list of the stems of typical words, run the stemming (lemmatising) program on every word of the full text, and mark for further examination the decisions in which the share of words outside the dictionary is markedly higher than the average, or than usual for the given type of decision.

In the Hungarian literature we can find further interesting methods that help detect noise in datasets. One of them is a published character-based language model,⁷ which measures how unusual, how perplexing, words are in a given context, which can also help find text errors, though with limits. After surveying 10% of the BHGY, I found that the median perplexity of an ordinary line of text is 1.19, while the perplexity of decorative separator lines is one million, that of lines containing anonymisation labels 300–400, and gross OCR errors show values of only around 100.⁸ For this reason, the method is only moderately suited to filtering out typos and letter errors, especially if we first filter out (mask) the values that are high even in normal cases, e.g. case numbers and letter-spaced words. In the 10% sample, the average score weighted by the number of characters was 1.29, and two decisions stand out sharply: the 2011 judgment (or order?) of the Curia (Supreme Court) No. Kfv.II.37.605/2010/4., which publishes only the case number, and the criminal judgment No. 43.B.44/2022/45. of the Budapest-Capital Regional Court, which sentenced the accused to two years’ imprisonment for budget fraud and (perhaps as part of the punishment) devoted 950 pages of the judgment to listing the serial number and amount of every invoice the defendant had issued over three years without paying the VAT on it. In a later paper, the research group that published the character-based language model used this tool to compare versions produced by different OCR models: where the versions differed, the language model chose the solution that seemed more likely to be correct.⁹

This character-based LSTM model is not a cure for every data error, but it is a useful addition to the analysis of Hungarian legal texts. Running it on the whole BHGY (6 billion characters) takes about 60–130 hours on a home or office machine with a GPU, and running the full BHGY set takes several hours even on a very expensive H200.

Beyond data cleaning: normalisation, canonicalisation

We will always meet peculiarities where we also have to decide whether a change is desirable or not. For example, 77.5% of the decisions contain words letter-spaced with spaces, such as “Í T É L E T E T” (JUDGMENT) or “I N D O K O L Á S” (REASONS), and not necessarily applied consistently or without errors.

As a rule, we do not want to analyse which typist, clerk or judge can properly use basic Word functions such as page breaks or centring, instead of solving the problem with spaces and frequent presses of the Enter key. In a dataset like this, letter-spacing no longer serves its original visual purpose, but it makes further automation considerably harder: for example, word recognition will often be wrong (“I n d o k o 1 á s”), spell checkers do not work on it, and so on.

That is why it is generally desirable to delete superfluous characters such as leading, trailing or double spaces, line breaks meant to “set” the spacing between lines and paragraphs, tabs, and the NUL characters at the end of files (which only take up space needlessly, and still affect nearly 5% of BHGY decisions).

Another such question is how to handle soft hyphens, page numbers built into the text, header information (the case number), and hard line breaks at the end of every line.

Individual normalisations also aim to support automation: here a punctuation mark or a structural unit of the text is brought into a canonical (normal) form. For example, they ensure that quotation marks are used according to one rule throughout the dataset (straight quotes, the low “99” „ and the high “66” “ or high “99” quotes, the guillemets « », etc.), or that Unicode characters that can also be composed of two marks are represented by a single character.¹⁰ László Laki and his co-authors prepared their text with similar steps, i.e. by deleting soft hyphens, joining words split at line ends, unifying quotation marks and merging multiple spaces, when they tried to correct OCR errors with neural models.¹¹

Also looking at the BHGY decisions, we can see that the marking of anonymised data is very heterogeneous: tanú1 or tanu1 (witness1), név4 (name4), cégnév1 (company name1), személy5 (person5), helység1 (locality1), telefonszám1 (telephone number1), but dots of various lengths and framing may also have been used (“….”, “……”, “(…)”), or initials marked by letters (“XY”, “X. Y.”, “A. B.”), or “xxxxx”. Quite often various suffixes were left in place (e.g. “Ivettel” (with Ivett) becomes “nevetel” (with name)). All this makes automatic processing harder, e.g. identifying the role of individual words in sentences, or identifying the roles in the text and the further analyses based on them (so-called “Named Entity Recognition”). The legal and technical difficulties of anonymisation, also with Hungarian court examples, were summarised by Gergely Csányi and his co-authors.¹²

Without the cleaning and normalisation above, splitting the text into grammatical sentences and analysing it will also be harder and of poorer quality. A line of text can of course contain several sentences; it is just not so easy to tell sentence-final full stops from the various known everyday and legal abbreviations, and capital letters do not provide a reliable solution either. Several authors in the Hungarian literature have already examined the task of splitting court decisions into sentences; I must highlight the thorough work of Csányi and his co-authors.¹³

Summary

We can see that thorough, high-quality preprocessing of the data is a precondition of all further legal work as well. If the scanned text, the line breaks and the sentence splitting, or the normalisation of anonymised forms are less reliable, this unfortunately also makes the automation of legal work harder and worsens its results.

If we set about data cleaning ourselves with the help of machines, we should take care not to delete the original text until the final version of the cleaning has been approved, because we will have to go back to earlier versions countless times when it turns out that we made a wrong decision. Following the guidance above, let us understand what kind of text the acquired dataset consists of, in what formats it is available, what character sets it uses and what kind of text to expect. Let us plan the conversion and the error search as multi-round operations, and always try to trace the cause of the errors we find. With several hundred thousand decisions, manual data cleaning and manual normalisation are simply not an option.

Data cleaning also matters because it lets us get to know the characteristics and types of the dataset better without having to read all two hundred thousand decisions. We can find the longest and the shortest judgments, and we can come across Easter eggs and odd ones out hidden in the BHGY, such as the 7,000 administrative decisions (decisions of authorities reviewed by the court and published as separate files next to the court decision), or personal data left in despite anonymisation.

Thanks to data cleaning, however, we can now confidently set out on the next part: with what methods and under what conditions we can search legal texts effectively (“information retrieval for lawyers”).

I have published the tools used for data cleaning, a general technical description of the steps and the reviewed word lists on GitHub. These include, for example, 359 words that are correct in legal texts but unknown to the Hunspell spell checker, 846 abbreviations found in court decisions with their meanings, and 127 common typos and OCR errors with their corrections. The downloaded and cleaned decisions and the scores from the character-based language model are not published.