A heuristics based ML-in-the-loop data extraction pipeline for Indian legal data

This is the story of how I (as a lawyer turned programmer) built Indian Contracts in Adjudicated Texts ("ICAT") as a free and open source software pipeline for extracting data from Indian judgments, and how I explored the data that was extracted to build iteratively.

Background:

1. In this talk, I will first explain how legal data is often the most regressively barricaded "digital commons" asset there is - due to various parties (publishers, online databases, etc.) creating privatised annotations.

2. In this landscape, my effort was unique because I wanted to get out of the limitations of imagining tech (read, LLMs) only as a way to productionise user-facing applications. I wanted to go beyond this and create something more foundational through FOSS, which I am doing iteratively (as described below).

How it started:

1. I noticed that there are patterns which occur in Indian judgments, which was replicated even in copies of those served by the IndianKanoon API. A contract clause, a citation, a definition, these were all things that were indented and hence available within the relevant html tags when served by IndianKanoon.

2. I began using the help of freelancers to annotate the data that a simple text-classifier model would extract. How this works was given in more detail in a blog post here - https://schemati.cc/ICAT

3. Hence, after collecting around 60 judgments I had made ICAT v1, an open source pipeline on HuggingFace that was accompanied by a dataset release and an exploratory notebook for a growing sampleset of data.

Learning from exploratory data analysis:

1. I used tools such as OpenNyAI's Named Entity Recognition Model and BERTTopic's Topic modelling to explore the data and found that most clauses coalesced around a few industries and laws.

2. Contracts have specific language that can be used alongside industry specific small language models for better analysis/Topic Modelling.

3. LLM's are useful for their larger context windows, whereas determinatively extracting the contract text is something that cannot be trusted to them due to the hallucination (and foreseeably, the semantic similarity in a generalised embedding vector).

Hence, I iterated again from my learnings

1. This is an end-to-end pipeline that is completely automated. It utilises a publicly available text-classification model for classifying html tagged indents as before. But also utilises LLM's to extract the full contract text as formatting errors could lead to lack of html tag boundaries.

2. Using the industry specific small language model for exploratory data analysis currently, with a public notebook that currently demonstrates how industry specific embeddings from LLMWare give the ability to search for contractual clauses.

Why is the dataset in "beta"? In the future, I plan to build not just a contractual clause search feature but also show how it can help in better drafting. For this purpose, I will be reviewing and releasing the data extracted with insights as I continue to iterate.

Understanding the dataset you work with

A large part of why I could build this was because I had an idea about Indian judgments and patterns in them, being a law graduate. Using exploratory data analysis also helped in building iteratively. But I had to know what I was looking for in many cases - hence, studying the subject matter proved effective.

Resource efficiency as a guiding principle

Building using ML-in-the-loop after annotating data by experts helps avoid many of the errors that LLM's are bound to make. Even as a non-tech graduate, I was able to build a simple text-classifier model (using huggingface's "autotrain") that does much of the heavy lifting.

Strategising an open-source data pipeline (and it's CC licensed data)

There is still a lot that can be built on top of this, but by building in the open I have shown how over the course of 2+ years a product can grow while you can learn through iterative development - in my case this is the example building for local use first, releasing what you have to gauge impact, and many more strategic encounters as I used this dataset internally.

Also, as suggested, I will talk more about how the project is novel considering the landscape of legal data - the nuances of how "laws" are public but often legal data is safeguarded.