What does the future hold for the role of AI in the delivery of legal services?
Keeping up with the LLMs
Will there 'come a point where no job is needed'?
Elon Musk, UK AI Summit
The improvement in LLM technology over the past 24 months has been incredible. The October 2023 LinksAI English Law Benchmark said that one of the problems in fairly marking these questions is overcoming the astonishment that they can provide any answer at all.
In the few short months since that report*, the world has become much more used to the idea that LLMs are capable of producing coherent answers. In fact, what we found astonishing were the improvements compared with the results reported in the LinksAI publication. We have already seen major new versions of several of these LLMs released (including LLaMa 3, released on 4 March 2024, Gemini 1.5, released for public preview on 9 April 2024, Claude 3, released on 18 April 2024 and GPT-4o released on 13 May 2024). Whether the next 24 months will see the same advancement is an open question.
It is possible there are inherent limitations to general LLMs, due to the mismatch between probabilistic natural language processing and the nature of legal reasoning. However, even if the capabilities of LLMs plateau, fine-tuning and training of LLMs are likely to continue to deliver performance improvements. We can't wait to reapply the Allens AI Australian law benchmark to significant future iterations of this technology and update this report. If the improvement in LLMs continues at its current rate, future editions of this report will be interesting.
Speciality models and 'hybrid search'
While our benchmark focuses on AI playing the role of legal advisor, many products coming onto the market now try to enlist AI in different roles to assist lawyers. For example, a number of providers are now developing and launching models specifically trained on vetted legal content (such as databases of cases, law and guidance) and designed to only base their conclusions on that content – thus splitting the language model from the knowledge base. Perhaps this is the way of the future. The citation problem could also be addressed by 'hybrid search' models, which deliver value by not only providing an answer but linking directly to vetted sources to substantiate it.
It may be that lawyers will be more likely to use one of several AI tools that are specifically developed to provide legal advice. Some tools that are currently entering the market are AI-powered 'legal assistants' designed to perform very specific legal tasks, such as summarising documents or searching databases, but there are some advice-orientated online tools. For example, AskAILawyer.com is a website providing free legal advice via AI-powered chatbots. The service provides different AI chatbots for different areas of law, including tax, family, insurance and personal injury. The legal jurisdiction can be set to a range of different countries, allowing the chatbot to provide responses based upon differing international laws.
Lawyers are still needed
Finally, it's worth remembering that, even if future generations of LLMs do eventually beat the Allens AI Australian law benchmark, there is more to the role of lawyers in the delivery of legal services. The ability to answer questions of law in a succinct and correct manner is but a fraction of what is required in the daily travail of an Australian lawyer.
Our benchmark question set reflects the questions asked of mid-level lawyers by senior lawyers. These are not the kinds of questions that businesspeople ask when they call their law firms. Instead, they often have a broad concern, or a business objective they want to achieve. An understanding of the law and an ability to apply it are two vastly different skill sets, the latter requiring a profound understanding both of the law and the business context in which it's being applied. Breaking the client’s requirements down into a series of legal steps that will achieve their aim with the minimum effort, expense and uncertainty is the interesting and creative part of being a lawyer. Answering nutshell questions is the easy bit.
* We ran the benchmarking questions through the LLMs in February 2024.