Technology

Benchmarks And Outcomes -- Moneyball For GenAI (Part II)

For a legal innovator, it is important to understand what success look likes for the larger organization.

By  
on

moneyballIt’s been just over a month since the former Oakland A’s beat the Texas Rangers in a 5-2 victory that closed out their tenure at the Oakland Coliseum — and indeed the city of Oakland itself. The A’s have announced plans to build a stadium in Las Vegas by spring 2028. If all goes accordingly, this could serve as an opportunity to maximize the value of the franchise while eliminating the challenge of attracting star talent to a small baseball market.

The A’s featured heavily in last month’s column too as I introduced the importance of benchmarking and selecting the appropriate metrics to ensure the outcomes and goals desired are achieved using the movie “Moneyball” for guidance. As depicted in that movie, former Oakland A’s manager Billy Beane embraced a different strategy to get around the small market problem: Measure the ability to get on base to ensure enough runs were scored to win enough games to make the playoffs. Of course what was once a competitive advantage is now known by all.

But before any tool — be it analytics or GenAI — can be put to good use, you need to understand your own goals. This will greatly inform the tactics and resources utilized to close the distance between objective and outcome.

What Represents Success?

For law departments, success may be supporting an aggressive corporation’s acquisition strategy. For highly regulated businesses, success may be compliance with increasing regulations or staying out of controversy. And for stable businesses, the focus may simply be on efficiency, productivity, or cost management. Different companies pursue different strategies and those strategies should inform the goals of the law department.

Goals for law firms may vary too. One firm may focus purely on partner profits.  Another may focus on growth and the acquisition of new clients. A small firm may be looking to be acquired. And a personal injury firm may look to increase the percentage of clients that receive a monetary award so their reputation grows and their marketing is more effective.

For a legal innovator, it is important to understand what success look likes for the larger organization.

Sponsored

Apply GenAI To Support Success

Baseball is a sport with limited change and well over a hundred years of metrics and analytics available. But we are in the very early days of the GenAI revolution. There aren’t hundreds of tried-and-true metrics — that is a struggle for legal innovators right now. To extend the baseball analogy, we are in the early innings of one of the first games of the season. So recommended KPIs and guidance on metrics are likely to change as the industry learns and technology matures.

Selecting and prioritizing the best projects and solutions to support the organization’s goals is important. Just because there is a new bright shiny object doesn’t mean that it supports the goals of the organization.

Shohei Ohtani is one of the most electric baseball players in modern history. He can pitch and he can hit the ball. He has 50 stolen bases and 50 home runs this year and helped the Dodgers clinch the 2024 World Series. Any baseball team would love to have him on their roster, but does his production map to success at any team? Probably, as Ohtani is a once-in-a-generation player. But what about the next tier of really solid players? It’s more like “does the baseball glove fit the fielder’s hand.” If your problem is in the outfield, you need an outfielder and not a catcher.

In the same way, selecting the proper solutions to support goals is important. They must align to the goals for success. If the goal of a corporation is to accelerate sales growth, then the law department might want to look at solutions that shorten sales cycles during contracting rather than the latest e-discovery tool. And for a law firm focused on client satisfaction, solutions that make the firm more responsive or that create better work product may be more important than solutions that simply reduce nonbillable costs or that improve productivity.

Sponsored

Understanding what solutions support success for your organization is essential.

Selecting KPIs For Solutions

Productivity metrics tend to be more measurable. Does a GenAI research tool save time? How much? The greater challenge is to understand whether a better result is achieved. What if a research tool could help discover a new legal argument? Or a new technique to comply with regulations?

Understanding qualitative metrics is important. Winning a court case or finding a creative way to settle a dispute may have more impact and value than the productivity benefits of a solution.

For any new solution, it is also essential to understand the metrics of the baseline. What if a firm would apply the same standard for hallucinations used in the Stanford study to first- or second-year associates using traditional legal research solutions? Recall that the Stanford study defined a hallucination as an answer that did not accurately reflect the current state of the law even if it was mostly correct. Perhaps GenAI legal research solutions might be even more attractive if the rate of hallucinations were understood.

GenAI has spawned a slew of new document drafting solutions. Productivity metrics like reductions in drafting effort, shorter cycles to complete a document, or fewer turns in a negotiation may be the absolute best metrics for document drafting. But it is also important to note that some solutions are genuinely hard to measure and document drafting is a great example.

How does someone decide if a clause in a contract is better and measure that objectively? It’s possible to benchmark against “market” for some clauses, like those that limit liability. But I’ve yet to hear anyone come up with a framework for evaluating whether a contract is better and produces a better outcome or reduces risk.

Assessing outcomes for GenAI solutions can be challenging given that there isn’t yet a strong body of data to help legal professionals benchmark performance.  Start with what success looks like for your organization. Understand what your baseline metrics are for current processes. If you don’t know, then measure them.  Prioritize and map new solutions against organizational success. And consider the importance of quantitative and qualitative metrics when evaluating solutions.

It’s truly an exciting time to be working in legal technology!

Ken Crutchfield HeadshotKen Crutchfield is Vice President and General Manager of Legal Markets at Wolters Kluwer Legal & Regulatory U.S., a leading provider of information, business intelligence, regulatory and legal workflow solutions. Ken has more than three decades of experience as a leader in information and software solutions across industries. He can be reached at [email protected].

Topics

, , , ,