Document Review: The Order of Batched Materials Matters
By Nicholas Cole
The order in which documents are reviewed plays a significant role in determining how quickly a reviewer can review documents and the overall costs to clients. In fact, it may be the most important factor to consider at the onset of a contract review. When documents are grouped by subject matter, document review speed tends to increase thereby reducing client cost. Below is an anecdote of my experience supporting this theory.
On a recent document review, with extremely tight deadlines, a team of contract reviewers was hired to perform the initial review and make a first cut as to document relevance. The Litigation Support Project Manager (“PM”) was tasked with batching out documents for the contract attorneys to review. The legal team decided to serve the documents to the reviewers in batches ordered chronologically. This is the most common way documents are batched for document review. Legal teams prefer this order because it is easily understood and legal teams think it provides the reviewers with a timeline and context to better understand the facts of a case. With the review underway, the PM began reporting some alarming reviewer statistics to the legal team. Review rates for each reviewer were below 30 documents per hour rather than the hoped-for industry standard of 50-60 documents per hour. At the initial rate, the completion date for the first level review was projected out to a date way beyond our deadline and the budget for the contract review alone would consume the budget for the entire matter. The reviewers needed to go faster. This led to a detailed investigation measuring the quality of the reviewers, the types of documents being reviewed and development of potential solutions to accelerate the rate of the review.
First, the PM analyzed the types of documents that had been batched for review to determine if large complex documents were causing the slower review rate. The document collection did contain an unusually large amount of complex, lengthy documents and, unfortunately, there is no solution for changing the types of documents that may be responsive to discovery requests. However, understanding our document collection make up allowed us to effectively analyze our reviewers’ progress. The PM compared the quality of the reviewers to determine if we had any disparity among their review rates. The initial results showed that some reviewers were reviewing at a much faster rate than other reviewers. However, when the document types and sizes were taken into account, the differences in review rates virtually disappeared. There were reviewers that had been assessed as ‘slow’ when in fact they just had more complex documents to review in their batches. Recruiting new reviewers and/or re-training was not going to improve our review pace, which was the team’s initial inclination in order to meet the deadlines.
In fact, new reviewers were more likely to slow us down. Any new reviewers would need time to familiarize themselves with the types of documents and case issues. Given that we could not change our document composition and we had no “bad” reviewers to replace, we tried highlighting key terms in our document collection to focus the reviewers on key portions of our documents thereby improving document review rates. Unfortunately, highlighting key terms did not accelerate review rates either. The review had been underway for just over two weeks, deadlines were looming and the team was in desperate need of a solution.
Second, after a lengthy meeting with the legal team, the PM convinced them to experiment and apply an imposed conceptual ‘sort order’ to the documents to determine how that would affect review rates. The PM created revised document review batches that had been ordered by email thread/chain and then the chains were grouped conceptually. Half the reviewers were assigned to complete the reordered batches and the remaining reviewers stayed the course with the original batches of chronologically-ordered documents. After two days, a comparison of rates between the two review groups was completed. The contract reviewers mining the chronologically-ordered batches still had review rates around 30 documents per hour. The reviewers completing the conceptually-ordered batches had review rates over 60 documents per hour. Applying the conceptual sort order to the review batches doubled reviewer efficiency. The reviewers, in effect, read faster and made quicker coding decisions because they were reviewing documents related to a single topic. They did not have to pause and think about how every topic related to the issues in the case and hop around the issue coding form.
The reviewers themselves also confirmed that conceptually-ordered batches increased proficiency. When the reviewers of the conceptually-ordered batches were interviewed, they said they were able to review faster because they only had to think about one topic at a time rather than all the topics related to the matter. They did not have to refer back to the coding/ background manual to refresh their memory on a particular subject or issue-coding nuance. The revised sort order put the team back on track for meeting our deadlines and budgetary restrictions. The client was being charged hourly for the contract reviewers, and revamping the order of how documents were being reviewed effectively halved the per document cost. Additionally, the second level review overturn rates had not increased, which indicated that the revised sort order had not negatively affected the error rate of the first level reviewers. In conclusion, when beginning a contract review, there are several considerations that the overseeing team should consider. First, highlighting has little effect on review rates. Whether a decision is made to forgo highlighting of key terms or designating numerous terms for highlighting, none of the changes made a measurable difference in review rates for the case above. Second, understand the types of documents that make up your document collection so you can accurately assess the quality of your reviewers and have realistic expectations about document review rates. If the documents in your collection are comprised of emails versus standalone documents, reviewing email strings will provide the context that chronological review sets out to accomplish but will accelerate review rates substantially. Finally, work with litigation support personnel to determine the types of documents that exist in your collection and determine what litigation tools are available to help accelerate the review process, or in some cases, whether a traditional, linear review is even necessary.