Discovery quality management
Discovery insights: Five questions
Establishing and maintaining a quality process during discovery activities is critical, particularly given the volume of data, variety of tools and differentiation of recognized practices that can apply. The eDiscovery industry has not defined a baseline for quality and, as such, multiple competing quality initiatives are underway. It is important to understand the impact of such quality initiatives on purchasers as well as the industry supply chain.
- eDiscovery process ISO/IEC 27050
- Quality management
- Discovery providers
- Actions required
An interview with Chris Knox, director and Steve Shebest, director— Deloitte Discovery, Deloitte Transactions and Business Analytics LLP
eDiscovery process ISO/IEC 27050
The International Organization for Standardization (ISO) recently gave its final approval for the development of an international standard relating to the use of Electronically Stored Information (ESI) during the eDiscovery process (ISO/IEC 27050). Its passage has heightened awareness of the lack of industry standards related to ESI and eDiscovery.
Whether the standard comes to fruition or not, ISO/IEC 27050–like other ISO initiatives–has invigorated interest and has sparked a discussion of safe, reliable processes that minimize error and reduce cost. This is the definition of quality of process.
What was the recent action by the International Organization for Standardization and how might it impact discovery?
Yes. The ISO 27050 initiative is one of several nods to the importance of quality of process in the eDiscovery industry. There have been efforts by various industry participants to address the lack of a quality benchmark by utilizing or encouraging not only ISO frameworks 270011 and 90012, but also other recognized quality-related systems including AICPA’s Service Organizational Control (“SOC”), Six Sigma and the Project Management Institute’s PMP structure. The Sedona Conference’s Commentary on Achieving Quality in the E-Discovery Process spoke to a need that each of these frameworks attempts to address: How do we apply the fundamentals of quality management to a set of processes such as eDiscovery?
Are there other initiatives in this area?
Quality management requires a systematic approach to quality assurance, control and improvement. Quality assurance (“QA”) means having checks in place to ensure each of your processes are executed correctly. Quality control (“QC”) validates the quality assurance effort by measuring the end result of your processes. Quality improvement, particularly continuous quality improvement, is the discipline of taking the variance from your expected results found either at the QA or QC stages and revising the appropriate process to minimize that risk of variance moving forward.
How do you attack variance of process?
Galileo is credited with admonishing us to count what is countable, measure what is measurable and to make measurable that which isn’t already measurable. Likewise, Peter Drucker suggested that if you can’t measure it, you can’t manage it. Six Sigma’s DMAIC framework proposes that we Define, Measure, Analyze, Improve and then Control.
There is a theme here worth acknowledging. For example, how do you know that the document reviewers understand the coding manual if you don’t audit their coding, particularly right after the training? Is one contract attorney moving significantly slower through the review sets due to a lack of effort or being stuck with 30-tab excel spreadsheets or very large files that take minutes for each to load to the review pane? Does each document in your production set coded as “redacted” actually have a redaction applied somewhere? Do your “exception” documents really constitute a valid exception or was there a correctable issue with processing? Ho w accurate are your search terms and how many “false positives” are they generating, resulting in additional review cost?
All of the various means to measure and identify outliers or variances from expectations will ideally roll up into a documented process for overall quality management. All of the systems and frameworks mentioned above are means to the end of framing that quality of process.
How do the fundamentals of quality management apply to electronic discovery?
The electronic discovery service provider market is a fractured one, with local, regional, national and global players. Further, providers differ in the scope of offered services across the spectrum from identification and collection, to processing and hosting to document review. Finally, providers utilize different technological tools with varying capabilities and limitations. All of this can contribute to an array of different risks and experiences for the same basic set of discovery requirements.
Due to the fractured marketplace and lack of an industry benchmark, service providers have sought to differentiate themselves in numerous ways, many of them related to quality. Some have achieved quality of process through audit regimen and/or certification in other areas, e.g. ISO 9001 or AICPA SOC 2. Others have endeavored to bolster their commitment to quality by ensuring that project personnel have a grounding or certification in a relevant discipline such as Six Sigma or the Project Management Institute’s PMP. Finally, others have eschewed formal frameworks but built in rigorous quality audits such as statistical sampling to verify processes.
How have providers of discovery services sought to distinguish themselves within the quality environment?
When vetting a service provider, ask the tough questions related to their Quality Management processes. For example, you may want to determine:
- If the provider has a documented plan in place to assure quality during each phase of collection, processing, review and production.
- If their approach is largely reactive, based just on the sampling of final results or more proactive through quality checks built into each sub-process.
- How often, if at all, are their processes audited and by whom.
- If gaps or deficiencies identified by audit or internal checks are rolled into any form of continuous improvement discipline.
- If the provider can demonstrate documented management commitment to quality throughout the organization.
- What metrics the provider can share with respect to the any of the quality measures they have in place.
Finally, even after you have selected your provider, consider applying a disciplined audit of the resulting work product. Have a plan tied to either your existing internal approaches to quality or expectations and previous pain points related to the eDiscovery process.
In the absence of an existing, independent benchmark for quality of process, what actions can a purchaser of discovery services take to proactively manage the quality of electronic discovery obligations?
Chris and Steve’s take
Quality management is something that can and should, begin at home as the industry matures and promulgates objective benchmarks for measuring the practices, partners and protocols of the discovery process. The recent developments on the international stage serve to underline not only the importance of understanding the fundamentals of quality management but also having a plan in place prior to entering the discovery process. As John Ruskin once said, “quality is never an accident; it is always the result of intelligent effort.” As the cost and intricacy of discovery of electronically stored information increases, the need for consistency and defensibility of process does as well. Working with your discovery team to understand how they are managing quality is a critical, yet often overlooked, key to success.
1ISO 27001 – sets out the requirements for an information security management system
2ISO 9001 - sets out the requirements for a quality management system