FDA’s proposed guidance for quality metrics raises questions about quantifying the tangibles and intangibles of quality culture.
By Andrew Harrison, Susan J. Schniepp
The long awaited, anxiously anticipated FDA guidance on quality metrics was finally distributed for comment on July 28, 2015. The official title of this guidance for industry is Request for Quality Metrics, Guidance for Industry (1) and its potential release has been looming on the horizon since 2012.
The intent of FDA to establish quality metrics first emerged in 2012 when Congress passed the Food and Drug Administration Safety and Innovation Act (FDASIA) enhancing FDA’s capability to proactively react to, prevent, and alleviate drug shortages. Specifically, Title VII Section 705 of the Act states FDA “shall inspect establishments described in paragraph [1] that are engaged in the manufacture, preparation, propagation, compounding, or processing of a drug or drugs (referred to in this subsection as ‘drug establishments’) in accordance with a risk-based schedule established by the Secretary.” Section 706 of the same act allows FDA to request certain information from companies in advance of or in lieu of inspections by stating, “Any records or other information that the Secretary may inspect under this section from a person that owns or operates an establishment that is engaged in the manufacture, preparation, propagation, compounding, or processing of a drug shall, upon the request of the Secretary, be provided to the Secretary by such person, in advance of or in lieu of an inspection…” (2).
In the Feb. 12, 2013 Federal Register Notice (3), FDA asked the industry to “assist the Food and Drug Administration in drafting a strategic plan on drug shortages as required by the Food and Drug Administration Safety and Innovation Act…” This notice asked a series of thought-provoking questions including “What metrics do manufacturers currently use to monitor production quality?” and “How frequently would such metrics need to be updated to be meaningful?”
After a few years of actively engaging and listening to industry in a variety of venues, this new guideline has finally been released.
The metrics proposed in the guideline are not new to the pharma industry. Many of them are currently being used by companies to internally measure performance. In some cases, the specified metrics are also reported to the agency via the annual report or are contained in the annual product review. The bio/pharma industry needs to review these metrics and ensure they will provide meaningful data while avoiding unintended consequences.
Defining quality culture
The underlying and understated tenet used to determine a company’s well-being is a measure of their quality culture. The culture of a company dictates the veracity of their metrics. The best way to ensure the data reported has merit is to assess the quality culture of the submitting organization. It is in this area that the new guidance lacks clarity. The guidance leaves the opportunity open to establish quality-culture metrics by stating, “these metrics are not intended to be an all-inclusive set of the quality metrics that FDA could consider useful to assess a product and manufacturer’s state of quality. For example, senior management commitment to quality is an important factor in evaluating the overall health of the PQS [pharmaceutical quality system] and quality culture” (1) and “…the Agency is committed to a dialog with industry to consider benchmarks and standards that could provide acceptable metrics that specifically demonstrate senior management’s commitment to a culture of quality …” (1). This commitment to establishing quality culture metrics is further evidenced by the section in the guideline titled “Optional Metrics Related to Quality Culture and Process Capability/Performance”. In this section, FDA “acknowledges the importance of quality culture to the overall state of quality of the product, process, and commitment to quality” (1).
Metrics related to quality culture
FDA proposes three voluntary metrics to try to get at the elusive quality culture. The first optional metric proposed is intended to measure senior management engagement by assessing whether the head of the quality unit and the head of the operations unit have signed the annual product review (APR) or product quality review (PQR). The second optional metric proposed is corrective action and preventive action (CAPA) effectiveness. The measurement for this metric is to indicate the percentage of corrective actions that required retraining of personnel, the assumption being that the root cause of the original deviation (real or due to insufficient analysis) was determined to be insufficient or ineffectual training. The third proposed metric is intended to measure a firm’s process capabilities through a series of three questions. The real question should be if these three optional metrics, taken together, shed any light on the quality culture.
Management engagement. Achieving a quality culture requires management and employees to establish an environment where responsibility, accountability, and reliability are paramount, and to understand the role each person performs in delivering a high-quality product to the customer and sustaining that performance on a continual basis. Management must educate employees and provide the tools and environment where they can perform their functions in an atmosphere that encourages excellence and continuous improvement. Assigning the head of quality and the head of operations the task of signing the APR or the PQR does not ensure management engagement nor does it mean that the quality culture is lacking. It is up to an organization to establish the appropriate level of responsibility and signing authority for APR and PQRs. It is up to senior management to provide the people charged with these activities the necessary resources to complete the task in a timely manner with the expectation that they will be held accountable for the contents.
Retraining personnel. The second optional quality culture metric is specific to CAPA. The proposed metric is to report the percentages of corrective actions involving the retraining of personnel. Without context supporting the retraining of personnel, this metric does not offer insight into the true culture of an organization. It could be argued that any CAPA that results in a reduction or elimination of a recurring deviation would require an element of training personnel. In fact, retraining of personnel on the CAPA issue, how it was solved, and how to implement the necessary change is evidence of management engagement. It should be expected that a majority of CAPAs involve some retraining of personnel.
Critical quality attributes. The third quality-culture optional metrics involves trying to use critical quality attributes (CQA) as a key indicator of a quality culture. Of the three optional metrics proposed, this one does provide some measurement of the existence of a quality culture. On the surface, the questions just seem to be a regurgitation of information contained in the APR or PQR. Upon closer evaluation, however, it is clear that FDA is trying to measure whether a company drives for continuous improvement through their review and assessment of threshold levels established with CQAs. Companies that have established CQAs and linked them to a requirement to issue a CAPA when they exceed the established threshold levels have demonstrated a commitment to continuous improvement. Continuous improvement programs are, in fact, reliable indicators of the presence of a quality culture.
Measuring intangibles
The establishment of simple quality metrics that not only measure the quality of the product but also reflect the quality culture of an organization is required to assist FDA in establishing a risk-based audit program. The problem is that it is difficult to measure something as intangible as culture with cold, hard data. The remaining question is: If taken together, are the three proposed optional metrics indicative of a quality culture? The answer is, maybe.
Careful thought and consideration should be exercised when determining what to measure, how often to measure, how to interpret and communicate the data, and what the expectation is for using the data to drive positive change. Management needs to be cognizant of the fact that whatever metrics are reported, they must be developed, evolved, and adjusted over time to maximize their impact on driving positive change.
When choosing a metric, it is important that the architects of the metric are aware of unintended consequences that may inadvertently drive negative behavior. Management attempting to incentivize achievement of the goal such as offering a financial award if the goal is achieved, may lead to inappropriate behaviors that do not address the real issue. In these cases, it is generally not the metric that will drive the behavior but rather use of behavioral rewards. Reward for achievement rather than analysis of the real underlying causes will not lead to sustainable positive change. When managed properly, metrics are an important tool to help drive positive change and quality process improvements.
Upon observation, an unhealthy quality culture is easy to identify. People in a poor culture do not understand their job and its importance to the business. They often appear stressed, and they hide their mistakes or blame others for their errors. There is no evidence of teamwork. People work in silos and rarely, if ever, seek input or advice from others. Metrics that could potentially be used to measure a poor culture include a large employee turnover, an overabundance of deviations attributed to human error, and lack of pride in the performance of employees’ jobs.
In contrast, a robust, healthy quality culture can be evidenced by alignment of goals between quality and operations, self-sustained work teams that focus on continual improvement, and employees who incorporate quality into their jobs on a daily basis. They are not afraid to speak up and offer suggestions for improvement to their colleagues. People understand the importance of their jobs and respect each other and their management. This culture welcomes inspections and views these inspections as another tool to use in their continual improvement initiatives. Metrics that could potentially be used to measure a healthy quality culture include a small employee turnover, deviations that identify a root cause other than human error, and pride in the performance of their jobs.
Conclusion
When establishing a metrics program, companies should evaluate numerous data input points including, but not limited to, product-quality attributes, manufacturing site performance, people metrics, and quality-system metrics. For product-quality metrics, companies should consider reporting on batch-specific data such as trending drug product, drug substance, and stability test results against customer complaint rates. Indirect product-quality metrics could include environmental monitoring, water trend results, and yield rates. When establishing site metrics, the company could look at inspection history including internal audit findings and maintenance history such as equipment age versus defect-failure rates. People metrics should consider ongoing job-specific training and education, skills and experience assessments, and employee turnover rate by job function and site. Quality systems metrics might look at change control, investigation root-cause trends, and release-testing cycle times.
There is no set requirement on which metrics a company should track to measure their overall performance. Each company should determine which metrics to track based on their operations, number of facilities they operate and where they are located, what types of products they manufacture, and what type of culture exists in their places of business.
The metrics chosen must be meaningful and written to provide a clear analysis of ongoing activities. It is important for operations and quality to agree on the metrics and how to report them to management to avoid overreaction to the data. It is not sufficient to simply report the data. The interpretation of the data is of crucial importance because it may include a root-cause analysis of its own.
References
1. FDA, Request for Quality Metrics Guidance for Industry, (Rockville, MD, July 2015).
2. The Food Drug Administration Safety and Innovation Act, Pub. L. 112-144, 126 Stat. 993 (2012).
3. FDA, “Food and Drug Administration Drug Shortages Task Force and Strategic Plan; Request for Comments” (Federal Register Feb. 12, 2013).