Multivariate Analysis: Pick Actionable Factors Redux

When performing multivariate analysis, say multiple linear regression, there's typically an objective (like "higher yields" or "troubleshoot campaign titers"). And there's typically a finite set of parameters that are within control of the production group (a.k.a. operators/supervisors/front-line managers).

This finite parameter set is what I call, "actionable factors," or "process knobs." For biologics manufacturing, parameters like

• Inoculation density
• pH/temperature setpoint
• Timing of shifts
• Timing of feeds
• Everything your process flow diagram says is important

are actionable factors.

Examples of non-actionable parameters include:

• Peak cell density
• Peak lactate concentration
• Final ammonium
• etc.

In essence, non-actionable parameters are generally measured and cannot be changed during the course of the process.

Why does this matter to multivariate analysis? I pick on this one study I saw where someone built a model against a commercial CHO process and proved that final NH4+ levels inversely correlates with final titer.



What are we to do now?  Reach into the bioreactor with our ammonium-sponge and sop up the extra NH4+ ion?

With the output of this model, I can do absolutely nothing to fix the lagging production campaign. Since NH4+ is evolved as a byproduct of glutamine metabolism, this curious finding may lead you down the path of further examining CHO metabolism and perhaps some media experiments, but there's no immediate action nor medium-term action I can take.

On the other hand, had I discovered that initial cell density of the culture correlates with capacity-based volumetric productivity, I could radio into either the seed train group or scheduling and make higher inoc densities happen.

by Oliver Yu

Contamination Time Window Redux

by Oliver Yu

There's a little LinkedIn brouhaha going on regarding the calculation of contamination time windows.

In the perfect world, there are no bioreactor or fermentor contaminations.

But if you were to have contaminations, the next best thing would to omnisciently know exactly what the true root cause is.

Since omniscience is not an option here, the next best thing is to narrow down the list of sterile-envelope manipulating actions on the contaminated bioreactor and perform root cause analysis to come up with MPC (most probable cause).

There are a lot of parallels between a crime scene investigation and bioreactor decontamination response. Just as crime-scene investigators attempt to determine the time of death to rule out potential causes outside the time frame, it is a good idea to compute a contamination time-window to rule in/out potential causes.

There are several assumptions in the contamination time-window calculation and which assumptions you use depends on your organization's risk profile. As in, which is worse?

• Coming up with too narrow a time window and risk eliminating the true root cause
• Coming up with too wide a time window and having too many potential causes to investigate?

To summarize, you run the exponential growth equation twice: the first use is to try to figure out the growth rate of the organism.

Step 1: Collect sterility samples and send samples to QC Micro.

Step 2: Get 2 pieces of information from QC Micro:

1. Concentration of contaminants in last sample (X)
2. Timestamp of last clean sample. (t₀)

Note, you already know the timestamp of the last sample (t).

Step 3: Pick an X₀ and try to compute μ.

X = X₀ e^{μ(t - t₀)}

bold is known.
blue is estimated.
red is what you're trying to determine.

The rational number to input here is the limit of detection of your sampling method.   If you're going for a narrower window, pick a higher number.

Now you have an estimated growth rate (μ) of the contaminant.

Step 4: Now try to determine the earliest insult (t₀)

X = X₀ e^{μ(t - t₀)}

Again, bold is known; blue is estimated; and red is what you're trying to determine.  Here, the key is to guess the concentration of the initial insult.  If you're afraid of ruling out the true root cause, pick 1 CFU.  If you're going after a realistic time frame to reduce scope, pick what you think a small bolus of contaminants would contain.

When you've solved for t₀, the second time, you now have computed the earliest time of contamination.

As you can see from this process, there are some serious assumptions.  But if applied correctly, you have a scientific and defensible basis for narrowing down potential bioreactor contamination causes and focusing your limited resources on rooting out the most probable cause.

Get Expert Contamination Consulting

Contamination Control of Cell Culture Bioreactors

by Oliver Yu

"Contamination Control"

A misnomer. I can see how they got that name... from "Pest Control," but I still hate it.

"Bioreactor control" makes sense as cell culture manufacturers try to direct the behavior of pH, dissolved oxygen, temperature, agitation, pressure...

But "contamination control"? No one is trying to direct the behavior of bioreactor contamination: Everyone tries to abolish bioreactor contaminations.

The abolition of bioreactor contamination in a large-scale setting is generally a team sport. It can take just one person to solve the contamination. But usually, the person who figures out what went wrong (the brains) is unlikely the same as the person who implements the fix (the hands). And in a GMP environment, the change implementation is a coordinated process involving many minds, personalities, and motivations. With all those people come an inordinate amount of politics for a goal that everyone seems to want to reach: no contams.

Immediate-/Medium-term Fixes

The first thing to realize is that operations management is usually the customer when it comes to solving bioreactor contaminations. Everyone's butt is on the line, but no group burns more resources responding to bioreactor contaminations than them. And in my experience, there is no "short-term" vs. "long-term" solution.

To operations management, there's just immediate solution and medium-term solution.

• Immediate solution :: what are you going to do for me today?
• Medium-term solution :: what lasting solution are you going to implement after the immediate solution?

Science... if it fits

The second thing to realize is that there's no room for science. The prime objective is to stop the contaminations. The secondary objective is to find the root cause. If identifying the root cause helps stop the contamination, that's a bonus; but root cause or not, you still must stop the contaminations.

For example, if your contamination investigation team thinks that there are five contamination risks, the directive from management will be to implement all CAPAs necessary to address all five risks. If the fixes work, "Great! You met the objective." Do you know what the true root cause was? Not a clue (it was one of those five, but you'll never know which one).

Political

The third thing to realize is that contamination response is as much political as it is technical.

• You can have the right solution, but present it the wrong way - and it's the wrong solution.
• You can formulate the right solution that is not immediately actionable, no one wants to hear about it.
• You can irrefutably identify the true root cause (thereby shining light on GMP deficiencies), and run against resistance.

Being right is different than being effective. And "bioreactor contamination control" at large-scale requires effectiveness. For in-house resources, it requires a keen understanding of interpersonal dynamics. For organizations that are at either a technical or political impasse, there are external bioreactor consultants who understand how to effectively troubleshoot and abolish bioreactor contaminations.

Abolish Bioreactor Contaminations

Who Are You Guys, Anyway?

So, I asked for a report to study Zymergi blog readers, and here's where the biotech/pharma readers are coming from:

This is a veritable who's who of the biotech world.  Obviously, you aren't all customers, but when it comes to large-scale biologics support, cell culture and bioreactor contaminations, readers and customers find themselves in good company.

Thanks for reading.

Note: All logos/trademarks belong to the trademark holder and inclusion on this list is not an endorsement of Zymergi or vice versa.

Putting Contamination TimeWindow to Use

by Oliver Yu

In a previous post, I introduced the calculation to estimate the earliest time of bioreactor contamination.

And the reason anyone'd ever bother running this calculation is to help direct the focus of contamination investigation.

Have a look at the example from the previous post. The sterility samples collected showed that the 12,000-liter bioreactor was "clean" all the way through 84-hours. By the time 108-hours culture duration rolled around, the dO2 and pH had crashed, prompting us to send the bottles to QC Micro. QC Micro reports that then 96-hour sample was also "hot."

There are folks who'd look at this data and say,

If we were clean at 84-hours, but hot at 96-hours, then bioreactor manipulations in that time-frame (84 to 96) are culprits for contamination.

But what if there were no bioreactor manipulations in that time frame but a sterile envelope manipulation at 77-hours?

Saying the 84-hour sample was "clean" is actually a mistake. It is more accurate to say, "Bioburden levels of the 84-hour sample were less than detectable." And using a clean 84-hour sample to vindicate prior manipulations would be a mistake by disqualifying true root causes.

On the other side of the spectrum are folks who say:

We need to look at every single sterile-envelope manipulation of the bioreactor starting from the time of inoculation at 0-hours.

This ocean-boiling approach is expensive and includes improbable root causes that ought to be disqualified.

The most effective approach lies somewhere in between and - we think - is to estimate the growth rate of the microbe by assuming the last "clean" sample was simply less-than-detectable. And computing this growth rate.

Using this growth rate to estimate the earliest 1 CFU could have penetrated sterile barriers is one scientifically defensible way of balancing the last-clean vs. boil-the-oceans approaches.

As for the assumptions of this method, they are:

• Constant growth rate of microbe. This method assumes that microbes entered the bioreactor in the growth phase and didn't stop. Since microbes (like spore-formers) can be in the stationary phase, the constant growth assumption tends to not include as much time as perhaps should be.
• 1 CFU inoculated the bioreactor. While it is unlikely that a bioreactor breach let in a single CFU or that the SIP killed all organisms except one, assuming 1 CFU tends to include more time and helps counter the assumption of constant growth.
• Once sample pulled, growth stopped. If the organism is an aerobe, this is a good assumption. If not, use the time of QC Micro count for (t).

Bioreactor contamination response is a lot like crime-scene response and investigation, and the contamination time-window calculation is a lot like estimating the time of death (of a murder victim). This information can be used to help rank probable cause and ultimately the most probable cause (i.e. identify the killer).

Get "Time of Contamination" Spreadsheet

Cell Culture Contamination Consultants

This is how we see customers who hire us to fix bioreactor contaminations:


No one likes getting contaminations.

No one wants to search the internet for "bioreactor contamination".

No one wants to pay consultants to help fix them.

But here you are...

...you found us because we write about bioreactor sterility concepts, principles and best-practices.

Sign a confidentiality agreement and give us a call.

Rebuttal to Atmospheric Breaks on Drains

Here's some feedback from an industry colleague regarding air-breaks on drains:

For one thing, BSL-2+ areas like [highly toxic] bacterial fermentation suites require the facility to have closed piping to avoid or minimize aerosol effects and biohazard contamination of people and the environment around the fermentor.  The BMBL 5th Edition (basically the biosafety bible) requires it for BSL-2 and above organisms.

Clearly we know that protecting workers is clearly paramount to safety.  But we also know that not every process uses "BSL-2 and above organisms."  A lot of facilities are designed "just in case" the expression system produces biohazard.

Closed-pipe drain headers requires a deep understanding of SIP cycle design and implementing a robust recipe to get rid of vacuum.

So in closing:

Most Interesting Customers in the World Hire Zymergi

This is generally our experience.

But we're professionals, so we get it.

For prospective customers, this is how it goes:

1. You give us a call and we basically have a phone interview.
   There's actually not much to talk about since we're not yet under confidentiality, so..
2. We need to sign non-disclosure.
   Typically, your Legal department wants us to sign your NDA.  In the meantime, if you need confidentiality you can sign Zymergi's Mutual Confidentiality Agreement.
3. Now, we can chat:
   * You tell us your problems
   * We tell you if we can help and if there's availability.
4. If business sounds worthwhile, you produce a purchase order.
   Vendor on-boarding is typically the rate-limiting step, but Zymergi has seen 36-hour turn arounds.

Next-day onsite service is usually impossible.  But depending on how adept you are at generating a purchase order (PO), we have been able to send consultants the following week.

Get Bioreactor Sterility Consulting

Drain Water vs. Clean Air - Drain Design for Bioreactor Contamination

UPDATE: The point isn't to install air-breaks at all costs.  The point is to use the correct BioSafety Level for your process, recognizing that a lot of facilities are overly-conservative for the processes they run.

On multiple consulting assignments, we are seeing an alarming trend where CIP manifolds and process piping are piped directly to drain.  We have identified direct piping to floor drains as contamination risks.  And our experience mitigating floor drain contamination risk is to cut the piping.

The main objection to this recommendation is that it would compromise the Class 100,000 clean room status of the process space.

With the cut in the piping, the worry is that contaminants from the drain are now able to enter the processing suite and will send your viable airborne particles beyond your environmental monitoring action limits.

But of the unfavorable options available, there's one that's obvious to us.

Your choices are as follows:

• Keep the bioreactor sipping drain water, but hey, you've got a Class 100,000 processing suite.
• Cut the pipes and get your bioreactor sipping fresh, 20 air-changes-per-hour filtered air.

It turns out that that we aren't the only ones who think this is true.  In a 2006 article on biocontamination control, @GENBio reported the "original views" of chemical engineer, Jim Agalloco:

...Trying too hard to protect the bioreactor environment can adversely affect the ability to sterilize equipment. For example, a steam sterilizer normally requires an atmospheric break between its drain and the facility drain, but some biotech companies object to that layout because it compromises the controlled environment.

Somehow, the viable airborne particles of the environment matter more than the ability to sterilize equipment.  They further state:

Eliminating the atmospheric break introduces more piping and surfaces, which leads to more opportunities for microbes to grow. To protect the outside of the tank, they purposely risk contaminating the inside.

Which is exactly our position on the matter.

We're aware that managing perceived action is as important as managing action.  But taking the action that keeps cell cultures from contamination is always defensible even if it flies in the face of perception.

Zymergi Bioreactor Sterility Consulting

4 out of 5 Best-Selling Medicines for 2013 are Biologics

by Oliver Yu

According to the business/investing website The Motley Fool, the best selling drugs for 2013 are:

1. Humira (4.8B) - biologic
2. Advair
3. Enbrel (4.1B) - biologic
4. Lantus (3.5B) - biologic
5. Avastin (3.3B) - biologic

The combined sales of the top 5 drugs come in near 20 billion dollars.  With 15.6 billion (79%) from the sales of biologics.

Using 2012 as baseline, Humira, Enbrel and Lantus were on the list of top selling biologics.  But Remicade, Rituxan and Herceptin all placed higher than Avastin.

Which gets me thinking... did the fool.com author work off incomplete data?

• 5 of 8 Top-Selling Drugs for 2012 were biologics
• Top-Selling Drugs for 2011

Genentech: Beware Lepto Contamination

A year ago on July 19th, 2012, Genentech VP of Biologics Quality (Anders Vinther) presented "A Novel Bacterial Contamination in Cell Culture Manufacturing" at the West Coast Chapter of the Parenteral Drug Association. (This same presentation was likely made elsewhere, but the only "Google-able" mention of it was on the WCC PDA website).

Most biotech/pharma companies are tight-lipped about their biologics manufacturing process problems.

by Oliver Yu
1. For one, they run proprietary processes: it's none of our business.
2. For two, Obamacare forces the FDA to approve a regulatory pathway for biosimilars: why share these growing pains with competitors who seek to eat their marketshare?
3. For three, why air out dirty laundry?

So when Genentech came forward with a very detailed presentation on bioreactor contamination and a prescription for how the rest of the biotech industry to handle this specific type of contamination, it's worth paying attention.

Their summary of events goes like this:

• Visual examination of cell culture indicates contamination of seed cultures
• Gram stain shows no bacteria
• 5-day incubation with standard plate count shows no growth
• No signs of contamination by looking at dO2, pH trends
• No evidence of contamination from standard QC testing methods

This contamination is the black swan event. Never in the history of cell culture manufacturing has anyone encountered a microbe that isn't detectable with standard methods.

Leptospira

After a second seed bioreactor contamination, Genentech was able to cultivate the bug and identify it as Leptospira. Leptospira is a coiled/spiral that survives in soil and water. It is motile, slow-growing obligate aerobe that favors liquid environments.

But the characteristics relevant to biologics manufacturing are:

• 0.1 micron in diameter - CAN PASS THROUGH 0.1 micron filters!
• Non-spore former - Not heat resistant
• Requires long-chained fatty-acid - Will not grow in media alone, requires presence of CHO cells

The remainder of the slides go through their root cause analysis and contamination investigation as well as global risk assessment (i.e. "CYA"). And it's certainly worth a gander.

For us, we would be wise to learn their lessons, which are:

• There's a bug out there that passes through 0.1 micron filters: L. licerasiae
• This bug (and related bugs) are not detectable with standard methods, so LOOK at your cultures!
• Update control strategies (consider heat-treatment and other barriers)

Battling contaminations is bad enough.  Now there's a bug out there that can get by sterile media filters and cannot be detected by no other method than by putting your eyeballs on it.

OSI PI Historian Software is Not only for Compliance

by Oliver Yu

In October 1999, I joined a soon-to-be licensed biotech facility as Associate (Fermentation) Engineer. They had just got done solving some tough large-scale cell culture start-up problems and were on their way to getting FDA licensure (which happened in April 2000).

As the Manufacturing Sciences crew hadn't yet bulked up to support full-time commercial operations, there were 4 individuals from Process Sciences supporting the inoculum and production stages.

My job was to take over for these 4 individuals so they could resume their Process Science duties. And it's safe to say that taking over for 4 individuals would've not been possible were it not for the PI Historian.

The control system had an embedded PI system with diminished functionality: its primary goal in life was to serve trend data to HMIs. And because this was a GMP facility and because this embedded PI was an element of the validated system, the more access restrictions you could place on the embedded PI, the better it is for GMP and compliance.

Restricting access to process trends is good for GMP, but very bad for immediate-term process troubleshooting and long-term process understanding, thus Automation had created corporate PI: a full-functioned PI server on the corporate network that would handle data requests from the general cGMP citizen without impacting the control system.

Back in the early-2000's, this corporate PI system was not validated... and it didn't need to be as it was not used to make GMP forward-processing decisions.

If you think about it: PI is a historian. In addition to capturing real-time data, it primarily serves up historical data from the PI Archives. Making process decisions involves real-time, data, which was available from the validated embedded PI system viewed from the HMI.

Nonetheless, the powers that be moved towards a validating the corporate PI system, which appears to be the standard as of the late-2000's. 

Today, the success for PI system installations in the biotech/pharma sector is measured by how flawlessly the IQ and OQ documents were executed.   Little consideration is really given to the usability of the system in terms of solving process issues or Manufacturing Sciences efficiency until bioreactor sterility issues come knocking and executive heads start rolling over microbial contamination.

Most PI installations I run into try to solve the compliance problem, not a manufacturing problem, and I think this largely the case because automation engineers have been sucked into the CYA-focus rather than value-focus of this process information:

• Trends are created with "whatever" pen colors.  
• Tags are named the same as the instrumenttag that came from the control system.  
• Tag descriptors don't follow a nomenclature
• Data compression settings do not reflect reality...
• PI Batch/EventFrames is not deployed
• PI ModuleDB/ AF is minimally configured

The key to efficiencies that allow 1 Associate Engineer to take over the process monitoring and troubleshooting duties of 4 seasoned PD scientists/engineers lie precisely having a lot of freedom in using and improving the PI Historian.  

If said freedom is not palatable to the QA folks (despite the fact that hundreds of lots were compliantly released when manufacturing plants allowed the use of unvalidated PI data for non-GMP decision), the answer is to bring process troubleshooters and data scientists in on at the system specification phase of your automation implementation.

If your process troubleshooters don't know what to ask for upfront, there are seasoned consultants with years of experience that you can bring onto your team to help.

Let's be clear: I'm not downplaying the value of a validated PI system; I'm saying to get user input on system design upfront.

Continuous Improvement of Bioreactor Sterility

In a lot of bioreactor contamination investigations, the root cause is never found, that is: the cause of bioreactor contamination is not conclusively determined.

This is quite disappointing in a cGMP environment because the way problems get fixed is that you find the root cause, the technical folk propose a solution, you write it up in a CAPA and push it through the "change implementation team" and you never have to deal with that problem again.

But if root cause is never found, there is no corrective action; there certainly is no preventative action and the chain reaction of cGMP improvement never takes place.

One reason the true the proverbial smoking gun is never found is because there are too many other "smoldering guns" at the crime scene. As they say of a theory that cannot be confirmed, but also cannot be denied on Discovery Channel's show Mythbusters, "It's plausible."

• It is plausible to tilt your system to be more susceptible to contamination after you've sent thousands of liters of untreated, un-decontaminated cell culture down the drain.
• It is plausible that floor drains with no air-breaks are the source of your microbial contaminations.
• It is plausible that after several dozen pure cultures that the status-quo valve sequence on your feed-delivery line was pulling a vacuum and that sucks non-sterile fluid into your bioreactor.
• It is plausible that misaligned or misfit elastomers was the sterility breach that failed to keep out microbial contaminants.
• It is plausible that you were using the wrong type of steam for sterilization.

One reason these plausible sterility risks exist was because you didn't know about them. (if so, call me).

Another reason these plausible sterility risks exist was because they weren't worth fixing because the system wasn't "broke." You had bigger fish to fry and it's hard justifying that precious budget dollars should be spent on a system that was "working" fine.

That reasoning works until you get back-to-back contaminations and your biotech manufacturing plant is perceived to be out-of-control.

Now, you're staring down a laundry list of potential causes, all of which are plausible, many of the solvable, none of which you can cross off your list as the true root cause.

Which gets me to the point of this missive. You're not going to be mired in contamination for your entire career. You're going to have periods of success. All the sterility risks you can address during that time will ensure that your periods of failure in the future are short-lived.

How to Measure Cell Culture Productivity

by Oliver Yu

So I've written about key performance indicators (KPIs) for cell culture in the past. And the KPI depends on whether you're talking seed/inoculum vs. production culture.

Final Specific Growth Rate

To recap, if you're in charge of monitoring seed/inoculum cultures, your goal in life is to deliver a specific amount of cells in the growth phase; therefore, your goal is to have a high final specific growth rate.


Compute specific growth rate by taking the slope of the natural log of the biomass against time.

Culture Volumetric Productivity

If you're in charge of production culture you want to have as high a concentration (of product) in as short a time as possible. Therefore, you seek a high volumetric productivity, which is:

For lab- and pilot-scale operations, this KPI is just fine.

Capacity-based Volumetric Productivity

For large-scale, commercial cell culture, however, volumetric productivity of the culture misses the mark and here's why:

Suppose you had a 2.5g/L process and you were able to get that in 10 days. Your volumetric productivity is 0.25 g/L/day. If your working volume is 10,000L this means you have 2.5 kilos of product.

Well, if your bioreactor is a 10kL, then you're making as much product as possible. But if your bioreactor is a 12kL, then you're not using your capacity to the fullest. This is why a better metric is capacity-based volumetric productivity:

In situations where your plant is always running, there's another metric.

Plant-based Volumetric Productivity

Even though culture duration (harvest timestamp - inoculation timestamp) is the only period when the product is being produced, the total time consumed should include the time spent cleaning, sterilizing and prepping the bioreactor. The time spent turning a dirty bioreactor (from the last harvest) to a prepped bioreactor is called the turn time, and when you include that into the equation, you get a true measure for the productivity of your biologics manufacturing plant called plant-based volumetric productivity:

Summary

Plant-managers are fine knowing the final titer or just the total culture mass since titer is just a single number coming out of QC and culture mass is what you're accountable for.

But if your team of cell culture scientists/engineers in MSAT are interested in comparing cultures on an apples to apples basis, they had better be control charting volumetric productivities.