Contamination Time Window Redux

by Oliver Yu

There's a little LinkedIn brouhaha going on regarding the calculation of contamination time windows.

In the perfect world, there are no bioreactor or fermentor contaminations.

But if you were to have contaminations, the next best thing would to omnisciently know exactly what the true root cause is.

Since omniscience is not an option here, the next best thing is to narrow down the list of sterile-envelope manipulating actions on the contaminated bioreactor and perform root cause analysis to come up with MPC (most probable cause).

There are a lot of parallels between a crime scene investigation and bioreactor decontamination response. Just as crime-scene investigators attempt to determine the time of death to rule out potential causes outside the time frame, it is a good idea to compute a contamination time-window to rule in/out potential causes.

There are several assumptions in the contamination time-window calculation and which assumptions you use depends on your organization's risk profile. As in, which is worse?

• Coming up with too narrow a time window and risk eliminating the true root cause
• Coming up with too wide a time window and having too many potential causes to investigate?

To summarize, you run the exponential growth equation twice: the first use is to try to figure out the growth rate of the organism.

Step 1: Collect sterility samples and send samples to QC Micro.

Step 2: Get 2 pieces of information from QC Micro:

1. Concentration of contaminants in last sample (X)
2. Timestamp of last clean sample. (t₀)

Note, you already know the timestamp of the last sample (t).

Step 3: Pick an X₀ and try to compute μ.

X = X₀ e^{μ(t - t₀)}

bold is known.
blue is estimated.
red is what you're trying to determine.

The rational number to input here is the limit of detection of your sampling method.   If you're going for a narrower window, pick a higher number.

Now you have an estimated growth rate (μ) of the contaminant.

Step 4: Now try to determine the earliest insult (t₀)

X = X₀ e^{μ(t - t₀)}

Again, bold is known; blue is estimated; and red is what you're trying to determine.  Here, the key is to guess the concentration of the initial insult.  If you're afraid of ruling out the true root cause, pick 1 CFU.  If you're going after a realistic time frame to reduce scope, pick what you think a small bolus of contaminants would contain.

When you've solved for t₀, the second time, you now have computed the earliest time of contamination.

As you can see from this process, there are some serious assumptions.  But if applied correctly, you have a scientific and defensible basis for narrowing down potential bioreactor contamination causes and focusing your limited resources on rooting out the most probable cause.

Get Expert Contamination Consulting

Contamination Control of Cell Culture Bioreactors

by Oliver Yu

"Contamination Control"

A misnomer. I can see how they got that name... from "Pest Control," but I still hate it.

"Bioreactor control" makes sense as cell culture manufacturers try to direct the behavior of pH, dissolved oxygen, temperature, agitation, pressure...

But "contamination control"? No one is trying to direct the behavior of bioreactor contamination: Everyone tries to abolish bioreactor contaminations.

The abolition of bioreactor contamination in a large-scale setting is generally a team sport. It can take just one person to solve the contamination. But usually, the person who figures out what went wrong (the brains) is unlikely the same as the person who implements the fix (the hands). And in a GMP environment, the change implementation is a coordinated process involving many minds, personalities, and motivations. With all those people come an inordinate amount of politics for a goal that everyone seems to want to reach: no contams.

Immediate-/Medium-term Fixes

The first thing to realize is that operations management is usually the customer when it comes to solving bioreactor contaminations. Everyone's butt is on the line, but no group burns more resources responding to bioreactor contaminations than them. And in my experience, there is no "short-term" vs. "long-term" solution.

To operations management, there's just immediate solution and medium-term solution.

• Immediate solution :: what are you going to do for me today?
• Medium-term solution :: what lasting solution are you going to implement after the immediate solution?

Science... if it fits

The second thing to realize is that there's no room for science. The prime objective is to stop the contaminations. The secondary objective is to find the root cause. If identifying the root cause helps stop the contamination, that's a bonus; but root cause or not, you still must stop the contaminations.

For example, if your contamination investigation team thinks that there are five contamination risks, the directive from management will be to implement all CAPAs necessary to address all five risks. If the fixes work, "Great! You met the objective." Do you know what the true root cause was? Not a clue (it was one of those five, but you'll never know which one).

Political

The third thing to realize is that contamination response is as much political as it is technical.

• You can have the right solution, but present it the wrong way - and it's the wrong solution.
• You can formulate the right solution that is not immediately actionable, no one wants to hear about it.
• You can irrefutably identify the true root cause (thereby shining light on GMP deficiencies), and run against resistance.

Being right is different than being effective. And "bioreactor contamination control" at large-scale requires effectiveness. For in-house resources, it requires a keen understanding of interpersonal dynamics. For organizations that are at either a technical or political impasse, there are external bioreactor consultants who understand how to effectively troubleshoot and abolish bioreactor contaminations.

Abolish Bioreactor Contaminations

Putting Contamination TimeWindow to Use

by Oliver Yu

In a previous post, I introduced the calculation to estimate the earliest time of bioreactor contamination.

And the reason anyone'd ever bother running this calculation is to help direct the focus of contamination investigation.

Have a look at the example from the previous post. The sterility samples collected showed that the 12,000-liter bioreactor was "clean" all the way through 84-hours. By the time 108-hours culture duration rolled around, the dO2 and pH had crashed, prompting us to send the bottles to QC Micro. QC Micro reports that then 96-hour sample was also "hot."

There are folks who'd look at this data and say,

If we were clean at 84-hours, but hot at 96-hours, then bioreactor manipulations in that time-frame (84 to 96) are culprits for contamination.

But what if there were no bioreactor manipulations in that time frame but a sterile envelope manipulation at 77-hours?

Saying the 84-hour sample was "clean" is actually a mistake. It is more accurate to say, "Bioburden levels of the 84-hour sample were less than detectable." And using a clean 84-hour sample to vindicate prior manipulations would be a mistake by disqualifying true root causes.

On the other side of the spectrum are folks who say:

We need to look at every single sterile-envelope manipulation of the bioreactor starting from the time of inoculation at 0-hours.

This ocean-boiling approach is expensive and includes improbable root causes that ought to be disqualified.

The most effective approach lies somewhere in between and - we think - is to estimate the growth rate of the microbe by assuming the last "clean" sample was simply less-than-detectable. And computing this growth rate.

Using this growth rate to estimate the earliest 1 CFU could have penetrated sterile barriers is one scientifically defensible way of balancing the last-clean vs. boil-the-oceans approaches.

As for the assumptions of this method, they are:

• Constant growth rate of microbe. This method assumes that microbes entered the bioreactor in the growth phase and didn't stop. Since microbes (like spore-formers) can be in the stationary phase, the constant growth assumption tends to not include as much time as perhaps should be.
• 1 CFU inoculated the bioreactor. While it is unlikely that a bioreactor breach let in a single CFU or that the SIP killed all organisms except one, assuming 1 CFU tends to include more time and helps counter the assumption of constant growth.
• Once sample pulled, growth stopped. If the organism is an aerobe, this is a good assumption. If not, use the time of QC Micro count for (t).

Bioreactor contamination response is a lot like crime-scene response and investigation, and the contamination time-window calculation is a lot like estimating the time of death (of a murder victim). This information can be used to help rank probable cause and ultimately the most probable cause (i.e. identify the killer).

Get "Time of Contamination" Spreadsheet

Cell Culture Contamination Consultants

This is how we see customers who hire us to fix bioreactor contaminations:


No one likes getting contaminations.

No one wants to search the internet for "bioreactor contamination".

No one wants to pay consultants to help fix them.

But here you are...

...you found us because we write about bioreactor sterility concepts, principles and best-practices.

Sign a confidentiality agreement and give us a call.

Rebuttal to Atmospheric Breaks on Drains

Here's some feedback from an industry colleague regarding air-breaks on drains:

For one thing, BSL-2+ areas like [highly toxic] bacterial fermentation suites require the facility to have closed piping to avoid or minimize aerosol effects and biohazard contamination of people and the environment around the fermentor.  The BMBL 5th Edition (basically the biosafety bible) requires it for BSL-2 and above organisms.

Clearly we know that protecting workers is clearly paramount to safety.  But we also know that not every process uses "BSL-2 and above organisms."  A lot of facilities are designed "just in case" the expression system produces biohazard.

Closed-pipe drain headers requires a deep understanding of SIP cycle design and implementing a robust recipe to get rid of vacuum.

So in closing:

When Was the Bioreactor Actually Contaminated?

by Oliver Yu

In a previous post, I glossed over detection of microbial contamination. I'm certainly no QC Micro expert, but a former co-worker, Mary Jane McRoberts, who was telling me the sensitivity of these QC Micro tests:

me: Hey MJ, what are the chances that there's a bug in the sample, but that your tests just happen to not catch it?

MJ: I tell you what.... if there's one CFU (colony forming unit) in there, my test is going to pick it up.

So suppose the final sample I hand over has exactly 1 CFU in the entire sample.

If you are using 40mL bottles to collect samples, that's a concentration of 1 CFU/40mL = 0.025 CFU/mL.

In a 12,000-liter bioreactor... a.k.a 12,000,000-mL bioreactor, you're looking at 300,000 colony forming units floating around in your production culture before your QC methods are sensitive enough to pick it up.

Knowing this 0.025 CFU/mL is crucial in estimating the contamination time-window.

Contamination TimeWindow

Anytime you have a bioreactor contamination, one (good) question that gets asked is: "So when did the contamination happen?"

This is because the signs of bioreactor contamination show up long after the insult as it takes time for the microbial contaminants to consume detectable amounts of oxygen and nutrients to crash the dO2 and pH signals.

All you need to compute this time-window is a spreadsheet of your contamination timeline:
And the equation for exponential growth:

X = X₀ e^{μ(t - t₀)}

where:

• X is the concentration at time t
• X₀ is the concentration at time t₀
• e is the natural log constant
• μ is the growth rate

If we want to know the time of microbial contamination, we're interested in solving for t₀.

X is given to us by QC Micro...in this example, QC Micro counted the last sample and found the concentration to be:

X = 2.2 x 10⁵CFU/mL

The time of contamination is known to us:

t = 4.5 days

And if we want to be uber-conservative, we assume that the initial insult was simply 1 CFU. So if our bioreactor is 12,000-liters, the initial concentration is 1 CFU/12,000,000mL or:

X₀= 8.3 x 10^-8CFU/mL

We know X, we know t, we know e, but we don't know μ, so at this point we have 1 equation, but 2 unknowns (μ and t₀).

One way to estimate μ is to assume that the "last clean sample" was just short of the detection limit: 0.025 CFU/mL (assuming 40 mL sample bottle).  Solving for the growth rate:

μ = ln ( X/X₀ ) / ( t - t₀ )
μ = ln ( 2.2 x 10⁵/ 0.025 ) / ( 4.5 - 3.5 ) = 16 day^-1

Since we now know the growth rate (μ), we can flip the equation around and solve for the time of the initial insult (t₀):

t₀ = t - ln ( X/X₀ ) / μ
t₀ = 4.5 - ln ( 2.2 x 10⁵/8.3 x 10^-8 ) / 16 = 3.14 days (culture duration)

Using a simple plug 'n chug of the exponential growth equation and plate counts from QC Micro, one can estimate the time at which the microbial contamination actually took place.

Question: What are the implicit assumptions of this method?

See also:

• Microbial contamination probabilities
• Bioreactor Sterility Consulting
• You Suck at SIP

Most Interesting Customers in the World Hire Zymergi

This is generally our experience.

But we're professionals, so we get it.

For prospective customers, this is how it goes:

1. You give us a call and we basically have a phone interview.
   There's actually not much to talk about since we're not yet under confidentiality, so..
2. We need to sign non-disclosure.
   Typically, your Legal department wants us to sign your NDA.  In the meantime, if you need confidentiality you can sign Zymergi's Mutual Confidentiality Agreement.
3. Now, we can chat:
   * You tell us your problems
   * We tell you if we can help and if there's availability.
4. If business sounds worthwhile, you produce a purchase order.
   Vendor on-boarding is typically the rate-limiting step, but Zymergi has seen 36-hour turn arounds.

Next-day onsite service is usually impossible.  But depending on how adept you are at generating a purchase order (PO), we have been able to send consultants the following week.

Get Bioreactor Sterility Consulting

Drain Water vs. Clean Air - Drain Design for Bioreactor Contamination

UPDATE: The point isn't to install air-breaks at all costs.  The point is to use the correct BioSafety Level for your process, recognizing that a lot of facilities are overly-conservative for the processes they run.

On multiple consulting assignments, we are seeing an alarming trend where CIP manifolds and process piping are piped directly to drain.  We have identified direct piping to floor drains as contamination risks.  And our experience mitigating floor drain contamination risk is to cut the piping.

The main objection to this recommendation is that it would compromise the Class 100,000 clean room status of the process space.

With the cut in the piping, the worry is that contaminants from the drain are now able to enter the processing suite and will send your viable airborne particles beyond your environmental monitoring action limits.

But of the unfavorable options available, there's one that's obvious to us.

Your choices are as follows:

• Keep the bioreactor sipping drain water, but hey, you've got a Class 100,000 processing suite.
• Cut the pipes and get your bioreactor sipping fresh, 20 air-changes-per-hour filtered air.

It turns out that that we aren't the only ones who think this is true.  In a 2006 article on biocontamination control, @GENBio reported the "original views" of chemical engineer, Jim Agalloco:

...Trying too hard to protect the bioreactor environment can adversely affect the ability to sterilize equipment. For example, a steam sterilizer normally requires an atmospheric break between its drain and the facility drain, but some biotech companies object to that layout because it compromises the controlled environment.

Somehow, the viable airborne particles of the environment matter more than the ability to sterilize equipment.  They further state:

Eliminating the atmospheric break introduces more piping and surfaces, which leads to more opportunities for microbes to grow. To protect the outside of the tank, they purposely risk contaminating the inside.

Which is exactly our position on the matter.

We're aware that managing perceived action is as important as managing action.  But taking the action that keeps cell cultures from contamination is always defensible even if it flies in the face of perception.

Zymergi Bioreactor Sterility Consulting

Genentech: Beware Lepto Contamination

A year ago on July 19th, 2012, Genentech VP of Biologics Quality (Anders Vinther) presented "A Novel Bacterial Contamination in Cell Culture Manufacturing" at the West Coast Chapter of the Parenteral Drug Association. (This same presentation was likely made elsewhere, but the only "Google-able" mention of it was on the WCC PDA website).

Most biotech/pharma companies are tight-lipped about their biologics manufacturing process problems.

by Oliver Yu
1. For one, they run proprietary processes: it's none of our business.
2. For two, Obamacare forces the FDA to approve a regulatory pathway for biosimilars: why share these growing pains with competitors who seek to eat their marketshare?
3. For three, why air out dirty laundry?

So when Genentech came forward with a very detailed presentation on bioreactor contamination and a prescription for how the rest of the biotech industry to handle this specific type of contamination, it's worth paying attention.

Their summary of events goes like this:

• Visual examination of cell culture indicates contamination of seed cultures
• Gram stain shows no bacteria
• 5-day incubation with standard plate count shows no growth
• No signs of contamination by looking at dO2, pH trends
• No evidence of contamination from standard QC testing methods

This contamination is the black swan event. Never in the history of cell culture manufacturing has anyone encountered a microbe that isn't detectable with standard methods.

Leptospira

After a second seed bioreactor contamination, Genentech was able to cultivate the bug and identify it as Leptospira. Leptospira is a coiled/spiral that survives in soil and water. It is motile, slow-growing obligate aerobe that favors liquid environments.

But the characteristics relevant to biologics manufacturing are:

• 0.1 micron in diameter - CAN PASS THROUGH 0.1 micron filters!
• Non-spore former - Not heat resistant
• Requires long-chained fatty-acid - Will not grow in media alone, requires presence of CHO cells

The remainder of the slides go through their root cause analysis and contamination investigation as well as global risk assessment (i.e. "CYA"). And it's certainly worth a gander.

For us, we would be wise to learn their lessons, which are:

• There's a bug out there that passes through 0.1 micron filters: L. licerasiae
• This bug (and related bugs) are not detectable with standard methods, so LOOK at your cultures!
• Update control strategies (consider heat-treatment and other barriers)

Battling contaminations is bad enough.  Now there's a bug out there that can get by sterile media filters and cannot be detected by no other method than by putting your eyeballs on it.

Continuous Improvement of Bioreactor Sterility

In a lot of bioreactor contamination investigations, the root cause is never found, that is: the cause of bioreactor contamination is not conclusively determined.

This is quite disappointing in a cGMP environment because the way problems get fixed is that you find the root cause, the technical folk propose a solution, you write it up in a CAPA and push it through the "change implementation team" and you never have to deal with that problem again.

But if root cause is never found, there is no corrective action; there certainly is no preventative action and the chain reaction of cGMP improvement never takes place.

One reason the true the proverbial smoking gun is never found is because there are too many other "smoldering guns" at the crime scene. As they say of a theory that cannot be confirmed, but also cannot be denied on Discovery Channel's show Mythbusters, "It's plausible."

• It is plausible to tilt your system to be more susceptible to contamination after you've sent thousands of liters of untreated, un-decontaminated cell culture down the drain.
• It is plausible that floor drains with no air-breaks are the source of your microbial contaminations.
• It is plausible that after several dozen pure cultures that the status-quo valve sequence on your feed-delivery line was pulling a vacuum and that sucks non-sterile fluid into your bioreactor.
• It is plausible that misaligned or misfit elastomers was the sterility breach that failed to keep out microbial contaminants.
• It is plausible that you were using the wrong type of steam for sterilization.

One reason these plausible sterility risks exist was because you didn't know about them. (if so, call me).

Another reason these plausible sterility risks exist was because they weren't worth fixing because the system wasn't "broke." You had bigger fish to fry and it's hard justifying that precious budget dollars should be spent on a system that was "working" fine.

That reasoning works until you get back-to-back contaminations and your biotech manufacturing plant is perceived to be out-of-control.

Now, you're staring down a laundry list of potential causes, all of which are plausible, many of the solvable, none of which you can cross off your list as the true root cause.

Which gets me to the point of this missive. You're not going to be mired in contamination for your entire career. You're going to have periods of success. All the sterility risks you can address during that time will ensure that your periods of failure in the future are short-lived.

If Kryptonite is Root Cause, What's the CAPA?

Superman is out there beating up and capturing your above-average-IQ'ed criminals. What do you suppose his success rate is?

100%, right?

I mean, how do you go up against a guy who can fly, has heat-ray vision, X-ray vision and can withstand all applications of the Second Amendment?

Superman succumbing to kryptonite

Answer: You can't.

But suppose Superman pits himself against an evil genius,  say Lex Luthor, who discovered that Superman loses his powers upon exposure to kryptonite.

Superman's success rate just plummeted to 0%.

Were he to perform an analysis while floating around helplessly in a pool of water with kryptonite chained around his neck to identify the root cause of his downfall, what do you suppose the most probable cause (MPC) would be?

• Would it be the gullibility of honest/small-town upbringing?
  If so, this wouldn't explain his high success against other criminals.

• Would it be Lex Luthor's cunning?
  If so, this root cause could not explain the supervillain Brainiac of similarly high intellect.

• Would it be the kryptonite?

Most people would stop here and say the true root cause of his new low-success rate would be the kryptonite itself. Take away the kryptonite and Superman is back to 100% success. Bring back the kryptonite:  0%.

But if the Son of Jor-El assigned blame to kryptonite, what's the CAPA?

 A federal regulation banning the possession of kryptonite?  Federally-licensed kryptonite dealers?  Universal-background check on kryptonite purchases?

by Oliver Yu

See, I say the true root cause is Superman's pre-existing condition that makes him vulnerable to kryptonite. If you (somehow) take away this vulnerability, Superman would have 100% success all the time.

If he hired me to increase his success rates, I'd craft a super-suit made of lead. The extra bulk would not encumber him given his strength nor would he be susceptible to lead poisoning. Maybe throw in lead face-paint, lead gloves and lead boots to deflect the radiation from the kryptonite.

The key to a permanent increase in success rates may be to challenge conventional thinking and to address pre-existing vulnerabilities in your process--no matter how much success your process delivered in the past.

See also:

• Get a FREE Case Study on Solution to Microbial Contamination
• You Suck at Bioreactor Sterility
• You Suck at SIP
• Contamination Failure Modes
• Signs of Cell Culture Contamination

Bertrand's Box Paradox

The Bertrand's Box Paradox refers to this puzzle:

You have 3 boxes before you.

• Box A contains two gold coins.
• Box B contains one gold coin and one silver coin.
• Box C contains two silver coins.

Like so:

Box A
Box B
Box C

Question: Suppose you chose a box at random and withdrew one gold coin. What are the chances that the next coin is also gold?

Well, If I withdrew a gold coin from one a random selection of the three boxes, then I must have either Box A or Box B. Since I have two remaining choices: one favors a gold coin and the other favors a silver coin, then the chances of me pulling out a gold coin is 50% (aka 50/50).

Box A
Box B

Seems like it should be, right? Turns out it's wrong.

Here's how I wrangled this problem. I did it by using differing gold and silver coins.

Box A
Box B
Box C

So 3 boxes at 2 coins a piece means there are actually 6 possible outcomes in which I can randomly select a box and pull out coins.  Here they are:

		1st coin	2nd coin
1	Choose Box A and snag the Gold Eagle first.
2	Choose Box A and snag the Gold Buffalo first
3	Choose Box B and snag the Gold Eagle first
4	Choose Box B and snag the Silver Eagle first
5	Choose Box C and select the Silver Eagle.
6	Choose Box C and grab a Silver coin.

These are my only six options. Per the paradox, I withdrew a gold coin first and not a silver coin. This means that I didn't pick Box C's two possibilities.  It also means that I didn't pick one of two Box B possibilities.

Essentially, I have three possibilities left:

		1st coin	2nd coin
1	Box A: Gold Eagle first, then Gold Buffalo
2	Box A: Gold Buffalo, then Gold Eagle
3	Box B: Gold Eagle, then Silver Eagle

From here, it's pretty easy to see that my second coin has two out of three chances of being gold and one out of three chances at being silver.

And thus correct answer to Bertrand's Box Paradox is 2/3 or 66.67%.

Why does this talk of probabilities matter to a Manufacturing Sciences team or cell culture engineer?

by Oliver Yu

Well, understanding the math behind bioreactor contamination, or recovery step yields, is one of the foundations in explaining real phenomena. This matters is because your biological system is multivariate.

Not only that, your process steps are sequential: Production cultures come after inoculum cultures; arvests after production cultures; ProA after harvest and so on and so forth. The success of this step often depend on the outcome of the previous step. And CofA attributes measured at a late purification step could be caused by some factor at the production culture stage.

Large-scale biologics manufacturing is complex, far more complex than picking a box with two coins and pulling them out one at a time. Yet the math behind the Bertrand's Box Paradox shows us that we muggles are susceptible to missing the mark when conditional probabilities are involved.

Credits: Images are from the US Mint and therefore in the public domain.