Biotech Manufacturing Software & Support

RIP, JPK

2023-04-26T10:54:00.003-07:00

I found out yesterday that Pat Kennedy passed away.

When I met Pat in the early 2000s, he was a quiet, spry gentleman about a decade older than I am now. He was the owner/founder of OSIsoft, the then-largest private software company in the SF Bay Area that you never heard of... unless you work in manufacturing.

Even if you've neither heard of him nor his company, his life's work has probably impacted you. He wrote the software that captures, archives, and retrieves time-series data... instrument data.

The app opened on nearly every monitor (and the massive display) is PI ProcessBook, the face of the software that he created: PI. And in my world, PI is used everywhere.

At my second job, I was the fermentation engineer at Genentech's Vacaville site. And my first day on the job, my boss - Jesse Bergevin - pulls up PI ProcessBook and tells me that my job is read these squiggles on the screen and divine what is happening in each of the bioreactors.

So from 1999 to 2004, every day of my life, I pulled up this app and did my job.

In 2003, I met Pat for the first time at his annual PI User's Conference in Monterey, California. He rents out the Monterey Bay Aquarium in the evening and throws a huge bash. A group of us were walking back to the hotel all tipsy and I recall asking him why he doesn't take OSIsoft public. He responded simply, "Well, you see, Oliver, if I go public, that means I get a new boss... and to be frank, I don't want a new boss; I like my current boss just fine." This was the first of many words that would shape my worldview.

Our paths would cross again the following year when I was a speaker at one of the conference sessions and of all the talks to attend during my time slot, Pat came to mine!

So I already knew from Herb Boyer & Bob Swanson - the founders of Genentech, that to change the world and get rich (or die trying), you had to be a founder. J. Patrick Kennedy confirmed this for me.

Around the same time, I was reading the book Rich Dad, Poor Dad. In the book, Robert Kiyosaki tells the story of his mentor: Rich Dad, and his father: Poor Dad. That if you live in an environment/mindset you don't like, and you see an environment or mindset that you do like, you need to go find someone with that better mindset and get them to be your mentor. Expecting to get rejected, I fired off a doomed email asking Pat to be my mentor:

---------- Forwarded message ---------
From: Oliver Yu
Date: Fri, Jun 24, 2005 at 1:42 PM
Subject: mentor
To: <[email protected]>

Hi Pat -

I am wondering if you would be my mentor. I have been reflecting on where I want to be down the road, and I think that there is no better way of getting to my final destination than to seek the instruction of those who have arrived.

As a start, I'd like to hear you tell your story. I want to know the tough decisions you faced as you founded OSIsoft and the actions you took. How much was instinct, how much was luck? What would you differently if any? And I suppose as I move forward with my pursuits, I'd like to check in with you on your opinion of how I am doing.

I guess for now, I'm asking to buy you dinner and an opportunity to sell you on being my mentor. And if things seem right, we can proceed on a basis that meets your schedule.

Sincerely,
Oliver

Pat responded:

---------- Forwarded message ---------
From: Pat Kennedy <[email protected]>
Date: Sat, Jun 25, 2005 at 9:54 AM
Subject: RE: mentor
To: Oliver Yu

Interesting request - can you tell me more about what you want to
accomplish? I am always up for a dinner with a customer, particularly ones that give presentations about how great we are.

Pat

And so, on June 29, 2005, I drove to San Leandro where he gave me words that I couldn't possibly understand. Only decades later did any of that mean anything to me... the few things that stand out in retrospect:

He never set out to be a founder/software mogul. He was a real estate investor. OSIsoft was his job; real estate was his business.
He understands the root of his success.

DuPont had published and given away data compression algorithms.
His business partner encouraged him to start a consulting firm using these algos.
He knew to pivot from software to consulting when his customers kept demanding his software and less his consulting

He understood that his success was a combination of skills, money, luck and serendipity: and that he isn't sure - with immense success - that he could repeat it.

In September of that year, Pat wanted to take his daughter on a tour of several PI systems (funny he didn't tour facilities, he "toured PI systems"). He shows up carrying a manila folder with two images and wanted to show me:

Picture of Arnold Schwarzenneger standing in the mission control room of the California ISO with PI ProcessBook opened in the background.
Satellite image of land that he acquired

I recall him saying that those two images represent his pride and joy: OSIsoft and real estate. But it wasn't just about the success. It was about his values. He seemed to do things his way. The values that he imbued into his company culture. The people that worked for him were fiercely loyal and loved him dearly.

Ultimately, I would start a consulting business... like Pat.

Ultimately, I would pivot that business to software... like Pat.

Ultimately, I would become a founder and reach high levels of success... like Pat.

My deep condolences to the Kennedy family and the OSIsoft PI community. Godspeed, Pat. May you rest in peace.

Test-Driven Development and GAMP

2016-10-05T14:45:00.002-07:00

In the world of software development, there's a development methodology called, "Test-Driven Development" aka ("TDD") where you basically write the test BEFORE you write the actual functional code. There are other tenants of TDD, but the notion that you write the test first is the main focus here.

By writing the test first, it forces you to look to see where your "goal posts" are.

I actually have no opinion on TDD other than to point out that it exists, and there are some principles with strong parallels in the FDA-regulated markets with respect to computer system validation.

The V-Model refers to the way in which (software) systems are validated and verified:

The only reason (I can think of) for why this is called a "V-model" is because someone is being clever with the Verification and Validation. But I digress.

The point of the V-model is that someone actually went through and documented what the users/stakeholders wanted. Then, they went through and figured out what functions match with those user/stakeholder desires. THEN, they went through and figured out how to make those functions happen:

User Requirement Specification (URS)
Functional Specification (FS)
Detailed Design Specification (DDS)

That's the downstroke of the V.

The upstroke is the part where you go about proving that what you designed actually made it into the actual system. And the functions you said are available are actually available:

So in the world of computerized system validation (CSV, not comma-separated variable), the Installation Qualification (IQ) is what verifies that the items of the detailed design got into the system. So the battery of tests that prove the correct software configuration was inputted.

Operational Qualification (OQ) is the battery of tests that prove that the functions specified perform as specified. For example, if your functional specification says that your impeller speed has to operate between 60 to 255 rpm, you should have a test in your OQ that runs the impeller at the boundary and within that operating range.

Installation Qualification
Operational Qualification

So what does any of this have to do with test-driven design? Well, anyone who has validated a computer system knows that for the most part, the V is executed exactly as it's drawn: first the downstroke, then the upstroke:

You first write the URS, then the FS, then the DDS. After this planning phase is over, there's the testing phase during which time you draft the IQ, OQ (and sometimes PQ). And by the time you get to executing these tests, you may find nonsensical tests or deviations.

And the reasons may be that you ran into these issues is because you didn't start with the tests (IQ/OQ) first.

The TDD methodology is saying that you start with the URS (what the stakeholders/users want). Then you skip ahead and write what a passing test looks like (IQ and OQ). Then you go backwards and fill out the specification and perform the build.

The thinking seems backwards on first-pass, but it isn't without merit.

Tes(u)la Model X Ownership - Day 1

2016-09-28T11:13:00.000-07:00

Today was the day I picked up my Tesla Model X from the showroom. The showroom spartan as they were clearing out inventory for end-of-Q3.

Paperwork was pretty slender since all the preparation happened up front, and I got in and out in about an hour.

Initial Impression

The new car smell is very strong with this car. Some people love it; I dislike it. Very dizzying.

What's also dizzying is the regenerative braking which comes with in 2 flavors: Standard vs Low. Coming from an automatic car, LOW is what you're going to be most used to, but since you're not capturing as much juice, this overall decreases the range of the EV. Standard takes a bit of getting used to. My test-drive was much the same. Standard basically means that the moment you let off the accelerator pedal, you're not coasting... you're actually braking. So if you're used to coasting in either a regular car or in a hybrid vehicle, this is going to noticeably jolt you forward everytime you abruptly let off the gas.

Between the new car smell and the STANDARD regenerative braking, it was a woozy day.

Key Fob

The Key Fob seems over-designed. It looks like a palm-sized plastic version of your car missing what designers call, "Affordance"... obvious things to press. Key fob looks like some sort of voodoo doll for the car. You press the front and the front-trunk opens up. You press the back and the rear hatchback doors open up. You press the windows and the Falcon-Wing doors open. Oddly, you press the hood of the car and it locks the car. Haven't tried stabbing it with a needle to see if the car screeches in pain.

And while there are power-doors on the driver and navigator doors, there's no way to open those doors using the key fob.

Affordance

An affordance is the possibility of an action on an object or environment. Additional meanings have developed, largely a result of misinterpretations. The original definition in psychology includes all actions that are physically possible.

So the convention with all doors (since the beginning of cars) is that you pull on it to open. You push it to shut. The Model X has this crazy thing where you push on this bar to open the door.

There are 2 User Experience principles being violated here (as I understand it from Joe Natoli at Give Good UX):

Tesla has gone against convention. Now, being conventional isn't all that great. But convention is hard for users to break, so if you're going to go against convention, you ought to have a really good reason to do it.

Tesla has also gone against the user experience principle of affordance. The possibility of action is an open door that comes towards you if you're on the outside of the car. Pushing is the action going towards in the inside of the car.

For now, I'm going to trust that this is only a cursory violation of good user experience principles and that there are good reasons for it.

Elon Musk's Advice for Entrepreneurs

2015-07-15T13:42:00.001-07:00

Without comment, because no one says it better than Elon.

What do you call doing the same thing over and over and GETTING different results?

2015-06-04T14:31:00.000-07:00

Controllable Parameters of Mammalian Cell Culture

2015-06-03T11:18:00.000-07:00

Einstein famously said:

Insanity [is] doing the same thing over and over again and expecting different results.

Which got me thinking...

Question: Is there such a thing as doing the same thing over and over again and getting different results?

Answer: Biotech Manufacturing

Rob Caren once asked: "How hard can (large-scale) cell culture be? It's ONE unit operation."

He also observed that large-scale chromatography shouldn't be that hard either:

Send the pool through that fixed-bed reactor.
When the optical density is not zero shut this valve and open that valve.
Try not to send product to drain.

On both counts, he's right, but I can only speak to the cell culture side.

When running bioreactors, there are a few parameters that are within your control. These parameters are sometimes called "knobs" because the production team can literally go to the control system and "turn a knob" (or click a few buttons on the SCADA) to change the parameter.

For the bioreactor or fermentor, those knobs are:

pH. The intracellular pH is known to change the activity of enzymes that run the rates of metabolic reactions. Since cells regulate intracellular pH, the best you can do is to control the extracellular pH. In reality, the pH for the process is specified. And if specified well, the specification will come with a target range and a proven acceptable range. At commercial scale, you ought to be able to operate within the proven acceptable range. For more information on pH control in mammalian cultures, see this.

Dissolved oxygen. Maximizing cellular productivity means aerobic metabolism. From university biochemistry, we know that anaerobic metabolism produces far less energy (2 ATP) than aerobic metabolism (38 ATP). We also know from chemistry that the solubility of oxygen in water is quite low (<10 mg/L when temperature > 15 degC). Therefore, it is important that the bioreactor supplies oxygen. Some bioreactors supply air and others supplement with oxygen. While the dissolved oxygen range is typically specified by the process, the air supply, air/oxygen mix, flow rates and sparge-type can be determined by the facility. As previously discussed, dissolved oxygen depends on other parameters such as agitation and temperature and can be changed within the specified range.

Temperature. Temperature control happens with the bioreactor jacket where water is flowed around the outside surface of the bioreactor. When the temperature gets too hot, the control system sends cooler water; and when the temperature of the culture gets too cold, the control system sends hotter water. While temperature is typically specified, there are processes that will intentionally cool the culture to reduce the rate of metabolic reactions and extend culture viability. Also, since temperature is defined in a range, the setpoint is a turnable knob.

Agitation. Agitation is typically not specified by the process, just that the cells must be suspended (i.e. not settled on the bottom of the bioreactor). In practice agitation rate is determined by power-to-volume calculations and stays constant for the bioreactor, nonetheless, this is a manipulatable parameter when running bioreactors.

Timing of Inoculation. Inoculation density is often specified by the process. But there's no way to "dial down" or "dial up" inoc density in a control system somewhere like you can with pH, dissolved oxygen, temperature and agitation. When cells grow, the cell density naturally increases, so the way to control inoculation density is to time it (i.e. wait vs. not wait).

Timing of feeds. In fed-batch cultures, additional nutrients are added to the culture. The additional nutrients tend to increase the osmolality and the additional volume can help dilute the cellular waste (like ammonium). Not in all processes, but in some processes, the timing of feeds have been shown to impact culture productivity.

Timing of shifts. Some processes are specified with changes in setpoints of the aforementioned parameters (e.g. pH or Temperature). The shift specifications come in the form of: "When the culture duration reaches X hours" or "when the cell density reaches Y x 10⁶," then change the set point up/down.

When building multivariate models, it is crucial that controllable parameters are modeled as factors and here's why:

When your model shows can correlate significant main effects and interactions to some process output (e.g. titer or quality attribute), you can actually step out of theory and prove it in practice.

New Features for ZOOMS 2.2.4 (shipped)

2015-03-20T17:32:00.001-07:00

There are a just a couple of usability improvements for ZOOMS 2.2.4.

Remove keywords with click.

Customers were telling us that it is easy to search for a trend by clicking on the various links within the search engine result page (SERP). But it was hard to remove some of the parameters that they didn't want. With ZOOMS 2.2.4, you're able to remove any keyword from the query with a simple click.

Below the search bar, there is a statement indicating that results were found including the timestamp of the search (in case the user has stepped away, they ought to know that the trend may be stale).

Further below the search bar is a list of all the keywords that were recognized by ZOOMS. Units are shown with a unit icon. Parameters are shown with a parameter icon..etc.

As of ZOOMS 2.2.4, these keywords are buttons with an "x" to the right-hand-side. Clicking the button will remove that keyword from the search box (so that you can re-search).

Show Values textbox travels

Another comment we were getting was that the "show values" textarea stayed at the top, and when you had multiple trends found, if you clicked "show values" on a lower trend, you could not see that the values actually populated. With 2.2.4, this issue has been fixed. The textarea "follows" you as you scroll:

This does not require a full re-install, just a single-file to update the changes. Go ahead and send email [email protected]">Zymergi Technical Support and we'll send you the file and instructions.

ZOOMS Data Historian Search Engine Configuration

2015-03-03T13:05:00.002-08:00

ZOOMS is your search engine for process (trend) data.

Step 1: Go to the home page (using web browser)
Step 2: Input your keywords (hit enter)
Step 3: View the SERP

(SERP is an abbreviation for "Search Engine Results Page.")

This is the way that Google has conditioned the world to find pages of information on the internet and this is the user experience we at Zymergi think ought to be available in the enterprise.

But how do you configure ZOOMS?

It turns out for ZOOMS to work with your system, you need to answer 4 questions:

Where should ZOOMS get time-series data?
Where should ZOOMS look to know about the points (i.e. tags or IO)?
Where should ZOOMS look to find out what units and parameters exist on your system?
Where should ZOOMS look to find "batches" that exist on your system?

In each of the cases, if the answer is, "I don't know," that's perfectly alright. ZOOMS has an internal database to keep track of all this.

Questions 1 and 2 refer to the time-series data and the point configuration. You see, ZOOMS is a search engine for trends, and trends are created when you plot values vs. time:

All this means is that you need some place to put the time-value pairs (OSI calls this the PI Archive); and you need some place to put the tag information (in OSIsoft PI, this is the "point database"). These two databases are responsible for the Y-axis.

Customers typically select OSIsoft PI as the time-series data source and the point data source. That's awesome. Zymergi is an OSIsoft Partner and frankly, we think they're fantastic.

Question 3 talks about units and parameters. Units and parameter are the alternative way that people refer to tags. People think of I/Os as a parameter of some piece of equipment, and so we need a place to store unit/parameter combinations.

ZOOMS was created back in 2008 (can you believe we're going on 7-years?) and back then, the data store for unit/"alias" data was the PI Module Database. Since then, OSIsoft has moved towards Asset Framework ("AF"), but since AF wasn't ready during ZOOMS development, we created our own metadata storage. So while any relational database will do, there is a local database that ZOOMS uses.

Question 4 is about time-windows. Since time-windows is what defines the X-axis of a trend, being able to search by time-windows is of utmost importance. In batch-processes (such as pharma and biotech companies), batch identification is used to identify time-windows, but if you're a continuous process, you can define batches as work-shifts, days of the week, fiscal metrics...whatever. It doesn't matter. What matters is that there is a way of storing time-window information:

This is why ZOOMS has a local time-window data store.

The key here is that being able to search for trend data depends on 4 data sources and thus the configuration of ZOOMS starts with setting these connection strings.

New Features in ZOOMS 2.2.3 (which has shipped, btw)

2014-12-09T18:12:00.001-08:00

Yep, ZOOMS 2.2.3 is shipped. There are multiple upgrades, including several under the hood, but the user-features are as follows:

Autocomplete Detects Text Within Words

We are fast learning from customers that despite S88, people naturally like having the equipment hierarchy in the unit name. It's natural and logical. As such, we've modified our autocomplete to search after delimiters such as the period (.) and the underscore (_). You can see below that the user typed "10" and yet items that have "10" after a legal delimiter show up.

TimeExpressions

Inputting time is a pain in the buttocks. ZOOMS already recognizes the PI time expressions (*, t, y) as well as (60s, 1m, 1h, 2d, 3w, 4y...) time intervals. But our users like not having to type the forward slashes (/) and colons (:) that come with inputting time like "11/23 23:00."

ZOOMS now recognizes several more oddball time formats that don't come standard. You can see below that "1123 2300 to 1130" resolved to: 11/23/2014 23:00:00 to 11/30/2014 0:00:00.

Downloading Raw and Evenly Spaced Data

The ability to download raw data has existed since 2.2.2. Raw data is the actual data captured in the data archive. Unfortunately, the captured data may not happen at the same timestamp and if you need to correlate values with other parameters. So for version 2.2.3, we have the ability to download evenly-spaced data in user-specified increments. Click that "timed CSV" button and you're going to get interpolated data values every 15 minutes:

Trend data visualization and data downloads remain a core offering. Stay tuned for more release announcements.

Request a ZOOMS demo

Could not load file or assembly 'System.Data.SQLite' or one of its dependencies.

2014-12-01T11:16:00.001-08:00

This is why you're here:

You're getting this error.

<add assembly="System.EnterpriseServices, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"/>
<add assembly="System.Web.Mobile, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"/>
<add assembly="*"/>
<add assembly="System.Runtime.Serialization, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL"/>
<add assembly="System.IdentityModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL"/>

You're on a Windows OS
You're trying to connect to SQLite (System.Data.SQLite.dll)
You're trying to get an IIS/ASP.NET web application to connect to SQLite
You're in luck.

This a pretty simple fix that we talked about before.

What you need to do is Enable 32-Bit Applications.

When you Enable 32-Bit Applications (by setting it to true), the worker process(es) serving the application pool will be in WOW64 (Windows on Windows64) mode... which means that your application pool is now a 32-bit process that loads only 32-bit applications.

Metadata Nomenclature for ZOOMS units and parameters

2014-11-24T19:13:00.004-08:00

So in the world of industrial automation, there's this S88 standard for batch. And while it doesn't address everything, it is an oldie, but goodie.

S88 talks about an equipment hierarchy, which is a way of saying:

Your equipment can be named like so that each unit is unique.

Here's an example:

Suppose you're standing in your plant looking at a your three production fermentors: F101, F102, and F103

And let's suppose that they are located in the Fermentation suite of your brewery.

Let's say your brewery is in Davis, California and the name of your company is Zymurgy Brewing.

Well then one way to name that production fermentor: F101 is as follows:

/Zymurgy/Davis/Fermentation/F101

and for the rest of your units:

/Zymurgy/Davis/Fermentation/ProductionFermentor/F101

/Zymurgy/Davis/Fermentation/ProductionFermentor/F102

/Zymurgy/Davis/Fermentation/ProductionFermentor/F103

Now let's suppose Zymurgy acquires a brewery in Dixon, CA with the same unit names: F101, F102, F103, but at that brewery, those units are portable tanks:

/Zymurgy/Dixon/Fermentation/PortableTanks/F101
/Zymurgy/Dixon/Fermentation/PortableTanks/F102

/Zymurgy/Dixon/Fermentation/PortableTanks/F103

By identifying units with their hierarchy, it's very difficult to get "collisions" where two units with the same name, have the same exact hierarchy.

Now it's easy to declare "F101, Agitation" or "F103, Air Flow"

Easy right?

Well suppose you didn't want to name your units that way. Suppose you want the unit to be:

DIX.F101

DIX.F102

DIX.F103

That's fine, too... at least within ZOOMS.

The next hiccup with metadata nomenclature is trying to figure out which part of the descriptive goes in the unit vs which part goes in the parameter.

For example, suppose you had a water chiller:

VV.CHILLEDWATER4000 could be the unit and Temperature could be the parameter.

Alternatively, you may go with:

VV.CHILLER4000 as the unit Water Temperature as the parameter.

Should "Water" be in the unit name or should it be in the parameter?

There's no hard rule... only that the unit and the parameter should have high reuse. This is to say, if you have multiple homologous units, yet one is "Temperature" while the other is "Water Temperature," there's probably a better way to handle your naming.

Could not load file or assembly 'X' or one of its dependencies.

2014-11-09T16:20:00.001-08:00

You're seeing this Configuration Eror:

And you're using IIS7 with ASP.NET and using 32-bit applications.

Chances are, you just installed a fresh copy of IIS and you haven't changed around any defaults.

You're in luck because the fix is easy. Here's what's going on.

The default installation of IIS 7 is to disable 32-bit Applications. Afterall, IIS 7 comes with Server 2008 and Server 2008 is an x64 system.

So all you have to do to get rid of this error is to Enable 32-bit Applications in IIS. Here are the steps:

Go to Start and type in inetmgr to launch IIS
In the left page, select Application Pools
In the middle pane, select the application pool corresponding to the web-application that is giving you this error.
In the right pane, select Advanced Settings:
This is what you'll see: Enable 32-bit Applications is set to False.
Go ahead and set that to True:
Click OK

And you're super awesome application that depends on 32-bit DLLs is back in business:

iOS 8 Safari Cannot Access Windows Authentication IIS websites

2014-10-05T19:23:00.000-07:00

So our flagship product, ZOOMS - search engine for OSIsoft PI - is a web-application... meaning that like Google, you get to the app by using a web-browser.

ZOOMS itself is browser-agnostic - we don't care what browser you use, just that you use a modern browser. It's HTML5 meaning that all modern browsers ought to be able to view it. And all modern browsers include Safari on your mobility devices.

ZOOMS is also built on the Microsoft stack, meaning that the operating system is Windows Server. The web-server is Internet Information Services (IIS). The database is OSIsoft PI. And the application programming language is ASP.NET.

For customers using Windows Active Directory, being able to authenticate users against the AD is useful and so that just means that before you get to use the website, you have to type in your active directory username and password.

Imagine my surprise when I heard from a customer that "ZOOMS isn't working."

After a lot of troubleshooting, it turns out that iOS Safari doesn't play well with Windows Authentication.

When trying to access intranet sites in our company that use Windows (IIS) Authentication, it challenges for the login and password 2 or 3 times, then nothing - just hangs. These sites work fine on iOS 7 devices - and worked on the same devices that were upgraded. Windows Authentication was broken in iOS 7.0 also and not fixed until a later update. Why does this keep getting broken?

This was exactly what happened to my customer. Gets challenged for their password twice and then the page hangs at ~20%.

For now, ZOOMS users should understand that iOS 8 is unsupported - not because of ZOOMS, but because of iOS 8 - and that if your situation is feasible: disable Windows Authentication until Apple comes up with a fix.

Mathematical Bummers and the American Canyon/Napa Earthquake

2014-08-25T23:04:00.003-07:00

As Northern California cleans up from the American Canyon/Napa Earthquake of August 24th, 2014, I am reminded of a mathematical exercise I was asked to perform in my early years as an engineer with my former employer.

The question posed by upper management was this:

Given estimated probabilities of an earthquake, grass fire, flood, etc... what is the probability in any given year that any of the natural disasters hit the factory. For example, suppose estimates from the U.S. Geological Survey for the probability of natural disasters in any given year were as follows:

p_earthquake = 5%
p_{grass fire} = 10%
p_flood = 2%

What is the probability that any of these hit?

Well, if you multiplied the probabilities, you'd simply get the chance that all 3 of these natural disasters happen in the same year... so 5% * 10% * 2% is 1 in 1,000.

But that was not, "What is the probability that all of these hit in the same year?" The question is what is the probability that any of them hit in the same year?

To get to the bottom of this, you simply ask yourself, "What is the probability that none of these things happen in any given year?"

Well,
the probability that no earthquake hits is 95%.
the probability that no grass fire hits is 90%.
the probability that no flood happens is 98%.

So the probability that none of these things happen is 95% * 90% * 98% = 83.8%

What's the probability that any one of these things happen? 16.2% a.k.a 1 in 6.

So we handed this answer into upper management and I wasn't in the room, but I was told that my director got scoffed at for presenting this figure.

It's been 13 or 14-years since I handed in that calculation and they've been rolling that dice every year. Now that it's happened (I slept through it, "shaken, not stirred"), I feel a bit vindicated.

Why does this matter to Zymergi or our customers? Well, it's the basics of bioreactor contamination probabilities.

If you add up the probabilities of your sterile envelope failures as well as your sterile operations failures, you end up with an N-sided dice that you roll each time you put up a production culture. And if you've done the math, you realize it doesn't take that many low probability vulnerabilities before they add up.

Process Data for the Rest of Us

2014-08-04T03:33:00.000-07:00

I first started using OSIsoft's PI in 1999 when I got hired as a fermentation engineer at the world's largest cell culture plant (at the time, 144,000-liters).

I remember the first week vividly. Me - not knowing anything about how read cell culture trends - watching my boss run his cursor over PI trends, nodding his head and then running off to meetings telling the operations staff what's happening in the bioreactors.

Over the years, I've put my eyeballs on thousands of cell culture runs and became an expert on the matter. Yet, no matter how many trainings gave to tech(nician)s, supe(rvisor)s, and managers to increase the population of process experts at the company, I got the sense that ultimately it was still my team's (i.e. manufacturing sciences') job to know what's happening to the cell culture runs in-progress.

OR, maybe not...

The VP of the plant had the PI administrator write a script to open up PI ProcessBook and snapshot current trends as a images and put it on a website (remember, this was back in 1999). Clearly management recognized the value of these trends, but was just too much activation energy to get PI data to "The People."

So when I left my awesome company (and awesome job), I set out to do one thing:

And by PI, I mean "process data."

Google had already pioneered the one text-box, one click format for accessing website data. So why is it that cGMP citizens can find information on the internet faster than they can find critical process data from within the enterprise?

This is why I bothered creating ZOOMS and this is why I think that there's a place for ZOOMS in the competitive space of trend visualization on the web.

It actually wasn't until the Jared Spool lecture at this year's OSIsoft PI User's Conference that I learned how to better enunciate this creation.

A quick recap:

The bottom of the escalator is where you are if you know nothing about an app (new users live here). The top of the escalator is if you know everything about the app (developers live here).

Current knowledge is where the user is today; and target knowledge is where a user needs to be to know enough about the application to perform his duties.

Mr. Spool tells us that intuitive design happens when the knowledge gap (the difference between target knowledge and current knowledge) is zero.

A key observation is that the more powerful and feature-rich the tool, the higher up the target knowledge is...and the harder the knowledge gap is to close.

The success of Google (which is to be replicated with ZOOMS in the process trend visualization context) is a modest number of features in order to lower the target knowledge... and thus diminish this knowledge gap and achieve intuitive design.

There are plenty of feature rich PI trend visualization tools for process experts. ZOOMS is process trends for the rest of us; in other words: PI for the "non-user."

At the end of the day, it's People, Process, and Technology... in that order. You can buy awesome technology, but if only a small minority of your people use it, you're neither capitalizing on their potential, nor your process.

ZOOMS v2.2 shipped!

2014-07-29T23:28:00.003-07:00

by Oliver Yu

Wow, what an exciting week.

ZOOMS version 2.2 is shipped! The Top 3 features in this release are:

Wildcard tag search
Digital States now rendered; Step=1 recognized
Full-text search on units

Wildcard Tag Search

Wildcard tag search was a feature that was added back to ZOOMS. This feature originally left behind because the idea behind ZOOMS is that users want to search metadata, not tags. As it turns out, there's a delay between when the tags get added to the data archive and when the ZOOMS metadata gets updated, and in between that time, these tags aren't findable on ZOOMS. Well, we listened to our customers and it's back in there.

ZOOMS detects if there is a wildcard in the query; if there is, ZOOMS will do a tag search and show the results asynchronously. This way, the slowness of the data archive does not impact the speed of the page load.

Digital States and Stepped Tags

Up until now, ZOOMS interpolated all values - which is great for a majority of numerical tags that come in at high frequency. But what about the tags that don't change in value... or are monotonic in nature (like setpoints or OFF/ON)? Well, the glitch in ZOOMS prior to this release is that they'd be plotted as interpolated and didn't make much sense.

This glitch is fixed. In the above image, the magenta tag is stepped.

Full-Text Search on Units

This was another great suggestion from one of our best customers. Prior versions of ZOOMS could identify the name of the unit; and therefore suggest it as a user typed it and combine it with parameters to produce a trend. The problem is users often know about the unit by its description or where it is in the equipment hierarchy... not necessarily by name. With this upgrade, any word that appears in the description or hierarchy of the unit gets recognized by ZOOMS and will show up in the SERP.

In the image below, you can see that the user is interested in the 12,000-liter bioreactors in the plant and has typed in his colloquial search term, "12kL."

ZOOMS will return the units T440 through T447 as that "12kL" string appears in the descriptor (but not the name).

Incidentally, in the above query, user could have searched on "ZYM", "CCP1", or "Fermentation" and gotten results. Now, the user has to know even less about the equipment to get a search result.

Previously added features:

How to Scale X-Axis for ZOOMS Trends

2014-06-20T11:18:00.001-07:00

by Oliver Yu

In the last post, we talked about scaling the Y-Axis of ZOOMS trends. What about the X-Axis?

Well, the X-axis (or as pedantic mathematicians would call it, "abscissa") for ZOOMS is always the time-axis. As with any chart, there is an "x-min" for the minimum value of x as well as an "x-max."

In the world of time-series data, there are more colloquial names:

x-min is starttime
x-max is endtime

Unfortunately, there are also hundreds of ways of representing time:

04-Jul-1776
12/8/1941

Type Time Range into Search Box

And typing in the starttime and endtime is one way to set the time range:

ZOOMS is able to interpret a lot of date inputs, but not all of them.

Use the Trend Buttons

What happens when you've got a trend and you want to go from there?
There are 3 buttons at the bottom of each trend.

Back Arrow is an arrow pointing to the left that will take the trend one time-range into the past.
Forward Arrow is an arrow pointing to the right that will take the trend one time-range into the future.
Refresh/Revert will restore the time-range to the original as-loaded trend.

Highlight Area On Trend

You can magnify a time range by highlighting it with the mouse: clicking and holding down the mouse button at one edge of the time range and dragging the mouse to the other edge of the time range:
When you release the mouse button, the trend will zoom to the selected time range.

Summary

In summary, there are at least 3 ways to scale the X-Axis of a ZOOMS trend:

Type in the time range in the search box.
Use the "back" and "forward" buttons
Highlight an area on a trend

How to Scale Y-Axis for ZOOMS Trends

2014-06-18T00:14:00.001-07:00

by Oliver Yu

There are several improvements with ZOOMS 2.1 and the first difference you'll notice is the trend controls... the buttons at the bottom of each trend that "control" what happens to the trend.

For this post, we'll be focusing on the Y-axis scaling buttons, which are 5 o'clock position when looking at the trend from straight on. There are 3 buttons:

multi
channel
single

Multi

By default, every trend is plotted as "multi," which is an abbreviation for multiple Y-axis. Each trace on the trend gets its own Y-axis scale where the bottom is the minimum value and the top is the maximum value:

While you get a fuller sense of each trace - since each trace spans the entire range of the graph, there is no sense of what the traces are like in absolute value.

Single

The left-most button is "single," which refers to plotting all the traces on a single Y-axis. Using this method, you can get a strong sense for how the traces relate to each other.
Compared to the multiple Y-axis, we can now tell that the orange sinusoid is oscillates in a lower range than either of the red or magenta traces.

Channel

The channel Y-axis scaling divides up the trend in N horizontal slices and plots each trace in its own channel. Using this method, you can hover your cursor over the entire trend and get a feel for what the values actually are.

There are plenty of awesome web-trending applications that give you more precise control over the actual scaling of the Y-axis, and we think they serve the tactical employees (e.g. operators and engineers) well.

The Zymergi implementation of Y-axis scaling is geared towards the time-crunched strategic users like managers and directors.

See also:

Securing the Process Data Historian Search Engine: ZOOMS

2014-06-17T15:16:00.000-07:00

by Oliver Yu

When I worked at an FDA-regulated commercial biologics manufacturing facility, access to data was limited to authorized individuals; 21 CFR Part 11 (10d) specifically calls for this control so that the authenticity, integrity, and, when appropriate, the confidentiality of electronic records are assured.

Needless to say, those who operate under cGMP regulations are worried about compliance with Part 11 have concerns with software that is trying to democratize the data.

Fair enough.

It turns out that you can have cake and eat it too. That is, you can have an easy-to-use, web-based interface for your trend data AND you can limit the access to authorized individuals.

Within Internet Information Services (IIS), the web-server on which ZOOMS runs, the default setting is to enable Anonymous Login and to disable everything else.

The way to secure ZOOMS is go to click on the website under which ZOOMS is installed and select the Authentication Feature:

When you double-click on IIS Authentication, you get a setting for how you want this web-server to be secured. There are 4 options to enable/disable. The way to ensure unauthorized access is not granted is to set Anonymous Authentication Status to Disabled.

And if you happen to be running Windows Active Directory and you want to use that as the method to control access, set Windows Authentication Status to Enabled.

Assuming that you're in an environment with Active Directory, you can use Active Directory credentials to control access to ZOOMS.

Now when I attempt to access ZOOMS, here's what I get:

(I'm on a Mac using Safari to access a Windows IIS server running ZOOMS)

And only when I input valid Windows credentials am I granted access to ZOOMS. At this point, access to ZOOMS has been limited from everyone with network access to everyone with a valid domain account.

If you want further restrictions, you simply need to set up ASP.NET URL Authorization where you specify the exact role that you want to allow access.

If you want to create an Active Directory group for just your users, you can do so. If you want to grant access to an existing Active Directory group, you can do so.

The key here is to Allow first, Deny last.

The point in all this is that you don't have to sacrifice the freedom of your information for the sake of security.

For more questions, contact Zymergi Technical Support at 650-646-4996.

Semantic Search Engine for Time-Series Data

2014-05-07T22:53:00.002-07:00

It's the year 2014 and there are several updates to the Zymergi Object-Oriented Metadata Search (i.e. a semantic search engine for time-series data).

The first update is that we've moved to HTML5. This means that we're dropping support for for IE8 or less. We're proven on the latest versions major web-browsers: Google Chrome, Mozilla Firefox and Apple Safari.

Here's a screenshot of what the search page looks like. There's no equivalent of "Google Instant" but given the metadata inputted by the administrator into ZOOMS, the software is able to guess what words you are meaning to type:

The second update is that ZOOMS uses a CSS Framework called, "Bootstrap." Bootstrap was developed by Twitter and they gave it out for free. The deal with CSS Frameworks is that someone has already done a lot of the heavy web-page styling. You just need to customize it a bit to fit your look.

The other thing about Bootstrap is that it is "responsive," meaning the webpage will "respond" differently depending on the size of your screen. Responsive web design (RWD) refers to whether or not a website is optimized for mobile devices.

No, we aren't wading into the time-series-data on mobile devices. We actually believe that Transpara has a great product in Visual KPI. We actually don't expect customers to be using ZOOMS on a phone or tablet on a regular basis, but we acknowledge that it can't be helped; in those cases, you can get your trend data with ZOOMS. Besides, it came for free, so why not add this value?

The last update is that the entire product has been refactored. The changes are all under the hood, but we've applied the MVC design pattern to separate the functions for a faster, more technically supportable system.

Below, you see a search engine results page (i.e. SERP) for the search terms inputted above. ZOOMS tells you exactly which typed words has which meaning:

Reactor1 was identified as a unit.
Temperature was identified as a parameter.
Volume was identified as a parameter.

ZOOMS is figuring out what the typed words mean and combining it into a search result. In this case, someone looking for the reactor temperature and volume may be interested in a trend with each of these two traces plotted.

It is by far the fastest way for an untrained individual to get/view time-series data in a process environment.

We'll be disclosing shortly, but in addition to existing features, there are exciting updates in the coming months… like the ability to run without connecting to other data historians (yep, there's going to be an internal one).

Want a test drive?

The Zymergi Mission

2014-05-04T15:50:00.002-07:00

Zymergi sees a world where there is greater wealth than today: a world where individuals have more time to spend on what they want (rather than what they must to do).

Look around the house. Chances are, there's a clothes washer/dryer. The washer is a reactor/centrifuge combination; the dryer is a tumbler... chemical processing units. Now, you don't have to waste time at the river with a washboard washing your clothes.

Oh, there's a fridge and freezer? That's refrigeration process with replete with compressor, expansion valve and working fluid and now you can store foods longer and not have to salt your foods or waste time on extra trips to the grocery store.

Gotta a car, too? That's the chemical processing unit where you convert hydrocarbon to CO2 + water, heat to work to get from point A to B. Now you can get to that job 40 miles away in 40 minutes instead of 8 hours.

The time you get back is possible because of these chemical processes. And these chemical processes are executed by these machines (i.e. "units"). And when you look beyond the symbols of wealth (which happen to be fancier units like yachts and airplanes and sports cars), you realize that the wealthy have limitless time for leisure, while the poor do not. No truer words were spoken when they invented the phrase, "Time is money."

Zymergi thinks the path to creating more wealth for us, our customers, and their customers is to help people who run chemical processes better "see" what their units are doing and get more out of them. The easier it is to "get data," the more time our customers get to "understand the process." And the more process understanding, the greater ability to change the course of the process for greater profitability.

The world where machines do more work means a world where humans have more leisure. Automation is the answer.

Even the world of Star Trek, where does this time...

...to explore strange new worlds, to seek out new life and new civilizations, to boldly go where no man has gone before

come from?

They (the crew) are enabled by the Starship Enterprise. The starship is simply a plant that manufactures light-years. What runs the enterprise? A "computer" that is simply the starship's SCADA. Again: automation systems.

In virtually every scenario of the future we want to live in, automation... units... chemical processes are there enabling humans to have more time... more leisure. This future of greater time and wealth is the Zymergi Vision.

The Zymergi Mission: the relentless pursuit of excellence in systems integration and automation software is the straight-line that you can draw from our reality today to our vision of tomorrow.

Ron Taticek, In Memoriam

2014-04-24T15:44:00.000-07:00

UPDATE: Memorial Service to celebrate Ron's Life will be held at 3pm on Saturday, May 3rd (2014) at:

Grissom's Funeral Home - map
267 East Lewelling Blvd.
San Lorenzo, CA 94580

---
I learned today that my mentor and long time advocate, Ron Taticek, passed away. Most recently, Ron was the Director of Technology for Genentech's Vacaville Operations.

As I look back on my career, he's the guy in the last row of the bleachers rooting for me the entire way.

In 1994, Ron was completing his doctorate when I was a freshman at Cornell; our paths didn't cross until 1998 when I arrived in California - jobless and he was a Sr. Group Leader in Cell Culture Fermentation R&D (CCFR&D) at Genentech.

I knew to call Ron because our advisor, Mike Shuler, told me, "If you're headed west, look up this guy, Ron Taticek and he'll help you."

I was skeptical, but just as Shuler said, Ron didn't hesitate to help out a kid he'd never met.... whose only connection was through his Ph.D advisor. I called poor Ron once a week asking about jobs at Genentech. (Back then, I thought the tenacity = employment); I called him so many times that I know his phone number to this day: 650-ABLE-JOY.

He agreed to have an "informal interview" where he had me come to South City and he showed me around and left me with the words, "Companies like Genentech value professional scientific experience. So go and get at least a year's worth of professional scientific experience and then give me a call."

It turns out that in August 1998, I got a job and it was an awesome medical device job; the only thing wrong with it was that it wasn't bioprocess. And so in August 1999...with a year and two hours of professional scientific experience under my belt, I dialed ABLE-JOY.

Ron answered cheerfully (he had a very smooth phone voice) and said, "Great to hear from you, there's a junior engineer position up at our new plant in Vacaville: I'll forward your resume!"

I got that Associate Engineer position in October of 1999 at Vacaville and in 2003, I had the opportunity to work directly with Ron when they were transferring Avastin to Vacaville. It was because of Ron's career jumpstart that I had the opportunity to work with the cell culture greats like Jess Bergevin, Dan Stark, Bob Kiss and Cartikeya Reddy (just to name a few).

What a lot of people don't know is that Ron Taticek was the lead cell culture engineer in developing the Avastin (version 1.0) cell culture process. The reason - in part - why literally tons of bevacizumab have made it to market is because of Ron's work developing the cell culture process.

In October 2004 when Ron became the Associate Director of Ferm(entation) MSAT, he asked me to join him in South City. I accepted and reported through him until I left Genentech in 2007, creating awesome tools, doing awesome support, and working with universally awesome people.

Since 2007, I'd check in with Ron a few times a year; we even sat down to have salads (apparently, our respective wives had put us on diets). It was then that I realized I've known him for 15-years. And it was then that I realized how "there" he was for me since I was some fresh-out-of-school kid.

Jan 2013 was last time I would see Ron in person... and I'm glad I got the chance to tell him how much I appreciated him. We parted ways like usual: promising to keep in touch as he ambled off to another meeting.

If there was anything left for me to say to you, Ron, it would be:

Your life's work mattered to millions of people. Your mentorship and advocacy profoundly impacted my life. And your true legacy: N & N will miss you dearly. Though you left us too soon... and it's not fair... you rocked this life by making a difference.

Rest in peace.

Lowest Number of FDA Inspections Since 2008

2014-03-28T06:00:00.000-07:00

FDAzilla has reported that the FDA conducted 39,774 inspections for the year ending 2013.

I guess on any given day, there are at least 109 auditing teams running around the world inspecting our GMP-regulated existence.

But it turns out that 39,774 inspections is a 5-year low for the FDA. As FDAzilla puts it, this is the first time since 2008 that the FDA has dipped under 40,000 inspections per year.

Was it the government shutdown? Was it the sequester? Who knows?

As we ramp into prime inspection season (Q1/Q2 are known times when there's a lot of FDA activity), how are you guarding against getting dinged by the FDA?

ZOOMS 2 - Fastest Way to View Trends

2014-03-25T22:30:00.003-07:00

So in addition to biologics manufacturing commentary, it turns out that Zymergi is actually a for-profit business that provides software, consulting and technical support.

For 2014, we're pretty much booked on the technical support side, but I wanted to take some time to talk about our software products.

Our flagship product is ZOOMS, which is an acronym for Zymergi Object-Oriented Metadata Search. And in 2014 - thanks to Edward Snowden, more people know what metadata is than in 2008 when we started ZOOMS (v1.0)

So why are we interested in searching metadata? Well, let's take a step back. When working the front lines of campaign monitoring and process support, we noticed that viewing trend data (i.e. tag values plotted against time) was the principal component of process understanding. The more trends a person reviewed, the more process knowledge they gained and the more they understood normal from abnormal.

And in all that time, very few people actually learn to speak "Automationese."

"Hey, did you see that weird thing going on with T100.GLOBAL.CLX.AI05.PV?"
- No one ever

In the automation world, everything is a tag. In the Manufacturing Sciences world, everything is about a measurable parameter within the process. So when you listen to the process scientists and engineers talk, it's always about some parameter (e.g. "Optical Density") in some unit (e.g. "Tank 100"). That right there is the metadata of a tag.

The tag takes care of the Y-axis on the trend. What about the X-axis?

The X-axis deals with time-windows: start times and end times and the metadata around the time-windows are called, "batches." Specifically using S88 terminology, people involved with campaign support are interested in Unit Procedures, a.k.a. "unitbatches."

I'll leave the formal definition of "unit procedure" up to the automation gurus, but to a Manufacturing Sciences data engineer, a unit procedure is a process that happens on a piece of process equipment.

So say you're making rituximab in production bioreactor T100 using the version 1.1 process and batch R2D201 ran from 20-Dec-2013 to 28-Dec-2013... that there is a unit procedure:

batchid	unit	product	procedure	start time	end time
R2D201	T100	rituximab	production culture version 1.1	20-Dec-13	28-Dec-13

The metadata around this time-window (i.e 12/20/2013 to 12/28/2013) are as follows:

R2D201
T100
rituximab
production culture version 1.1

So it stands to reason that if an internet user who knows nothing about a subject can type keywords into Google and get the most relevant results on that subject; that in 2014, a process citizen who doesn't know too much about the process ought to be able to type some keywords into a webpage and get some process trends.

And now they can: Introducing ZOOMS 2:

Learn More

Yes, We're Still Alive

2014-03-18T22:03:00.002-07:00

Yup, we're still alive around here. Crazy busy with some exciting things afoot.

Next week, we're at the OSIsoft PI User's Conference 2014 in San Francisco:

So if you're around and want to meet up, come find me @zymergi.