Process Data for the Rest of Us

I first started using OSIsoft's PI in 1999 when I got hired as a fermentation engineer at the world's largest cell culture plant (at the time, 144,000-liters).

I remember the first week vividly.  Me - not knowing anything about how read cell culture trends - watching my boss run his cursor over PI trends, nodding his head and then running off to meetings telling the operations staff what's happening in the bioreactors.

Over the years, I've put my eyeballs on thousands of cell culture runs and became an expert on the matter.  Yet, no matter how many trainings gave to tech(nician)s, supe(rvisor)s, and managers to increase the population of process experts at the company, I got the sense that ultimately it was still my team's (i.e. manufacturing sciences') job to know what's happening to the cell culture runs in-progress.

OR, maybe not...

The VP of the plant had the PI administrator write a script to open up PI ProcessBook and snapshot current trends as a images and put it on a website (remember, this was back in 1999).  Clearly management recognized the value of these trends, but was just too much activation energy to get PI data to "The People."

So when I left my awesome company (and awesome job), I set out to do one thing:



And by PI, I mean "process data."

Google had already pioneered the one text-box, one click format for accessing website data.   So why is it that cGMP citizens can find information on the internet faster than they can find critical process data from within the enterprise?

This is why I bothered creating ZOOMS and this is why I think that there's a place for ZOOMS in the competitive space of trend visualization on the web.

It actually wasn't until the Jared Spool lecture at this year's OSIsoft PI User's Conference that I learned how to better enunciate this creation.  


A quick recap:

The bottom of the escalator is where you are if you know nothing about an app (new users live here).  The top of the escalator is if you know everything about the app (developers live here).

Current knowledge is where the user is today; and target knowledge is where a user needs to be to know enough about the application to perform his duties.

Mr. Spool tells us that intuitive design happens when the knowledge gap (the difference between target knowledge and current knowledge) is zero.

A key observation is that the more powerful and feature-rich the tool, the higher up the target knowledge is...and the harder the knowledge gap is to close.

The success of Google (which is to be replicated with ZOOMS in the process trend visualization context) is a modest number of features in order to lower the target knowledge... and thus diminish this knowledge gap and achieve intuitive design.

There are plenty of feature rich PI trend visualization tools for process experts.  ZOOMS is process trends for the rest of us; in other words: PI for the "non-user."

At the end of the day, it's People, Process, and Technology... in that order.  You can buy awesome technology, but if only a small minority of your people use it, you're neither capitalizing on their potential, nor your process.

The Zymergi Mission

Zymergi sees a world where there is greater wealth than today: a world where individuals have more time to spend on what they want (rather than what they must to do).

Look around the house. Chances are, there's a clothes washer/dryer. The washer is a reactor/centrifuge combination; the dryer is a tumbler... chemical processing units. Now, you don't have to waste time at the river with a washboard washing your clothes.

Oh, there's a fridge and freezer? That's refrigeration process with replete with compressor, expansion valve and working fluid and now you can store foods longer and not have to salt your foods or waste time on extra trips to the grocery store.

Gotta a car, too? That's the chemical processing unit where you convert hydrocarbon to CO2 + water, heat to work to get from point A to B. Now you can get to that job 40 miles away in 40 minutes instead of 8 hours.

The time you get back is possible because of these chemical processes. And these chemical processes are executed by these machines (i.e. "units"). And when you look beyond the symbols of wealth (which happen to be fancier units like yachts and airplanes and sports cars), you realize that the wealthy have limitless time for leisure, while the poor do not. No truer words were spoken when they invented the phrase, "Time is money."

Zymergi thinks the path to creating more wealth for us, our customers, and their customers is to help people who run chemical processes better "see" what their units are doing and get more out of them. The easier it is to "get data," the more time our customers get to "understand the process." And the more process understanding, the greater ability to change the course of the process for greater profitability.

The world where machines do more work means a world where humans have more leisure. Automation is the answer.

Even the world of Star Trek, where does this time...

...to explore strange new worlds, to seek out new life and new civilizations, to boldly go where no man has gone before

come from?

They (the crew) are enabled by the Starship Enterprise. The starship is simply a plant that manufactures light-years. What runs the enterprise? A "computer" that is simply the starship's SCADA. Again: automation systems.

In virtually every scenario of the future we want to live in, automation... units... chemical processes are there enabling humans to have more time... more leisure.   This future of greater time and wealth is the Zymergi Vision.

The Zymergi Mission: the relentless pursuit of excellence in systems integration and automation software is the straight-line that you can draw from our reality today to our vision of tomorrow.

  

ZOOMS 2 - Fastest Way to View Trends

So in addition to biologics manufacturing commentary, it turns out that Zymergi is actually a for-profit business that provides software, consulting and technical support.

For 2014, we're pretty much booked on the technical support side, but I wanted to take some time to talk about our software products.

Our flagship product is ZOOMS, which is an acronym for Zymergi Object-Oriented Metadata Search.  And in 2014 - thanks to Edward Snowden, more people know what metadata is than in 2008 when we started ZOOMS (v1.0)

So why are we interested in searching metadata? Well, let's take a step back. When working the front lines of campaign monitoring and process support, we noticed that viewing trend data (i.e. tag values plotted against time) was the principal component of process understanding. The more trends a person reviewed, the more process knowledge they gained and the more they understood normal from abnormal.

And in all that time, very few people actually learn to speak "Automationese."

"Hey, did you see that weird thing going on with T100.GLOBAL.CLX.AI05.PV?"
- No one ever

In the automation world, everything is a tag. In the Manufacturing Sciences world, everything is about a measurable parameter within the process. So when you listen to the process scientists and engineers talk, it's always about some parameter (e.g. "Optical Density") in some unit (e.g. "Tank 100"). That right there is the metadata of a tag.

The tag takes care of the Y-axis on the trend. What about the X-axis?

The X-axis deals with time-windows: start times and end times and the metadata around the time-windows are called, "batches." Specifically using S88 terminology, people involved with campaign support are interested in Unit Procedures, a.k.a. "unitbatches."

I'll leave the formal definition of "unit procedure" up to the automation gurus, but to a Manufacturing Sciences data engineer, a unit procedure is a process that happens on a piece of process equipment.

So say you're making rituximab in production bioreactor T100 using the version 1.1 process and batch R2D201 ran from 20-Dec-2013 to 28-Dec-2013... that there is a unit procedure:

batchid	unit	product	procedure	start time	end time
R2D201	T100	rituximab	production culture version 1.1	20-Dec-13	28-Dec-13

The metadata around this time-window (i.e 12/20/2013 to 12/28/2013) are as follows:

• R2D201
• T100
• rituximab
• production culture version 1.1

So it stands to reason that if an internet user who knows nothing about a subject can type keywords into Google and get the most relevant results on that subject; that in 2014, a process citizen who doesn't know too much about the process ought to be able to type some keywords into a webpage and get some process trends.

And now they can: Introducing ZOOMS 2:



Learn More

ZST is 67% faster.

For those of you who don't know, the ZST (Zymergi SQL Tool) is a Windows-app that gets your database information and puts it in any format you want.  Typically, customers want to convert their relational data to HTML5 so that they can view (and act on ) it with a desktop or mobile-device browser.



Well, there's good news.  ZST is now 67% faster.  Now, this increased speed may not matter in an enterprise environment where apps are feature-rich, but slow.  But for our customers with external-facing websites, more speed = more revenue.

So why would the enterprise customer want ZST?

Well, the first thing is that this is commercial off-the-shelf (COTS) software.  Secondly, you're essentially adding search and browsing capabilities to your existing relational data.  And ultimately, you're looking to institutionalize tribal knowledge.

• Boosting speed of ZST
• ZST Support Page

MSAT to Automation, MSAT to Automation. Come in, Automation

by Oliver Yu

When I was running cell culture campaign monitoring and we were using PI to review trends to understand physical phenomenon, there were times the trends didn't make any sense.

After digging a little, we found out that the data was simply recording with too little resolution either in the Y-direction or the X-direction.

Here's a short blog post describing the words to say to Automation (as well as some perspective) to get some more resolution in your data.

Compression Settings

If the data seems sparse in the Y-direction (e.g., you expect to see oscillation but only see a straight line), it could because the compression settings are such that too much data gets filtered out. For OSI PI, there are two types of filter settings: (1) exception and (2) compression.

Exception is responsible for filtering out repeat data between the data interface and PI.

Compression is responsible for filtering out repeat linear data within PI (between the snapshot and the archive).

Every point attribute can be viewed from within PI ProcessBook. And if you find that your exception or compression settings are too wide, view them within PI and make a note of what they ought to be, then go on and tell your Automation team.

In my experience, you'll find a reluctance within Automation for changing the individual settings on points. Generally, there is a standard or a rule that is applied uniformly to the set of points. For example, you're using Broadley-James pH probes in both cell culture and purification and we (cell culture) ask for a 0.005 compdev on bioreactor pH probes, shouldn't the buffer prep pH probes also be set to 0.005 compdev?

Automation has to balance the tension between customer (your) needs as well as defensible system configuration.

Generally speaking, you're going to be asking for changes to compdev of excdev point attributes, and if you're asking for more data to be collected, you want these numbers to be smaller.

Scan Rate Settings

What if after improving compression to filter out less data you still find that there is not sufficient resolution in the data to observe the physical phenomena that you know is happening? Well, the only place left to check is in the scan rate of the data... sparseness of data along the X-axis.

A point's scan rate is set based on a list of pre-defined intervals in the data interface. The data interface is a piece of software that transfers data from the source (control system) to the destination (data historian). If the interface is configured well, it will have sensible scan rates:

1. Every second
2. Every 2 seconds
3. Every 5 seconds
4. Every 10 seconds
5. Every 20 seconds
6. Every 30 seconds
7. Every minute
8. Every 5 minutes

It isn't always like this, but very often you'll see these intervals. The scan rates are defined in the interface configuration file and once set, they rarely change. The way it works is this, the first entry in the interval configuration is gets assigned: 1... the second entry: 2... the third entry: 3.

And whatever you set the point's location4 attribute is what it's scan rate is.

So suppose 00:00:05 is the third entry. Then a point whose location4=3 has a scan rate of every-5-seconds.

In a lot of cases, you simply tell your PI administrator you want the scan rate to be "every second," after which he's on the hook for looking up that scan rate in the interface. But FYI, if they said they made the change to the point but the location4 attribute is the same before and after, they're just BSing you.

There are a lot of considerations that need to get balanced when figuring out this stuff.  What's working against you is the default settings that come out-of-the-box with PI... as well as a generation taking the path of least resistance.

Get FREE Paper on PI Compression

10 Ways to Tell If You Suck at Cell Culture Support

Here are 10 ways to tell if your support of large-scale cell culture, well, sucks:

1. Key performance indicators.
   You don't know what the right KPIs are for cell culture, but you're 100% certain that it's titer.
2. Proven Acceptable Limits.
   You don't have any defined for your critical process parameters and you failed to demand them of Process Development.
3. Control charts. You're not using them or you don't know how to make them, and your bar graphs are just fine, thankyouverymuch. They're not just fine and it's because you can't identify:
4. Special cause vs. common cause variability.
   You investigate common cause variability because that titer seemed too low or too high.
5. C_pK. You don't know what process capability is and you're not calculating them.
6. Histograms. You aren't looking at the distribution of your KPIs.
7. Bi-variate Analysis.
   Linear-regressions, ANOVA, Tukey-Kramer.  You have no idea what this stuff is, 我還不如寫中文.
8. Multivariate Analysis.
   You're not doing these and when you do, Y-responses are treated as X-factors.
9. Local Lab. You don't have a local MSAT lab to run satellite experiments to confirm the hypothesis generated from the plant.
   
   A lot of people assume that you can use the resources of a distant process development lab; but then again, a lot of people like blood sausage.
10. Excel. You're still using Excel to store data. You're still using Excel to analyze data. If you're looking to play varsity cell culture support, you really need to be using a cell culture data management system.

by Oliver Yu

See also:

• MSAT plant support: local data
• Campaign monitoring v. Process Improvement
• MSAT consulting

How to Use ZOOMS (for OSIsoft PI System)

This was where our OSI PI Search Engine was as of 2008



In a single textbox, you could type in any set of words... just like Google.

And after you typed in a few concepts like:

• Reactor1
• Temperature
• Concentration
• Volume

ZOOMS could figure out that Reactor1 was a unit while Temperature, Concentration and Volume were process parameters.

Not needed, but if you felt like typing in a time-window (in OSIsoft PI-ese), it would simply show you the trend that you meant to see.

Perfect for people who don't have PI ProcessBook installed... which is basically management, scheduling, QA, instrumentation, process engineering... you know, everyone.

Get The ZOOMS Brochure

PI DataLink: Cannot Insert Trend

There's this feature in PI DataLink where you can insert a PI Trend:




It's a weird feature.  As in... if you wanted to see a trend, why not use PI ProcessBook?

See the PI ProcessBook Best Practices For BioPharma.

Whatever the case, if you click that link and nothing happens, then you have a problem with your PI DataLink.

It's a symptom whose disease is solved by deleting either of these two folders in order to fix the problem:

• For Windows XP:

  C:\Documents and Settings\%USERNAME%\Application Data\Microsoft\Forms

• For Windows Vista, Windows 7:

  C:\Users\%USERNAME%\AppData\Roaming\Microsoft\Forms

It's actually the same exact root cause as those Compile error in hidden module: modAddin errors.

Death to Cell Culture Spreadsheets!

by Oliver Yu

I've been around a few cell culture operations in my time... fermentation, too. And ubiquitous to virtually every operation - be it lab, plant or pilot-plant - are these Excel spreadsheets where research assistants or operators input "cell count" data.

A "cell count" is when a sample of the cell culture fluid is taken from the miniferm/bioreactor. This sample prepped with trypan blue, dropped on a hemocytometer and counted by an operator under a microscope.

Pipetting fluids into a hemocytometer by Jacopo Werther

The trypan blue is absorbed by the carcasses of dead cells while an integral cell membrane of the live cell is impervious to trypan blue.

The operator counts the number of blue dots (dead cells), the number of clear dots (live cells) and thus determine the viability (% of live cells).

From the total number of cells and the sample volume, the operator can estimate the total cell population in the bioreactor.

Also while we have this fresh cell culture sample, a small volume is separately sent through machines that determine the concentration of glucose, lactate, sodium and ammonium, pO2, pCO2 as well as the pH and osmolality.

All this cell count and cell culture metabolite data gets inputted into spreadsheets where we can compute the rate of cell growth so that we can figure out when to transfer/harvest the cell cultures.

Because of the detailed data, these spreadsheets become the crown jewels of process understanding. Cell growth and metabolite profiles at large-scale are benchmarked against their small-scale equivalents. New experimental designs are determined from the results of data from the previous experiment.

And all this is possible because of these cell culture spreadsheets.

The thing is, there are severe limitations to running your core enterprise information this way: spreadsheets are not a viable way to manage data: databases are.

And it is for this reason that we are creating the world's first commercial off-the-shelf cell culture database to manage this core data.

Built by cell culture engineers for cell culture engineers.

Sign up for the beta

How Does Someone Go About Stealing My Process (PI) Data?

One savvy prospect asked that question, and it's a great question:

A lot of user requirement specifications (URS) or detailed design specs require data security.

And if keeping your process data secure was not in your company's best interest, it is a requirement to be compliant with the infamous 21 CFR Part 11- the regulation governing how the conditions under which FDA views electronic data the "same as paper."

21 CFR Part 11, Subpart B mandates that the manufacturer

(d) Limiting system access to authorized individuals.

And demands that the use

(k) Use of appropriate controls over systems documentation including: (1) Adequate controls over the distribution of, access to, and use of documentation for system operation and maintenance.

In response to securing data and the trade secrets embedded in your company's process data, a lot of time goes into securing the server... We:

• Create Active Directories and we map them to PI Identities.
• We have PI Trusts to authorize specific computers
• We bifurcate our networks and put PI behind fortinets and firewalls.

All this protects the server - but none of it protects your data from theft if you just back up your PI data to a network drive.

You see, if I wanted to steal your company's data, I wouldn't go anywhere NEAR your PI server. I'd go looking for your backup files.

Your backup files contains all the files needed to restore your PI system. If I get these files, I can recreate your PI system on my own box where I have admin rights and mine your data all day long.

As for that prospect... they're now a customer.

Excdev/Compdev vs. Excdevpercent/Compdevpercent

There are two ways of configuring data compression in PI... and there seems to be a lot of confusion surrounding it:

1. As absolute value
2. As percent of span

Absolute Value - excdev, compdev

Most people understand setting excdev and compdev best. pH is a parameter that goes from 1 to 14. If you think increments of 0.01 are significant to your process and you think that your pH probe can measure pH that accurately, then you ought to set excdev = 0.01, compdev = 0.005 (see the whitepaper).



Excdev and compdev hold values measured in the engineering units of the parameter being measured. In the previous example, 0.01 pH units. If your tag is measuring temperature, it'd have engineering units of degC or degF or K.

Because excdev and compdev are expressed in terms that you can relate to, this is the most popular method of speciying the compression settings.

Percent of Span - excdevpercent, compdevpercent

The less popular method of setting compression is by specifying the compression settings as a percentage of span.

What does "Percent of span" mean?

Every PI point has a SPAN attributes. The SPAN is the largest value you expect the tag to archive. In our pH example, there is no such thing as a pH greater than 14, so people often set SPAN = 14.

When you specify compression settings with excdevpercent/compdevpercent, what you do is you set it equal to the percent of the range. For example, suppose we wanted to set the exception to 0.01 pH units. We can do that by doing this calculation:

excdevpercent = excdev / ( SPAN ) * 100

excdevpercent = 0.01 / 14 * 100

excdevpercent = 0.0714

As a caveat, this number is computed as a percent of SPAN and not as a percent of the range (SPAN - ZERO).

This seems like a lot of trouble - why bother doing this calculation when you can set the actual value? Here's why:

If you have a lot of points to configure, you don't have time to go through each one. Also, more often than not, your user requirements are specified as a percent of the range of the instrument. This is why it is sometimes faster, more efficient to configure data compression with the exc-/compdevpercent settings.

What happens if I specify both?

Under the hood, if you specify EXCDEV or COMPDEV, the PI Server will compute and store the values as EXCDEVPERCENT and COMPDEVPERCENT. If you specify them as percentages, they'll simply be stored as the percentages.

If you happen to specify both the -dev and -devpercent, the -devpercent will override the -dev settings.

SQLite Insert Rate ranges from 50 to 250 inserts/second.

SQLite is a file-based database. It's used in browsers, it's used on your iPhone. We use it as a placeholder for RDBMS. We're troubleshooting a data-write problem and it turns out that the problem is somewhere else.

Our hardware is WinXP 32-bit, Lenovo laptop running SQLite.NET and plotted here is the number of rows inserted per second.



If you have an application that needs no more than 30 inserts/second, SQLite works just fine.

Troubleshooting OSI PI compression

I just got back from a client where I was getting a copy of their PI server configuration. My customer offhandedly asked me about the size of his archives- "Is it normal to use 600 megabytes every 2 days?" Off-the-bat, I could tell there was something wrong with the data compression of this system. This PI server was < 5000 points and it collects data from about 20 production units.

Customers with similarly sized-plants and run-rates burn through 600 megabytes a MONTH. The largest cell culture facility west of the Mississippi goes through 1000 megabytes a month, so this particular client was definitely looking at something obvious and something that is statistically outside of normal.

Here's how I troubleshot it:

Look for compressing = 0

The PI Point attribute that determines if the data to a point is to be compressed is the compressing attribute. This value ought to be 1. A lot of people like turning this off for low-frequency tags but it's like unprotected copulation - you're not necessarily going to get pregnant, but there's a chance that an errant configuration runs your system down.

Look for compdev = 0

The compdev point attribute determines what data makes it into the archive. Compdev settings ought to be 0.5 instrument accuracy according to solid recommendations on PI data compression for biotech manufacturers. If you find yourself loathe to define this number, I'd make compdevpercent = 0.1. What this does is it eliminates repeats from the archive.

Use PI DataLink to look for diskspace hogs

The easiest way to identify which tags are the culprit is to pull it up in PI DataLink and use the Calculated Tag feature to find tags with high event count. Start by looking in the last hour... then in the last 12-hours, then last day, then last week. The blatant offenders should be obvious within even 1-minute.

In the case of my customer, he had 7 tags out of about 5000 that was uncompressed. Each of these 7 tags was collecting 64 events/sec. 3840 events/minute. 5.5 million events per day. All told, these 7 tags were recording 39 million zeros into the archive per day... burning through diskspace faster than Nancy Pelosi likes to spend your income.

Modern hardware has made these problems insignificant, but burning through diskspace is a latent problem that rears its head at the most inopportune moment.



and learn about OSIsoft's data compression settings and what they ought to be.

Version Control

GMP environments require very strict control. Whether or not regulations mandate them, controlling the process and the manufacturing formula is, frankly, a good idea.

The problem with controlling GMP documents and GMP control-system recipes is the onerous change-control process that has evolved over the years. And my observation of this change-control process is that it was design by regulators and not computer scientists.

It's important to bring out computer scientists because managing source code is a core function of companies that develop software. In fact, version control is so sophisticated that it has become distributed and there are distributed version control systems (like Veracity DVCS) that can help cGMP-regulated companies manage their GMP documents and recipes.

I actually have yet to see version control software applied to GMP industries probably because people don't understand it nor how it works. In fact, only recently did I get a primer on it.

That primer came in the form of the beginner book on version control called "Version Control By Example" by a fellow named Eric Sink. And while he may not have written this book for QA managers in big pharma... every QA/CC manager in big pharma ought to have a copy of his book.

It goes through the evolution of change control. It talks about central repositories and how the industry is moving towards distributed repositories. It imbues the newbie reader with a shared vocabulary so that people who understand the importance of version control can express their needs to people who write version control software. Get a print version from Amazon here.

At Zymergi, we believe that future of QA change control and document management is to turn to proven methods and technology. And looking to the technical folk in the software version control space is where I think the robust solution lies.

Upgrading to PI Server 2010 for PI Batch Users

PI Server 2010 is the latest PI server offering from OSIsoft. I don't know for a fact, but this seems like a marketing nomenclature to emulate Microsoft's Office 2007, Windows Server 2008...etc. It'll remind me the way my Office 97 makes me feel 14-years behind the times.

Whatever the case, the internal versioning system remains the same: PI Server 2010 is still version 3.4.385.59. What is drastically different is that PI Server 2010 requires (mandates/coerces) users to have PI Asset Framework (PI AF).

Ok, so what's PI AF? PI AF is essentially a scaleable PI Module Database, and what makes it scaleable is that it's built on Microsoft SQL Server. This means that you need to have SQL Server (or SQL Server Express) installed somewhere. Over time, the PI Module Database will be deprecated in favor of PI AF. So the default behavior of the PI Server 2010 is to copy the ModuleDB to PI AF and put it in read-only mode.

The problem is that there are PI applications that use PI ModuleDB that have NOT been moved to PI AF... for us in biotech, that's PI Batch. So in order to keep these customer happy, OSIsoft provides an option for PI AF to be synchronized with PI ModuleDB, but this requires preparation. The PI MDB to AF Preparation Wizard is what achieves this and this wizard comes with PI SMT 2010... which means you need to install PI SMT 2010 next.

Once the PI MDB to AF Preparation Wizard is run and all the errors fixed, you can proceed with upgrading your PI server to PI Server 2010.



This gives you the overview of upgrading to PI Server 2010. This upgrade is not as straightforward as previous upgrades because of the AF mandate. The devil is in the details and you should run through this process several times before apply it in the GMP environment.

OSI PI Tag Compression Setting Recommendations

OSIsoft PI Tag data compression has been the subject of considerable debate for the regulated markets over the years. So much so that I wrote a paper about it almost 5-years ago.

Get the paper here: http://zymergi.com/osi-pi-data-compression-paper.htm

You see, GMP managers are reluctant to discard data the FDA calls "Original data." Yet other managers rationalize that disk-space is cheap... as are IT resources, so why not collect as much data as you can?

There are many reasons, but from the perspective of the user who has to go through that data, we want enough to study the process, but not so much that we're buried in the haystack.

This is where rational settings for data compression come into play. Data compression on OSI PI servers let you conserve on administration/IT costs, filter out useless data, while providing your scientific staff with "the right amount" of data for continuous process improvement.

By the same token, if you have thousands of PI Points, you may not have the resources to rationally examine each process measurement and come up with customized excdev and compdev settings for each.

So what’s the answer?

We think the answer is to understand that every instrument has error… the range of error is called, “Instrument accuracy.” Repeat values that are within that range are not different than the first value and ought to be filtered out.

We also think that if points are removed along a straight line that those points ought to be within one instrument accuracy of that line.

It turns out that setting excdev to instrument accuracy and compdev to 0.5 excdev is where this happens.

Get our FREE 3-page paper

It's an easy read.