Regression Strategies for Large Data Sets

ABSTRACT for AIChE 2017 Spring Meeting

Regression Strategies for Large Data Sets

James C Cross III

The MathWorks, Inc.

james.cross@mathworks.com

617-605-5818

Plant operators and engineers are increasingly using historian data to gain insight into the relationships between and amongst process parameters and product attributes.  A universal goal is the determination of the operating conditions that produce the highest product output and/or quality per unit cost.  Models provide insight, but seldom account for myriad physical nuances which sometimes exhibit significant influence.

The quantity of data that must be analyzed in pursuit of the sought inferences is invariably large, consisting of hundreds or thousands of quantities sampled at many millions of points in time.  It is seldom possible to process data sets of this size in computing resource memory.

A number of frameworks for partitioning large data sets have been developed and popularized, though adoption by industrial companies has been notably slower than by IT-centric enterprises.  This paper endeavors to demystify the processing of out-of-memory data for an engineering audience.

Regression, which is fundamental in data analysis, is used to motivate use of the techniques.  This paper begins with computing simple statistics.  Subsequently, some strategies for matrix manipulations (relevant to both direct and iterative methods) are discussed.  Finally, an illustrative example of a large data set regression is presented.