A Novel Variable Selection Method for Spectrum-Based Soft Sensor Development

A Novel Variable Selection Method for Spectrum-based Soft Sensor Development

Jangwon Lee*, Jin Wang*, Q. Peter He*⁺

*Department of Chemical Engineering, Auburn University, Auburn, AL 36849 USA

(JL: jzl0164@auburn.edu; JW: wang@auburn.edu; ⁺QPH: qhe@auburn.edu)

In the last few decades, spectroscopic techniques such as near-infrared (NIR) spectroscopy have gained wide applications in oil and gas industry. As a result, various soft sensors have been developed to predict sample properties from its spectroscopic readings. Because the readings at different wavelengths are highly correlated, it has been shown that variable selection could significantly improve a soft sensor’s prediction performance and reduce the model complexity. Existing variable selection methods focus on selecting the variables (i.e., wavelengths or wavelength segments) that are strongly correlated with the dependent variable to improve the prediction performance. Although many successful applications have been reported, such variable selection methods do have their limitations, such as wavelengths selected do not have clear relationship with the chemical bounds or functional groups presenting in the sample. As a result, these methods could face robustness issue and their performances can be highly sensitive to the choice of training data, and deteriorated performance when testing on new samples.

In this work, we present a novel variable selection method that integrate the variable stability and variable importance in the projection (VIP) score and transform them into the probability of variable importance. By incorporating the random selection principle from the genetic algorithm (GA), variables are randomly selected based on the variable importance probabilities, which prevents certain wavelengths never get selected if deterministic variable importance criteria are used. Using several case studies including gasoline and biodiesel, we compare the performance of the proposed method to the existing variable selection methods, including competitive adaptive reweighing sampling (CARS), variable importance in the projection (VIP), genetic algorithm (GA), etc. We show that the proposed method has several advantages: (1) significantly better performance in all case studies; (2) identifies wavelength segments that are clearly related to the chemical bounds and functional groups presenting in the sample; (3) better robustness; (4) fewer parameters and much simpler tuning/training than GA.