Assessment of Partitioning Algorithms for Modeling Spectral Reflectance Data over Visible, Near-Infrared, and Short-wave Infrared Region

English Abstract:

Soil assessment using vis-NIR diffuse reflectance spectroscopy (DRS) operating in 350-2500 nm range has been proved to be very effective technique in last few decades. In soil-science literature, partitioning algorithms, used for partitioning the data into calibration and validation set for predictive modelling, have been compared in the light of representativeness (measured by Kullback-Leibler Divergence between whole and calibration-set in this study), and regression performance, also the effect of calibration-set size has been assessed. But, the accuracy evaluation of DRS models would be more meaningful only if a ‘uniformity’ between calibration and validation dataset exists and is rarely accounted in soil DRS literature. This study compares the performance of different data partition algorithms namely, Kennard-Stone (KS), sample set partitioning based on joint x-y distances (SPXY), Optimizable K-Dissimilarity measure (OptiSim), Conditioned Latin Hypercube (cLHS), Sorting and KS based on clusters (KSC, newly introduced in this study) in terms of uniformity between data subsets (measured by an index newly introduced in this study) and regression statistics of external validation. The study was conducted using five different soil properties (organic carbon, sand, clay, extractable Fe and geometric mean diameter) from three distinctly different soil groups (Vertisols, Alfisols and Laterite).

Our results show that KS gives lowest uniformity (means highest Uniformity-index) for all soil attributes, while cLHS, Sorting and KSC give highest uniformity. Our results are not conclusive in terms of regression performance. However, prediction accuracy of sand content in laterite soils is improved significantly when KSC (RPD = 2.29) is used, similarly prediction accuracy of GMD of alfisols is improved significantly by using cLHS (RPD = 2.59) and Sorting (RPD = 2.28). Overall, our results indicate that any of the cLHS, Sorting and KSC can be used for data partitioning.

Key-words: Uniformity, KS, Spxy, OptiSim, Sorting, cLHS, KSC