TheSNPpit - A Multi Panel SNP Data Management Package

TheSNPpit is a database system for managing multi panel SNP genotype data. It has implemented three new ideas: highly compressed vector storage in a database, set based manipulation, and a very fast export written in C with Perl as the base for the framework and PostgreSQL as the database backend. Its novel subset system allows the creation of named subsets based on the definition of SNP (through MAF, no calls, and chromosomes) and sample lists at basically no storage costs, thus avoiding the issue of proliferating file copies. The named subsets are exported for down stream analysis.

Plink ped and map files are processed as inputs and outputs with a 2 bit storage of SNPs in the database allowing to fill 4 mio SNPs/1MB. TheSNPpit can handle SNP data from panels of any size. To investigate performance scaling, a database with more than 18.5 mio samples has been created with 3.4 trillion SNPs from 12 panels ranging from 1000 through 20 mio SNPs resulting in a database of 850GB. The performance scales linearly with the number of SNPs and is largely independent from panel and database size. Import speed is around 6 mio SNPs/sec, export between 60 and 120 mio SNPs/sec.

A completely generalized procedure allows storage of phenotypes, which will automatically be exported with the SNP genotypes for further downstream processing. Being command line based, imports and exports can easily be integrated into pipelines. TheSNPpit is available under the Open Source model.