Welcome to gwas-norm#
The gwas-norm repostory and package aims to make is as simple as possible to standardise the format of GWAS summary statistics files so they can be used in downstream applications in a uniform way.
There is a massive amount of GWAS summary level data in the public domain. This covers many different phenotyes from molecular trails such as eQTLs and pQTLs to disease traits. Unfortunately, there has never really been an agreed standard for data sharing of GWAS summary level data and the available datasets exhibit substantial heterogeneity and are of varying quality, with some being directly usable and others bordering on deliberate obfuscation of the data within them.
gwas-norm provides an interface where users can define the attributes of the dataset they want to normalise, and use that definition to perform the normalisation in a scalable way. With gwas-norm you can normalise a single dataset in an interactive shell, or you can scale it up to many thousands of datasets (for example an eQTL/pQTL study) and run in parallel with hpc-gwas-norm. It will handle some of the most common scenarios and produce a flat file with:
A standardised column order
Uniform genome assembly
Variant IDs
Basic functional annotations
Independent variant assessment (not implemented yet).
What is gwas-norm#
When you install gwas-norm, you get access to both programs that you can run from the unix command-line and a python application programming interface (API) that you can use to interact with generalisable components of the package, or use to integrate gwas summary statistic normalisation into your own pipelines.
Whist it is anticipated that most users will want to use only the command line scripts, gwas-norm and hpc-gwas-norm (the code formatting of gwas-norm distinguishes the program from the package of the same name), however, both the command line endpoints and the API are documented.
Next steps…#
To get gwas-norm up and running you will want to:
Contents#
Setup
- Getting Started with GWAS Normalisation
- Setup a configuration file
- Environment variables
- Building the mapping files
- Introduction
- Version information
- The populations in the mapping file
- The mapping VCF file format
- Download reference genomes
- Installing helper bash scripts for building the mapping files
- Mapping file data
- ALFA
- GNOMAD2 exome allele counts
- GNOMAD3 allele counts
- 1000 genomes allele counts
- UKBB SNPstats files
- Merging all the data sources
- Running VEP
- Splitting the mapping file into common/rare
- The variant synonyms file
- Installing Ensembl VEP
- Updating Ensembl VEP
GWAS normalisation
- Overview and usage
- Key features
- Overview
- Usage
- The XML metadata file
- Prerequisites
- The XML elements
- The
<gwas_data>root - The
<study>/<study_file>elements - The
<analysis>/<analysis_key>elements - The
<file>element - The
<phenotype>element - The
<caveat>element - The
<definition>element - The
<column>element - The
<info>element and info attributes - The cohort elements
- The population elements
- The
<test>element - Example XML files
- The
- Input file columns
- The GWAS effect type
- The GWAS analysis type
- The directory structure of the normalised data
- Output files
- The XML metadata file
Variant mapping
Programmer reference
- Command-line endpoints
- gwas-norm API
gwas_normpackagegwas_norm.gwas_normgwas_norm.processorsgwas_norm.configgwas_norm.commonChrPosSpecMsgadd_column_name()bsd_chksum_file()bsd_chksum_str()check_abs_path()check_analysis_type()check_effect_type()check_parent()compress_file()convert()count_lines()create_chrpos_spec_str()create_uni_id()error_on_empty()expand_relative_path()get_column_name()get_file_name()get_old_analysis_id()get_open_method()get_tmp_file()md5_file()norm_name()parse_bool()parse_chrpos_spec_str()passthrough()safe_move()stdopen()
gwas_norm.constantsChrPosSpecMsgadd_column_name()bsd_chksum_file()bsd_chksum_str()check_abs_path()check_analysis_type()check_effect_type()check_parent()compress_file()convert()count_lines()create_chrpos_spec_str()create_uni_id()error_on_empty()expand_relative_path()get_column_name()get_file_name()get_old_analysis_id()get_open_method()get_tmp_file()md5_file()norm_name()parse_bool()parse_chrpos_spec_str()passthrough()safe_move()stdopen()
gwas_norm.metadatasub-packagegwas_norm.utilssub-packagegwas_norm.variantssub-packagegwas_norm.variants.vcf_infoALLELECADD_INFO_FIELDCADD_KEY_LOOKUPCADD_MAIN_DELIMITERCADD_PHREDCADD_RAWCLINORIGIN_BICLINORIGIN_DENCLINORIGIN_GERMCLINORIGIN_INCCLINORIGIN_INHCLINORIGIN_LOOKUPCLINORIGIN_MATCLINORIGIN_NTCLINORIGIN_OTHCLINORIGIN_PATCLINORIGIN_SOMCLINORIGIN_UNICLINORIGIN_UNKNCLINSIG_AFFCLINSIG_ASSCCLINSIG_BENCLINSIG_CONFCLINSIG_DRUGCLINSIG_ERRCLINSIG_LBENCLINSIG_LOOKUPCLINSIG_LPATHCLINSIG_NPCLINSIG_OTHCLINSIG_PATHCLINSIG_PROTCLINSIG_RISKCLINSIG_SENCLINSIG_USIGCLINVAR_ID_DELIMITERCLINVAR_MAIN_DELIMITERCLINVAR_MAPPED_FIELDSCLINVAR_ORIGINCLINVAR_SIGNIFCLINVAR_SUB_DELIMITERCLNACCCLNDISDBCLNDNCLNHGVSCLNORIGINCLNREVSTATCLNSIGCLNVICODING_SEQUENCECONSEQUENCECONSEQUENCESCONSEQUENCE_LOOKUPCaddKeysClinVarKeysClinVarOriClinVarSigDOWNSTREAMFEATUREFEATURE_ELONGATIONFEATURE_TYPEFEAT_TRUNCATIONFIVE_PRIME_UTRFRAMESHIFTGENEINCOMPLETE_TERMINAL_CODONINFRAME_DELINFRAME_INSINTERGENICINTRONMATURE_MIRNAMISSENSENC_TRANSNC_TRANS_EXONNMD_TRANSPOLYPHENPROT_ALTERINGREG_REGIONREG_REGION_ABLATIONREG_REGION_AMPSIFTSPLICE_ACCEPTORSPLICE_DONORSPLICE_DONOR_FIFTH_BASESPLICE_DONOR_REGIONSPLICE_POLYPRIM_TRACTSPLICE_REGIONSTART_LOSTSTART_RETAINEDSTOP_GAINEDSTOP_LOSTSTOP_RETAINEDSYNONYMOUSSoTFBS_ABLATIONTFBS_AMPTF_BINDING_SITETHREE_PRIME_UTRTRANS_ABLATIONTRANS_AMPUPSTREAMVCF_MISSINGVEP_INFO_FIELDVEP_KEY_LOOKUPVEP_MAIN_DELIMITERVepKeyscadd_info_parser()clinvar_most_significant()parse_clinvar()parse_clinvar_dbs()parse_clinvar_delim()parse_clinvar_disease_db()parse_clinvar_origin()parse_clinvar_significance()parse_clinvar_var_id()parse_float()parse_none()parse_return()parse_vep_consequence()validate_header_metadata()vep_info_parser()vep_worst_consequence()
gwas_norm.variants.normgwas_norm.variants.mappergwas_norm.variants.resolversgwas_norm.variants.constantsALLELE_DELIMITERDNAALLOWED_TYPESBLANK_ALLELEIDCHRSTARTENDSTRANDREFALTSTRAND_FLIPREF_FLIPPARTIAL_ALLELE_MATCHCOORD_OFFSETMAPPING_DECODEMAPPING_DECODE_STRALTALT_ALLELE_INFERREDBALANCEDCHRCOMP_TRANSLATEColumnDELETIONDNA_DEL_REGEXDNA_DEL_STRDNA_REGEXDNA_STRDataSetENDENSEMBL_DELETIONENS_ID_REGEXENS_ID_STRERRORFLAGSIDINSERTIONIS_PALINDROMICMapCoordMappingFlagMappingResultNORMALISEDNO_DATAPARTIAL_ALLELE_MATCHPOP_START_IDXREFREF_FLIPRS_REGEXRS_STRSTARTSTRANDSTRAND_FLIPUNKNOWN_INDELVCF_FORMAT_IDXVCF_ID_IDXVCF_INFO_IDXdataset_bits()decode_mapping_flags()main()
gwas_norm.variants.common
gwas_norm.variants.downloadssub-packagegwas_norm.hpcsub-package
- Testing in GWAS norm
Project admin