Hi,

I'm working on targeted resequencing data and I'm doing a multi-sample variant calling with the HaplotypeCaller. First, I tried to call the variants in all the targeted regions by doing the calling at one time on a cluster. I thus specified all the targeted regions with the -L option.

Then, as it was taking too long, I decided to cut my interval list, chromosome by chromosome and to do the calling on each chromosome. At the end, I merged the VCFs files that I had obtained for the callings on the different chromosomes.

Then, I compared this merged VCF file with the vcf file that I obtained by doing the calling on all the targeted regions at one time. I noticed 1% of variation between the two variants lists. And I can't explain this stochasticity. Any suggestion?

Thanks!

Maguelonne

I am using Genotype Given Allele with Haplotype Caller
I am trying to explicitely request all annotations that the documentation says are compatible with the Haplotype caller (and that make sense for a single sample .. e.g. no hardy weinberg ..)

the following annotations all have "NA"
GCContent(GC) HomopolymerRun(Hrun) TandemRepeatAnnotator (STR RU RPA)
.. but are valid requests because I get no errors from GATK.

This is the command I ran (all on one line)

java -Xmx40g -jar /data5/bsi/bictools/alignment/gatk/3.4-46/GenomeAnalysisTK.jar -T HaplotypeCaller --input_file /data2/external_data/Weinshilboum_Richard_weinsh/s115343.beauty/Paired_analysis/secondary/Paired_10192014/IGV_BAM/pair_EX167687/s_EX167687_DNA_Blood.igv-sorted.bam --alleles:vcf /data2/external_data/Kocher_Jean-Pierre_m026645/s109575.ez/Sequencing_2016/OMNI.vcf --phone_home NO_ET --gatk_key /projects/bsi/bictools/apps/alignment/GenomeAnalysisTK/3.1-1/Hossain.Asif_mayo.edu.key --reference_sequence /data2/bsi/reference/sequence/human/ncbi/hg19/allchr.fa --minReadsPerAlignmentStart 1 --disableOptimizations --dontTrimActiveRegions --forceActive --out /data2/external_data/Kocher_Jean-Pierre_m026645/s109575.ez/Sequencing_2016/EX167687.0.0375.chr22.vcf --logging_level INFO -L chr22 --downsample_to_fraction 0.0375 --downsampling_type BY_SAMPLE --genotyping_mode GENOTYPE_GIVEN_ALLELES --standard_min_confidence_threshold_for_calling 20.0 --standard_min_confidence_threshold_for_emitting 0.0 --annotateNDA --annotation BaseQualityRankSumTest --annotation ClippingRankSumTest --annotation Coverage --annotation FisherStrand --annotation GCContent --annotation HomopolymerRun --annotation LikelihoodRankSumTest --annotation MappingQualityRankSumTest --annotation NBaseCount --annotation QualByDepth --annotation RMSMappingQuality --annotation ReadPosRankSumTest --annotation StrandOddsRatio --annotation TandemRepeatAnnotator --annotation DepthPerAlleleBySample --annotation DepthPerSampleHC --annotation StrandAlleleCountsBySample --annotation StrandBiasBySample --excludeAnnotation HaplotypeScore --excludeAnnotation InbreedingCoeff

Log file is below( Notice "weird" WARNings about) "StrandBiasBySample annotation exists in input VCF header"..
which make no sense because the header is empty other than the barebone fields.

This is the barebone VCF
head /data2/external_data/Kocher_Jean-Pierre_m026645/s109575.ez/Sequencing_2016/OMNI.vcf

fileformat=VCFv4.2

CHROM POS ID REF ALT QUAL FILTER INFO

chr1 723918 rs144434834 G A 30 PASS .
chr1 729632 rs116720794 C T 30 PASS .
chr1 752566 rs3094315 G A 30 PASS .
chr1 752721 rs3131972 A G 30 PASS .
chr1 754063 rs12184312 G T 30 PASS .
chr1 757691 rs74045212 T C 30 PASS .
chr1 759036 rs114525117 G A 30 PASS .
chr1 761764 rs144708130 G A 30 PASS .

This is the output

INFO 10:03:06,080 HelpFormatter - ---------------------------------------------------------------------------------
INFO 10:03:06,082 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.4-46-gbc02625, Compiled 2015/07/09 17:38:12
INFO 10:03:06,083 HelpFormatter - Copyright (c) 2010 The Broad Institute
INFO 10:03:06,083 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk
INFO 10:03:06,086 HelpFormatter - Program Args: -T HaplotypeCaller --input_file /data2/external_data/Weinshilboum_Richard_weinsh/s115343.beauty/Paired_analysis/secondary/Paired_10192014/IGV_BAM/pair_EX167687/s_EX167687_DNA_Blood.igv-sorted.bam --alleles:vcf /data2/external_data/Kocher_Jean-Pierre_m026645/s109575.ez/Sequencing_2016/OMNI.vcf --phone_home NO_ET --gatk_key /projects/bsi/bictools/apps/alignment/GenomeAnalysisTK/3.1-1/Hossain.Asif_mayo.edu.key --reference_sequence /data2/bsi/reference/sequence/human/ncbi/hg19/allchr.fa --minReadsPerAlignmentStart 1 --disableOptimizations --dontTrimActiveRegions --forceActive --out /data2/external_data/Kocher_Jean-Pierre_m026645/s109575.ez/Sequencing_2016/EX167687.0.0375.chr22.vcf --logging_level INFO -L chr22 --downsample_to_fraction 0.0375 --downsampling_type BY_SAMPLE --genotyping_mode GENOTYPE_GIVEN_ALLELES --standard_min_confidence_threshold_for_calling 20.0 --standard_min_confidence_threshold_for_emitting 0.0 --annotateNDA --annotation BaseQualityRankSumTest --annotation ClippingRankSumTest --annotation Coverage --annotation FisherStrand --annotation GCContent --annotation HomopolymerRun --annotation LikelihoodRankSumTest --annotation MappingQualityRankSumTest --annotation NBaseCount --annotation QualByDepth --annotation RMSMappingQuality --annotation ReadPosRankSumTest --annotation StrandOddsRatio --annotation TandemRepeatAnnotator --annotation DepthPerAlleleBySample --annotation DepthPerSampleHC --annotation StrandAlleleCountsBySample --annotation StrandBiasBySample --excludeAnnotation HaplotypeScore --excludeAnnotation InbreedingCoeff
INFO 10:03:06,093 HelpFormatter - Executing as m037385@franklin04-213 on Linux 2.6.32-573.8.1.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_20-b26.
INFO 10:03:06,094 HelpFormatter - Date/Time: 2016/01/19 10:03:06
INFO 10:03:06,094 HelpFormatter - ---------------------------------------------------------------------------------
INFO 10:03:06,094 HelpFormatter - ---------------------------------------------------------------------------------
INFO 10:03:06,545 GenomeAnalysisEngine - Strictness is SILENT
INFO 10:03:06,657 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Fraction: 0.04
INFO 10:03:06,666 SAMDataSource$SAMReaders - Initializing SAMRecords in serial
INFO 10:03:07,012 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.35
INFO 10:03:07,031 HCMappingQualityFilter - Filtering out reads with MAPQ < 20
INFO 10:03:07,170 IntervalUtils - Processing 51304566 bp from intervals
INFO 10:03:07,256 GenomeAnalysisEngine - Preparing for traversal over 1 BAM files
INFO 10:03:07,595 GenomeAnalysisEngine - Done preparing for traversal
INFO 10:03:07,595 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING]
INFO 10:03:07,595 ProgressMeter - | processed | time | per 1M | | total | remaining
INFO 10:03:07,596 ProgressMeter - Location | active regions | elapsed | active regions | completed | runtime | runtime
INFO 10:03:07,596 HaplotypeCaller - Disabling physical phasing, which is supported only for reference-model confidence output
WARN 10:03:07,709 StrandBiasTest - StrandBiasBySample annotation exists in input VCF header. Attempting to use StrandBiasBySample values to calculate strand bias annotation values. If no sample has the SB genotype annotation, annotation may still fail.
WARN 10:03:07,709 StrandBiasTest - StrandBiasBySample annotation exists in input VCF header. Attempting to use StrandBiasBySample values to calculate strand bias annotation values. If no sample has the SB genotype annotation, annotation may still fail.
INFO 10:03:07,719 HaplotypeCaller - Using global mismapping rate of 45 => -4.5 in log10 likelihood units
INFO 10:03:37,599 ProgressMeter - chr22:5344011 0.0 30.0 s 49.6 w 10.4% 4.8 m 4.3 m
INFO 10:04:07,600 ProgressMeter - chr22:11875880 0.0 60.0 s 99.2 w 23.1% 4.3 m 3.3 m
Using AVX accelerated implementation of PairHMM
INFO 10:04:29,924 VectorLoglessPairHMM - libVectorLoglessPairHMM unpacked successfully from GATK jar file
INFO 10:04:29,925 VectorLoglessPairHMM - Using vectorized implementation of PairHMM
WARN 10:04:29,938 AnnotationUtils - Annotation will not be calculated, genotype is not called
WARN 10:04:29,938 AnnotationUtils - Annotation will not be calculated, genotype is not called
WARN 10:04:29,939 AnnotationUtils - Annotation will not be calculated, genotype is not called
INFO 10:04:37,601 ProgressMeter - chr22:17412465 0.0 90.0 s 148.8 w 33.9% 4.4 m 2.9 m
INFO 10:05:07,602 ProgressMeter - chr22:18643131 0.0 120.0 s 198.4 w 36.3% 5.5 m 3.5 m
INFO 10:05:37,603 ProgressMeter - chr22:20133744 0.0 2.5 m 248.0 w 39.2% 6.4 m 3.9 m
INFO 10:06:07,604 ProgressMeter - chr22:22062452 0.0 3.0 m 297.6 w 43.0% 7.0 m 4.0 m
INFO 10:06:37,605 ProgressMeter - chr22:23818297 0.0 3.5 m 347.2 w 46.4% 7.5 m 4.0 m
INFO 10:07:07,606 ProgressMeter - chr22:25491290 0.0 4.0 m 396.8 w 49.7% 8.1 m 4.1 m
INFO 10:07:37,607 ProgressMeter - chr22:27044271 0.0 4.5 m 446.4 w 52.7% 8.5 m 4.0 m
INFO 10:08:07,608 ProgressMeter - chr22:28494980 0.0 5.0 m 496.1 w 55.5% 9.0 m 4.0 m
INFO 10:08:47,609 ProgressMeter - chr22:30866786 0.0 5.7 m 562.2 w 60.2% 9.4 m 3.8 m
INFO 10:09:27,610 ProgressMeter - chr22:32908950 0.0 6.3 m 628.3 w 64.1% 9.9 m 3.5 m
INFO 10:09:57,610 ProgressMeter - chr22:34451306 0.0 6.8 m 677.9 w 67.2% 10.2 m 3.3 m
INFO 10:10:27,611 ProgressMeter - chr22:36013343 0.0 7.3 m 727.5 w 70.2% 10.4 m 3.1 m
INFO 10:10:57,613 ProgressMeter - chr22:37387478 0.0 7.8 m 777.1 w 72.9% 10.7 m 2.9 m
INFO 10:11:27,614 ProgressMeter - chr22:38534891 0.0 8.3 m 826.8 w 75.1% 11.1 m 2.8 m
INFO 10:11:57,615 ProgressMeter - chr22:39910054 0.0 8.8 m 876.4 w 77.8% 11.4 m 2.5 m
INFO 10:12:27,616 ProgressMeter - chr22:41738463 0.0 9.3 m 926.0 w 81.4% 11.5 m 2.1 m
INFO 10:12:57,617 ProgressMeter - chr22:43113306 0.0 9.8 m 975.6 w 84.0% 11.7 m 112.0 s
INFO 10:13:27,618 ProgressMeter - chr22:44456937 0.0 10.3 m 1025.2 w 86.7% 11.9 m 95.0 s
INFO 10:13:57,619 ProgressMeter - chr22:45448656 0.0 10.8 m 1074.8 w 88.6% 12.2 m 83.0 s
INFO 10:14:27,620 ProgressMeter - chr22:46689073 0.0 11.3 m 1124.4 w 91.0% 12.5 m 67.0 s
INFO 10:14:57,621 ProgressMeter - chr22:48062438 0.0 11.8 m 1174.0 w 93.7% 12.6 m 47.0 s
INFO 10:15:27,622 ProgressMeter - chr22:49363910 0.0 12.3 m 1223.6 w 96.2% 12.8 m 29.0 s
INFO 10:15:57,623 ProgressMeter - chr22:50688233 0.0 12.8 m 1273.2 w 98.8% 13.0 m 9.0 s
INFO 10:16:12,379 VectorLoglessPairHMM - Time spent in setup for JNI call : 0.061128124000000006
INFO 10:16:12,379 PairHMM - Total compute time in PairHMM computeLikelihoods() : 22.846350295
INFO 10:16:12,380 HaplotypeCaller - Ran local assembly on 25679 active regions
INFO 10:16:12,434 ProgressMeter - done 5.1304566E7 13.1 m 15.0 s 100.0% 13.1 m 0.0 s
INFO 10:16:12,435 ProgressMeter - Total runtime 784.84 secs, 13.08 min, 0.22 hours
INFO 10:16:12,435 MicroScheduler - 727347 reads were filtered out during the traversal out of approximately 4410423 total reads (16.49%)
INFO 10:16:12,435 MicroScheduler - -> 2 reads (0.00% of total) failing BadCigarFilter
INFO 10:16:12,436 MicroScheduler - -> 669763 reads (15.19% of total) failing DuplicateReadFilter
INFO 10:16:12,436 MicroScheduler - -> 0 reads (0.00% of total) failing FailsVendorQualityCheckFilter
INFO 10:16:12,436 MicroScheduler - -> 57582 reads (1.31% of total) failing HCMappingQualityFilter
INFO 10:16:12,437 MicroScheduler - -> 0 reads (0.00% of total) failing MalformedReadFilter
INFO 10:16:12,437 MicroScheduler - -> 0 reads (0.00% of total) failing MappingQualityUnavailableFilter
INFO 10:16:12,437 MicroScheduler - -> 0 reads (0.00% of total) failing NotPrimaryAlignmentFilter
INFO 10:16:12,438 MicroScheduler - -> 0 reads (0.00% of total) failing UnmappedReadFilter

Hello there,

I am calling variants on 800 exome samples using Haplotypercaller for some reasons the caller stopped the analysis on certain location on chromosome 5 (after 5 weeks and i have 10 weeks to go). The error is below (one of the samples was malformed)
_INFO 02:04:51,540 ProgressMeter - chr5:180670788 1.05846818873E11 4.5 w 25.0 s 31.7% 14.1 w 9.6 w
INFO 02:06:01,736 ProgressMeter - chr5:180687847 1.05853600709E11 4.5 w 25.0 s 31.7% 14.1 w 9.6 w
INFO 02:07:01,737 ProgressMeter - chr5:180687847 1.05853600709E11 4.5 w 25.0 s 31.7% 14.1 w 9.6 w
INFO 02:08:11,738 ProgressMeter - chr5:180687847 1.05853600709E11 4.5 w 25.0 s 31.7% 14.1 w 9.6 w
INFO 02:09:11,739 ProgressMeter - chr5:180687847 1.05853600709E11 4.5 w 25.0 s 31.7% 14.1 w 9.6 w
INFO 02:10:11,740 ProgressMeter - chr5:180687847 1.05853600709E11 4.5 w 25.0 s 31.7% 14.1 w 9.6 w
INFO 02:11:11,741 ProgressMeter - chr5:180687847 1.05853600709E11 4.5 w 25.0 s 31.7% 14.1 w 9.6 w

ERROR ------------------------------------------------------------------------------------------

ERROR A USER ERROR has occurred (version 3.8-1-0-gf15c1c3ef):

ERROR

ERROR This means that one or more arguments or inputs in your command are incorrect.

ERROR The error message below tells you what is the problem.

ERROR

ERROR If the problem is an invalid argument, please check the online documentation guide

ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.

ERROR

ERROR Visit our website and forum for extensive documentation and answers to

ERROR commonly asked questions https://software.broadinstitute.org/gatk

ERROR

ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.

ERROR

ERROR MESSAGE: File /media/daruma/sea/sample483.fastq.gz.mdup.realigned.fixed.recal.bai is malformed: Premature end-of-file while reading BAM index file sample483.fastq.gz.mdup.realigned.fixed.recal.bai. It's likely that this file is truncated or corrupt -- Please try re-indexing the corresponding BAM file.

_
I will re analysis the sample and want to resume the analysis to speed up the process i was wondering if Haplotypecaller has resume function. If it doesnt what is the best way to work around the issue?
- shall i modify the exome target file ? if yes, shall i start from the begining of chroomsome 5 or find the closer position to where the analysis stopped ?

• why not including resume feature by feeding the analysis VCF file where it was stopped?
  would like to hear other suggestions that was not mentioned

Thanks in advance

Hello, maybe I'm asking a naive question and maybe it has been answered somewhere else, but as the title states are there differences in the algorithm of the HaplotypeCaller between GATK3.8 release and GATK4?
I'm calling variants in two exomes, using the same bam files as input of the variant caller, and I tried to compare variants at the end of the pipeline. I found that GATK4 calls a slightly higher number of variants than GATK3.8 and that this one in the final step step (after filtering and selecting variants) retains variants filtered away from the GATK4 HC, is this expected or do I have to go trough my pipeline and check for errors?
Thank you

Hi,
I began a run using the --emitRefConfidence GVCF command asking for a .g.vcf file to be output for a single sample. Which is running (yay)!
So far, it is still running and has thrown up several warning messages. Two of them I understand, but these two I am not sure about. Can anyone throw some light on these for me please?

WARN 18:42:38,322 AnnotationUtils - Annotation will not be calculated, genotype is not called
WARN 18:42:38,342 AnnotationUtils - Annotation will not be calculated, genotype is not called
WARN 18:42:38,358 HaplotypeScore - Annotation will not be calculated, must be called from UnifiedGenotyper, not HaplotypeCaller

Many thanks,
Jenni

Dear GATK team,
I have prepared 100 WES processed BAMs and try to call variants and output them in a single VCF. I used scatter-gather WDL scripts following https://software.broadinstitute.org/wdl/documentation/article?id=7614 .
I submit the job to one node which has 40 cores. I tried more cores (from 2 or more nodes), but it seems haplotypecaller only runs on one node. One node should have about 126G memory.
The problem is the job seems need too much memory and the node is unavailable to access.
Does the scatter-gather needs a lot memory ?

Part of my log file:
2018-03-28 21:20:13,921 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:11:1]: Status change from - to WaitingForReturnCodeFile
2018-03-28 21:20:13,921 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:53:1]: job id: 4800
2018-03-28 21:20:13,922 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:63:1]: Status change from - to WaitingForReturnCodeFile
2018-03-28 21:20:13,923 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:25:1]: Status change from - to WaitingForReturnCodeFile
2018-03-28 21:20:13,924 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:53:1]: Status change from - to WaitingForReturnCodeFile
2018-03-28 21:20:13,945 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:84:1]: job id: 4937
2018-03-28 21:20:13,955 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:84:1]: Status change from - to WaitingForReturnCodeFile

stderr of one shard-0/execution:
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/work/home/jiahuan/Heart_WES/HTX/cromwell-executions/jointCallingGenotypes/d71d6520-d369-43c8-bfce-f0521358f8e9/call-HaplotypeCallerERC/shard-0/execution/tmp.2jWDLA

note: GATK3.7

Thanks!

Hello,
I'm looking at the HC tag that comes out of the --bamOutput option in HaplotypeCaller in the corresponding bam file that is produced. I've noticed that some real reads (i.e. ones not labeled as "ArtificialHaplotype") do not have a HC tag and are NA. What does this mean? That the read was not used and discarded or that no local re-assembly was performed? Many thanks.

I am using a combination of HaplotypeCaller local (non-spark), in GVCF mode, followed by GatherVcfs to merge them, and I get very different call results across runs. I would expect the probabilities/confidence values to change slightly, but not so much the number of calls. Is this normal?

I'm using the gatk from docker://broadinstitute/gatk:4.beta.6 . My BAM/BAI files pass validation.

I see other posts about results being non-deterministic. But I'm not passing any of the -nt or -nct flags in this case.

I'm splitting all my contigs (bed file) in roughly equal-sized chunks, and calling HaplotypeCaller, like so. The VCF file produced changes a lot if I do 8 chunks, vs 128. I'm not sure whether that makes things worse.

# chunk 000
java -jar /gatk/gatk.jar HaplotypeCaller -R ANN0859.bam --emitRefConfidence GVCF -L bed_chunk_000.bed -O ANN0859.bam_000.g.vcf -hets 0.010000
# chunk 001
java -jar /gatk/gatk.jar HaplotypeCaller -R ANN0859.bam --emitRefConfidence GVCF -L bed_chunk_001.bed -O ANN0859.bam_001.g.vcf -hets 0.010000
...

I merge them like so (passing all the chunks in order):

java -jar /gatk/gatk.jar GatherVcfs -I ANN0859.bam_000.g.vcf -I ANN0859.bam_001.g.vcf ...

The entire bed is sorted, and the chunks are not overlapping. I've made sure that I'm not losing any contigs when I split my bed file.

To provide an example difference for one of the chromosomes, I get the following calls (for 128 chunks) in the final output gVCF:

HanXRQChr00c0117        2497    .       G       <NON_REF>       .       .       END=2580        GT:DP:GQ:MIN_DP:PL      0/0:0:0:0:0,0,0
HanXRQChr00c0117        10708   .       G       <NON_REF>       .       .       END=25539       GT:DP:GQ:MIN_DP:PL      0/0:0:0:0:0,0,0
(EOF)

And if I divide the work in 8 (longer) chunks, that last section just explodes into 1960 different calls:

HanXRQChr00c0117        10708   .       G       <NON_REF>       .       .       END=14265       GT:DP:GQ:MIN_DP:PL      0/0:0:0:0:0,0,0
HanXRQChr00c0117        14266   .       C       <NON_REF>       .       .       END=14267       GT:DP:GQ:MIN_DP:PL      0/0:1:3:1:0,3,42
...
HanXRQChr00c0117        14309   .       T       C,<NON_REF>     0.13    .       DP=2;MLEAC=0,0;MLEAF=nan,nan;RAW_MQ=7200        GT:PGT:PID      ./.:0|1:14309_T_C
HanXRQChr00c0117        14310   .       T       <NON_REF>       .       .       END=14315       GT:DP:GQ:MIN_DP:PL      0/0:1:3:1:0,3,45
HanXRQChr00c0117        14316   .       T       C,<NON_REF>     0.13    .       DP=2;MLEAC=0,0;MLEAF=nan,nan;RAW_MQ=7200        GT:PGT:PID      ./.:0|1:14309_T_C
HanXRQChr00c0117        14317   .       T       <NON_REF>       .       .       END=14321       GT:DP:GQ:MIN_DP:PL      0/0:1:3:1:0,3,45
...
HanXRQChr00c0117        14358   .       T       <NON_REF>       .       .       END=14359       GT:DP:GQ:MIN_DP:PL      0/0:4:12:4:0,12,180
HanXRQChr00c0117        14360   .       A       G,<NON_REF>     30.02   .       DP=4;ExcessHet=3.0103;MLEAC=1,0;MLEAF=0.5,0;RAW_MQ=14400        GT:AD:DP:GQ:PL:SB       1/1:0,1,0:1:3:45,3,0,4
5,3,45:0,0,1,0
...
HanXRQChr00c0117        25479   .       T       <NON_REF>       .       .       END=25484       GT:DP:GQ:MIN_DP:PL      0/0:8:24:8:0,24,296
HanXRQChr00c0117        25485   .       T       <NON_REF>       .       .       END=25485       GT:DP:GQ:MIN_DP:PL      0/0:8:21:8:0,21,315
HanXRQChr00c0117        25486   .       T       <NON_REF>       .       .       END=25521       GT:DP:GQ:MIN_DP:PL      0/0:6:18:6:0,18,217
HanXRQChr00c0117        25522   .       A       <NON_REF>       .       .       END=25524       GT:DP:GQ:MIN_DP:PL      0/0:7:15:7:0,15,225
HanXRQChr00c0117        25525   .       T       <NON_REF>       .       .       END=25539       GT:DP:GQ:MIN_DP:PL      0/0:5:9:3:0,9,133
(EOF)

I thought at first that maybe the chunk boundaries were at play, but those contigs are in the middle of a chunk file.

I was thinking of how allele depth / frequency affect variant calling. I know that the HC is using Bayes to call variant. The question is how is allele depth / frequency information used during the variant calling?

Based on Bayes: The most possible variant allele is: P(G|D) = arg max P(G) * P(D|G). I initial thought that P(D|G) is the allele frequency of data. But feel something is not right.

Could you please share some comment? Thanks

Hi GATK Team! I was wondering if there is a way to get Haplotype Caller in ERC BP Resolution mode to emit details for every base for every position? That is, for each position rather than just emitting in the ALT column, or in some cases an alternate allele or multiple alternate alleles and , is it possible to get it to emit every single base that had support in any of the reads - that is, samtoolsmpileup-like information where for every position there is information on the number of reads for A/C/T/G?

In particular, I am hoping to do this for a given set of positions I am interested in - would be great if there is an option for this as well.

I was hoping to potentially use this rather than an mpileup since local reassembly is performed on these sites.

Thanks a lot!

Alon

Apologies if this was addressed elsewhere, but I have looked carefully through the documentation. Simply put: using the joint analysis pipeline of GATK's HaplotypeCaller (-ERC GVCF) -> CombineGVCFs-> GenotypeGVCFs, are SNPs called when they are fixed in only 1 sample? For example, imagine 3 samples total, where the correct genotypes should be: A/A, A/A, T/T. Will this variant be called correctly, given sufficient coverage/quality? This is very important to what I am trying to accomplish, as I only have 3 samples, and would like to analyze SNPs that are fixed between samples. It's unclear to me that this pipeline will work, since HaplotypeCaller will not see any find these variants within samples individually. But maybe GenotypeGVCFs finds them? This is not clearly documented.

Hi all,

I prepared a clean Bam file following GATK Best Practice and used GATK4 HaplotypeCaller to create a gvcf with ploidy1 option:

gatk-4.0.2.1/gatk HaplotypeCaller --native-pair-hmm-threads 24 -I KU_filtered_sorted_mdup.bam -O HC.KU.raw.snps.indels.g.vcf -R ref.fasta -ploidy 1 --emit-ref-confidence GVCF

When I validated the gvcf, ValidateVariants threw errors at the end:

<br />11:27:55.681 INFO  ProgressMeter - Traversal complete. Processed 124689522 total variants in 3.8 minutes.
11:27:55.681 INFO  ValidateVariants - Shutting down engine
[April 10, 2018 11:27:55 AM JST] org.broadinstitute.hellbender.tools.walkers.variantutils.ValidateVariants done. Elapsed time: 3.82 minutes.
Runtime.totalMemory()=4682940416
java.lang.IllegalArgumentException: Illegal character in path at index 15:HC.KU.raw.snps.indels.g.vcf
    at java.net.URI.create(URI.java:852)
    at org.broadinstitute.hellbender.engine.FeatureInput.makeIntoAbsolutePath(FeatureInput.java:242)
    at org.broadinstitute.hellbender.engine.FeatureInput.toString(FeatureInput.java:314)
    at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2886)
    at java.util.Formatter$FormatSpecifier.print(Formatter.java:2763)
    at java.util.Formatter.format(Formatter.java:2520)
    at java.util.Formatter.format(Formatter.java:2455)
    at java.lang.String.format(String.java:2940)
    at org.broadinstitute.hellbender.engine.FeatureDataSource.close(FeatureDataSource.java:589)
    at org.broadinstitute.hellbender.engine.FeatureManager.lambda$close$9(FeatureManager.java:505)
    at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
    at org.broadinstitute.hellbender.engine.FeatureManager.close(FeatureManager.java:505)
    at org.broadinstitute.hellbender.engine.GATKTool.onShutdown(GATKTool.java:857)
    at org.broadinstitute.hellbender.engine.VariantWalker.onShutdown(VariantWalker.java:95)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:180)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:199)
    at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:159)
    at org.broadinstitute.hellbender.Main.mainEntry(Main.java:202)
    at org.broadinstitute.hellbender.Main.main(Main.java:288)
Caused by: java.net.URISyntaxException: Illegal character in path at index 15: /media/yoshi/My Book/Aet_v4.0_ChrSeqSplit/HC.KU-2103.raw.snps.indels.g.vcf
    at java.net.URI$Parser.fail(URI.java:2848)
    at java.net.URI$Parser.checkChars(URI.java:3021)
    at java.net.URI$Parser.parseHierarchical(URI.java:3105)
    at java.net.URI$Parser.parse(URI.java:3063)
    at java.net.URI.<init>(URI.java:588)
    at java.net.URI.create(URI.java:850)
    ... 19 more

When I ran GenotypeGVCFs using the gvcf, it ran to completion, but threw the same "java.lang.IllegalArgumentException" errors at the end.

My java is Java runtime: OpenJDK 64-Bit Server VM v1.8.0_162-8u162-b12-0ubuntu0.16.04.2-b12.

Could you please help me with this problem?
Many thanks.

I am running HaplotypeCaller with the --alleles option and it is crashing. I've traced it to a very long line in the --alleles VCF file, with a very long allele. The line is:

chr4    1394536 .       AATGTGGAGTGCCCGCCTGCTCACACGTGCCCATGTGGAGTGCCCGCCTGCTCATGTGCCCATGTGGAGTGCCCGCCTGCTCACACATGTCGATGCGGAGTGCCCGCCTGCTCACACATGCCC     A,AATGTGGAGTGCCCGCCTGCTCATGTGCCCATGTGGAGTGCCCGCCTGCTCACACATGTCGATGCGGAGTGCCCGCCTGCTCACACATGCCC,CATGTGGAGTGCCCGCCTGCTCACACGTGCCCATGTGGAGTGCCCGCCTGCTCATGTGCCCATGTGGAGTGCCCGCCTGCTCACACATGTCGATGCGGAGTGCCCGCCTGCTCACACATGCCC .
    .       .

This VCF was generated by Mutect2 for creation of a panel-of-normals.

The command and output were:

twtoal@carcinos>          env -i LD_LIBRARY_PATH=/share/carvajal-archive/PACKAGES/local/miniconda/miniconda3_perl/envs/R_pipeline/lib/R/library/rJava/libs:/share/carvajal-archive/PACKAGES/local/miniconda/miniconda3_perl/envs/R_pipeline/jre/lib/amd64/server PATH=/bin:/share/carvajal-archive/PACKAGES/local/miniconda/miniconda3_perl/envs/R_pipeline/bin /share/carvajal-archive/PACKAGES/src/java/jdk8_latest/bin/java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=1 -Djava.io.tmpdir=/share/carvajal-archive/tmp -jar /share/carvajal-archive/PACKAGES/src/GATK/gatk-4.0.3.0/gatk-package-4.0.3.0-local.jar HaplotypeCaller --QUIET             -R /share/carvajal-archive/REFERENCE_DATA/genomes/GRCh38_decoy_LCCpanel/Homo_sapiens_assembly38_LCCpanel.fasta             -I DATA/VO-56/N2/VO-56N2.recal.bam             -O DATA/VO-56/N2/VO-56N2.PONdepths.vcf.gz             -L GLOBAL/PON4.bed             --alleles GLOBAL/PON6.vcf.gz             --genotyping-mode GENOTYPE_GIVEN_ALLELES             --annotation-group StandardAnnotation             --seconds-between-progress-updates 1             --verbosity INFO
14:23:07.181 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/share/carvajal-archive/PACKAGES/src/GATK/gatk-4.0.3.0/gatk-package-4.0.3.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
14:23:07.438 INFO  HaplotypeCaller - Initializing engine
14:23:08.919 INFO  FeatureManager - Using codec VCFCodec to read file file:///share/carvajal-archive/SEQ_DATA/PANELS/MSEQ_GC_Panel_03_28_2017/GLOBAL/PON6.vcf.gz
14:23:09.003 INFO  FeatureManager - Using codec BEDCodec to read file file:///share/carvajal-archive/SEQ_DATA/PANELS/MSEQ_GC_Panel_03_28_2017/GLOBAL/PON4.bed
14:23:09.014 INFO  IntervalArgumentCollection - Processing 10 bp from intervals
14:23:09.046 INFO  HaplotypeCaller - Done initializing engine
14:23:09.163 INFO  HaplotypeCallerEngine - Disabling physical phasing, which is supported only for reference-model confidence output
14:23:11.216 INFO  NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/share/carvajal-archive/PACKAGES/src/GATK/gatk-4.0.3.0/gatk-package-4.0.3.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
14:23:11.230 INFO  NativeLibraryLoader - Loading libgkl_pairhmm_omp.so from jar:file:/share/carvajal-archive/PACKAGES/src/GATK/gatk-4.0.3.0/gatk-package-4.0.3.0-local.jar!/com/intel/gkl/native/libgkl_pairhmm_omp.so
14:23:11.547 WARN  NativeLibraryLoader - Unable to load libgkl_pairhmm_omp.so from native/libgkl_pairhmm_omp.so (/share/carvajal-archive/tmp/twtoal/libgkl_pairhmm_omp3184435069396731524.so: /usr/lib/x86_64-linux-gnu/libgomp.so.1: version `GOMP_4.0' not found (required by /share/carvajal-archive/tmp/twtoal/libgkl_pairhmm_omp3184435069396731524.so))
14:23:11.547 INFO  PairHMM - OpenMP multi-threaded AVX-accelerated native PairHMM implementation is not supported
14:23:11.547 INFO  NativeLibraryLoader - Loading libgkl_pairhmm.so from jar:file:/share/carvajal-archive/PACKAGES/src/GATK/gatk-4.0.3.0/gatk-package-4.0.3.0-local.jar!/com/intel/gkl/native/libgkl_pairhmm.so
14:23:11.786 WARN  IntelPairHmm - Flush-to-zero (FTZ) is enabled when running PairHMM
14:23:11.786 WARN  IntelPairHmm - Ignoring request for 4 threads; not using OpenMP implementation
14:23:11.787 INFO  PairHMM - Using the AVX-accelerated native PairHMM implementation
14:23:11.941 INFO  ProgressMeter - Starting traversal
14:23:11.941 INFO  ProgressMeter -        Current Locus  Elapsed Minutes     Regions Processed   Regions/Minute
14:23:13.127 INFO  VectorLoglessPairHMM - Time spent in setup for JNI call : 9.267520000000001E-4
14:23:13.127 INFO  PairHMM - Total compute time in PairHMM computeLogLikelihoods() : 0.008127312000000001
14:23:13.128 INFO  SmithWatermanAligner - Total compute time in java Smith-Waterman : 0.10 sec
14:23:13.128 INFO  HaplotypeCaller - Shutting down engine
java.lang.IllegalArgumentException: expected haplotypes.size() >= eventsAtThisLoc.size() + 1
        at org.broadinstitute.hellbender.utils.Utils.validateArg(Utils.java:681)
        at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.AssemblyBasedCallerGenotypingEngine.createAlleleMapper(AssemblyBasedCallerGenotypingEngine.java:152)
        at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCallerGenotypingEngine.assignGenotypeLikelihoods(HaplotypeCallerGenotypingEngine.java:131)
        at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCallerEngine.callRegion(HaplotypeCallerEngine.java:565)
        at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller.apply(HaplotypeCaller.java:218)
        at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.processReadShard(AssemblyRegionWalker.java:295)
        at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.traverse(AssemblyRegionWalker.java:271)
        at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:893)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:134)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:179)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:198)
        at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
        at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
        at org.broadinstitute.hellbender.Main.main(Main.java:289)

Hello,

I have 3 de novo transcriptomes for which I am trying to genotype all SNPs. Originally, I asked a question about whether the joint genotyping pipeline will correctly identify SNPs fixed in one sample (e.g. A/A, A/A, T/T). That question is posted here, and though I'm still unclear about the answer, I've encountered a much bigger problem.

Using GATK 4.0.1.2, my pipeline was this one:
Pre-processing BAM file using best practices -->
HaplotypeCaller (-ERC GVCF) on each sample separately -->
CombineGVCFs -->
GenotypeGVCFs -->
VariantFiltration

However, when I got to CombineGVCFs (v4.0), the program didn't work at all. It would read the vcf files in ("Using codec VCFCodec to read file...") and then freeze forever, even with huge amounts of memory.

I considered using GenomicsDBImport instead of CombineGVCFs, but could not find precise instructions on how to separate and then concatenate by interval with -L (remember these are transcriptomes, and there are 250,000+ contigs in the reference, so processing contigs separately is not trivial). There does not seem to be an established pipeline for doing this, although several threads (e.g. this one) have mentioned CatVariants and GatherVCFs. I tested GenomicsDBImport on the first contig using -L TRINITY_DN32849_c0_g1 (name of that contig), but received an error.

Based on this post, I decided instead to skip CombineGVCFs altogether, and try GenotypeGVCFs v3.8 directly, by importing all 3 samples there (-V).
java -jar $GATK_HOME \
-T GenotypeGVCFs \
-R $fa_file \
--variant output.229.g.vcf \
--variant output.230.g.vcf \
--variant output.231.g.vcf \
--useNewAFCalculator \
-nt $threads \
-o cohort.vcf

Despite using 32 threads, the estimated runtime is 307 weeks, or 6 YEARS! Obviously this won't work.

Is there any version of GATK that is capable of genotyping my samples, in a reasonable amount of time? I'm completely stuck, and ready to give up. Any help would be very much appreciated!

Paul

I'm running HaplotypeCaller (GATK 4) with --alleles argument and it is crashing on a particular --alleles VCF file record with a stack trace with error "expected haplotypes.size() >= eventsAtThisLoc.size() + 1".

My command is:

java -jar $GATK4 HaplotypeCaller -R /share/carvajal-archive/REFERENCE_DATA/genomes/GRCh38_decoy_LCCpanel/Homo_sapiens_assembly38_LCCpanel.fasta -L chr16:58920000-58930000 -I HapCrash.bam -O HapCrashOut.vcf.gz --alleles HapCrash.vcf.gz --genotyping-mode GENOTYPE_GIVEN_ALLELES --verbosity DEBUG 2>&1 | tee HapCrashLog.txt

I'm attaching a log file of the output, as well as a copy of the HapCrash.bam file and HapCrash.vcf.gz file.

Hello , I recently saw a webinar by NCBI "Advanced Workshop on SRA and dbGaP Data Analysis" (ftp://ftp.ncbi.nlm.nih.gov/pub/education/public_webinars/2016/03Mar23_Advanced_Workshop/). They mentioned that they were able to run GATK directly on SRA files.

I downloaded GenomeAnalysisTK-3.5 jar file to my computer. I tried both these commands:

java -jar /path/GenomeAnalysisTK-3.5/GenomeAnalysisTK.jar -T HaplotypeCaller -R SRRFileName -I SRRFileName -stand_call_conf 30 -stand_emit_conf 10 -o SRRFileName.vcf

java -jar /path/GenomeAnalysisTK-3.5/GenomeAnalysisTK.jar -T SRRFileName -R SRR1718738 -I SRRFileName -stand_call_conf 30 -stand_emit_conf 10 -o SRRFileName.vcf

For both these commands, I got this error:
ERROR MESSAGE: Invalid command line: The GATK reads argument (-I, --input_file) supports only BAM/CRAM files with the .bam/.cram extension and lists of BAM/CRAM files with the .list extension, but the file SRR1718738 has neither extension. Please ensure that your BAM/CRAM file or list of BAM/CRAM files is in the correct format, update the extension, and try again.

I don't see any documentation here about this, so wanted to check with you or anyone else has had any experience with this.

Thanks
K

Hi， I find that there are multiple steps in determining active regions in haplotypecaller, so I wonder when is assembly-region-padding is applied, is it applied during steps in determining active regions or after determining active regions

Hi,

I have a number of samples that consist of multiple individuals from the same population pooled together, and have been truing to use HaplotypeCaller to call the variants. I have set the (ploidy to 2 * number of individuals) but keep getting the same or similar error message, after running for several hours or days:

ERROR ------------------------------------------------------------------------------------------

ERROR A GATK RUNTIME ERROR has occurred (version 3.3-0-g37228af):

ERROR

ERROR This might be a bug. Please check the documentation guide to see if this is a known problem.

ERROR If not, please post the error message, with stack trace, to the GATK forum.

ERROR Visit our website and forum for extensive documentation and answers to

ERROR commonly asked questions http://www.broadinstitute.org/gatk

ERROR

ERROR MESSAGE: the combination of ploidy (180) and number of alleles (9) results in a very large number of genotypes (> 2147483647). You need to limit ploidy or the number of alternative alleles to analyze this locus

ERROR ------------------------------------------------------------------------------------------

and I'm not sure what I can do to rectify it... Obviously I can't limit the ploidy, it is what it is, and I thought that HC only allows a maximum of six alleles anyway?

My code is below, and any help would be appreciated.

java -Xmx24g -jar ~/bin/GenomeAnalysisTK-3.3-0/GenomeAnalysisTK.jar -T HaplotypeCaller
-nct 6 \
-R ~/my_ref_sequence \
--intervals ~/my_intervals_file \
-ploidy 180 \
-log my_log_file \
-I ~/my_input_bam \
-o ~/my_output_vcf

Hi,
I read how haplotypecaller assign genotype in the following link, it says that for each position haplotypecaller will calculate the best genotype, so I wonder, if the positions are close, then will haplotypecaller calculate genotypes separately for them? if so, will the selected haplotypes for these two positions be different?
https://software.broadinstitute.org/gatk/documentation/article.php?id=4442

Hello,

I would like to replicate the behaviour of gakt described in Mallick et al. 2016 for the Simon's genomes data set. They explain in the supplementary information the following:

"GATK UnifiedGenotyper has a built-in prior for Bayesian SNP calling that assumes that the site is more likely to be homozygous for the reference allele than homozygous for the variant allele. For a diploid sample, the default priors for a homozygous reference, heterozygote and homozygous non-reference genotypes are (0.9985, 0.001, 0.0005), respectively. When there is ambiguity in a heterozygote, GATK prefers the reference homozygote. This is a reference bias, and while this bias is not typically problematic for medical studies, it can complicate interpretation of population genetics signals. With the Genome Sequencing and Analysis Group at the Broad Institute, we developed an alternative model that was integrated into the UnifiedGenotyper, allowing reference-bias free priors to be specified. We are using a prior (0.4995, 0.001, 0.4995). Details are at: https://www.broadinstitute.org/gatk/gatkdocs/org_broadinstitute_gatk_tools_walkers_ genotyper_UnifiedGenotyper.php#--input_prior."

I think these two examples might just do the thing:
(using either 3.x or 4.0.x)

java -jar ~/software/GenomeAnalysisTK-3.8-0-ge9d806836/GenomeAnalysisTK.jar -T HaplotypeCaller --emitRefConfidence GVCF --reference_sequence ~/hs37d5.fasta --input_file ~/file.bam --input_prior 0.001 --input_prior 0.4995

java -jar ~/software/gatk-package-4.0.3.0-local.jar -T HaplotypeCaller --emitRefConfidence GVCF--reference_sequence ~/hs37d5.fasta --input_file ~/file.bam --input_prior 0.001 --input_prior 0.4995

Does this makes sense, sorry?
These examples assume the two prior options have positional assingments to AC=1 -> 0/1 , and AC=2 -> 1/1 , ... and that as stated in the documentation about priors, AC=0 becomes 1 minus the sum of the two previous, thus effectively:

prior(0/0)=0.4995, prior(0/1)=0.001, prior(1/1)=0.4995

To understand the whole thing I'm building on these previous posts from @tommycarstensen , @magicDGS and @saeschba . Thanks guys too, and any info or extra feedback you may have, please let me know.

https://gatkforums.broadinstitute.org/gatk/discussion/8787/input-prior-default-value
https://gatkforums.broadinstitute.org/gatk/discussion/5877/caller-input-prior-option
https://gatkforums.broadinstitute.org/gatk/discussion/9489/should-it-say-ac-0-in-the-input-prior-documentation-for-the-haplotypecaller

This last question/topic makes me wonder too if AC should not be better understood here in terms of GT. I'm mostly familiar with the VCF format and AC stands there for allele count, which is a property of a site across many samples. Here in HaplotypeCaller we go over one sample at a time, not many. Maybe some inheritance from UnifiedGenotyper?

Best regards and many thanks for your comments,
Rodrigo