Dear GATK team,
I have prepared 100 WES processed BAMs and try to call variants and output them in a single VCF. I used scatter-gather WDL scripts following https://software.broadinstitute.org/wdl/documentation/article?id=7614 .
I submit the job to one node which has 40 cores. I tried more cores (from 2 or more nodes), but it seems haplotypecaller only runs on one node. One node should have about 126G memory.
The problem is the job seems need too much memory and the node is unavailable to access.
Does the scatter-gather needs a lot memory ?
Part of my log file:
2018-03-28 21:20:13,921 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:11:1]: Status change from - to WaitingForReturnCodeFile
2018-03-28 21:20:13,921 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:53:1]: job id: 4800
2018-03-28 21:20:13,922 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:63:1]: Status change from - to WaitingForReturnCodeFile
2018-03-28 21:20:13,923 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:25:1]: Status change from - to WaitingForReturnCodeFile
2018-03-28 21:20:13,924 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:53:1]: Status change from - to WaitingForReturnCodeFile
2018-03-28 21:20:13,945 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:84:1]: job id: 4937
2018-03-28 21:20:13,955 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(d71d6520)jointCallingGenotypes.HaplotypeCallerERC:84:1]: Status change from - to WaitingForReturnCodeFile
stderr of one shard-0/execution:
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/work/home/jiahuan/Heart_WES/HTX/cromwell-executions/jointCallingGenotypes/d71d6520-d369-43c8-bfce-f0521358f8e9/call-HaplotypeCallerERC/shard-0/execution/tmp.2jWDLA
note: GATK3.7
Thanks!