Martian Runtime - '3.1.0-v3.2.3'
2019-11-05 14:56:06 [build  ] Built with Go version go1.12.5
2019-11-05 14:56:06 [cmdline] mrp __10X_TCPO_premRNA_Transcript.mro 10X_TCPO_premRNA_Transcript
2019-11-05 14:56:06 [pid    ] 606
2019-11-05 14:56:06 [environ] HOSTNAME=scc-ka1.scc.bu.edu
2019-11-05 14:56:06 [environ] SGE_TASK_STEPSIZE=undefined
2019-11-05 14:56:06 [environ] SHELL=/bin/bash
2019-11-05 14:56:06 [environ] NHOSTS=1
2019-11-05 14:56:06 [environ] SGE_O_WORKDIR=/net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172
2019-11-05 14:56:06 [environ] TMPDIR=/scratch/318800.1.linga
2019-11-05 14:56:06 [environ] SGE_O_HOME=/usr4/spclpgm/kkarri
2019-11-05 14:56:06 [environ] SGE_ARCH=linux-x64
2019-11-05 14:56:06 [environ] SGE_CELL=default
2019-11-05 14:56:06 [environ] LC_ALL=C
2019-11-05 14:56:06 [environ] PYTHONUSERBASE=/share/pkg.7/cellranger/3.1.0/install/bin/cellranger-cs/3.1.0/lib
2019-11-05 14:56:06 [environ] USER=kkarri
2019-11-05 14:56:06 [environ] LD_LIBRARY_PATH=
2019-11-05 14:56:06 [environ] SGE_TASK_LAST=undefined
2019-11-05 14:56:06 [environ] TENX_SUBCMD=count
2019-11-05 14:56:06 [environ] TENX_PRODUCT=cellranger
2019-11-05 14:56:06 [environ] SGE_TASK_ID=undefined
2019-11-05 14:56:06 [environ] SGE_BINARY_PATH=/usr/local/ogs-ge2011.11.p1/sge_root/bin/linux-x64
2019-11-05 14:56:06 [environ] PATH=/share/pkg.7/cellranger/3.1.0/install/bin/cellranger-tiny-fastq/3.0.0:/share/pkg.7/cellranger/3.1.0/install/bin/cellrange...
2019-11-05 14:56:06 [environ] SGE_STDERR_PATH=/net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/TCPO_10X_prmRNA_transcript.o318800
2019-11-05 14:56:06 [environ] PWD=/net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172
2019-11-05 14:56:06 [environ] SGE_STDOUT_PATH=/net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/TCPO_10X_prmRNA_transcript.o318800
2019-11-05 14:56:06 [environ] SGE_ACCOUNT=sge
2019-11-05 14:56:06 [environ] LANG=C
2019-11-05 14:56:06 [environ] JOB_SCRIPT=/var/spool/sge/scc-ka1/job_scripts/318800
2019-11-05 14:56:06 [environ] JOB_NAME=TCPO_10X_prmRNA_transcript
2019-11-05 14:56:06 [environ] SGE_ROOT=/usr/local/sge/sge_root
2019-11-05 14:56:06 [environ] SGE_JOB_SPOOL_DIR=/var/spool/sge/scc-ka1/active_jobs/318800.1
2019-11-05 14:56:06 [environ] ENVIRONMENT=BATCH
2019-11-05 14:56:06 [environ] SHLVL=3
2019-11-05 14:56:06 [environ] HOME=/usr4/spclpgm/kkarri
2019-11-05 14:56:06 [environ] TENX_COPYRIGHT=Copyright (c) 2019 10x Genomics, Inc.  All rights reserved.
2019-11-05 14:56:06 [environ] SGE_CWD_PATH=/net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172
2019-11-05 14:56:06 [environ] NQUEUES=1
2019-11-05 14:56:06 [environ] SGE_O_LOGNAME=kkarri
2019-11-05 14:56:06 [environ] SGE_O_MAIL=/var/spool/mail/kkarri
2019-11-05 14:56:06 [environ] PYTHONPATH=/share/pkg.7/cellranger/3.1.0/install/bin/cellranger-cs/3.1.0/tenkit/lib/python:/share/pkg.7/cellranger/3.1.0/insta...
2019-11-05 14:56:06 [environ] JOB_ID=318800
2019-11-05 14:56:06 [environ] LOGNAME=kkarri
2019-11-05 14:56:06 [environ] TENX_SCRIPTDIR=/share/pkg.7/cellranger/3.1.0/install/bin/cellranger-cs/3.1.0/bin
2019-11-05 14:56:06 [environ] TENX_BANNER=cellranger count (3.1.0)
Copyright (c) 2019 10x Genomics, Inc.  All rights reserved.
-------------------------------------------------------------------------------
2019-11-05 14:56:06 [environ] XDG_DATA_DIRS=/usr4/spclpgm/kkarri/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
2019-11-05 14:56:06 [environ] SGE_TASK_FIRST=undefined
2019-11-05 14:56:06 [environ] TENX_VERSION=3.1.0
2019-11-05 14:56:06 [environ] SGE_O_PATH=/usr/java/default/jre/bin:/usr/java/default/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/...
2019-11-05 14:56:06 [environ] SGE_O_HOST=scc4
2019-11-05 14:56:06 [environ] SGE_O_SHELL=/bin/bash
2019-11-05 14:56:06 [environ] NSLOTS=16
2019-11-05 14:56:06 [environ] SGE_STDIN_PATH=/dev/null
2019-11-05 14:56:06 [environ] BASH_FUNC_module()=() {  eval `/usr/local/Modules/$MODULE_VERSION/bin/modulecmd bash $*`
}
2019-11-05 14:56:06 [environ] _=/share/pkg.7/cellranger/3.1.0/install/bin/martian-cs/v3.2.3/bin/mrp
2019-11-05 14:56:06 [environ] MROFLAGS=--vdrmode=rolling
2019-11-05 14:56:06 [environ] MROPATH=/share/pkg.7/cellranger/3.1.0/install/bin/cellranger-cs/3.1.0/tenkit/mro:/share/pkg.7/cellranger/3.1.0/install/bin/cel...
2019-11-05 14:56:06 [version] MRO Version=3.1.0
2019-11-05 14:56:06 [options] --jobmode=local
2019-11-05 14:56:06 [options] --maxjobs=0
2019-11-05 14:56:06 [options] --jobinterval=0
2019-11-05 14:56:06 [options] --vdrmode=rolling
2019-11-05 14:56:06 [options] --stackvars=false
2019-11-05 14:56:06 [options] --zip=false
2019-11-05 14:56:06 [options] --limit-loadavg=false
2019-11-05 14:56:06 [options] --noexit=false
2019-11-05 14:56:06 [options] --nopreflight=false
2019-11-05 14:56:06 [jobmngr] Job config = /share/pkg.7/cellranger/3.1.0/install/bin/martian-cs/v3.2.3/jobmanagers/config.json
2019-11-05 14:56:06 [jobmngr] Using 64 logical cores available on system.
2019-11-05 14:56:06 [jobmngr] Using 226 GB, 90% of system memory.
2019-11-05 14:56:06 [jobmngr] WARNING: configured to use 226GB of local memory, but only 212.9GB is currently available.
2019-11-05 14:56:10 [filesys] Pipestance path /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript
2019-11-05 14:56:10 [filesys] Pipestance filesystem type nfs
2019-11-05 14:56:10 [filesys] 24 TB and 1,089,487,735 inodes available.
2019-11-05 14:56:10 [filesys] Pipestance filesystem nfs mount options: rw,nosuid,nodev,relatime,rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=...
2019-11-05 14:56:10 [filesys] Bin path /share/pkg.7/cellranger/3.1.0/install/bin/martian-cs/v3.2.3/bin
2019-11-05 14:56:10 [filesys] Bin filesystem type nfs
2019-11-05 14:56:10 [filesys] Bin filesystem nfs mount options: rw,nosuid,nodev,relatime,rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,tim...
Serving UI at http://scc-ka1:35443?auth=YWnBfVrivjjjaBylqqEiXHFMo8cHbpBNS2VJmkKpte4

Running preflight checks (please wait)...
2019-11-05 14:56:10 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.EXPAND_SAMPLE_DEF
2019-11-05 14:56:11 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.EXPAND_SAMPLE_DEF.fork0.chnk0.main
2019-11-05 14:56:59 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.EXPAND_SAMPLE_DEF
2019-11-05 14:57:01 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.CELLRANGER_PREFLIGHT_LOCAL
2019-11-05 14:57:02 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.CELLRANGER_PREFLIGHT_LOCAL.fork0.chnk0.main
2019-11-05 14:57:02 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.CELLRANGER_PREFLIGHT
2019-11-05 14:57:03 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.CELLRANGER_PREFLIGHT.fork0.chnk0.main
2019-11-05 14:57:13 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.CELLRANGER_PREFLIGHT
2019-11-05 14:57:14 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.CELLRANGER_PREFLIGHT_LOCAL
2019-11-05 14:57:15 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.CHEMISTRY_DETECTOR.DETECT_CHEMISTRY
2019-11-05 14:57:15 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.CHEMISTRY_DETECTOR.DETECT_CHEMISTRY.fork0.ch...
2019-11-05 14:57:16 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.DISABLE_FEATURE_STAGES
2019-11-05 14:57:17 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.DISABLE_FEATURE_STAGES.fork0.chnk0.main
2019-11-05 14:57:17 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.CHOOSE_DIMENSION_REDUCTION
2019-11-05 14:57:18 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.CHOOSE_DIMENSION_REDUCTION.f...
2019-11-05 14:57:20 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.DISABLE_FEATURE_STAGES
... 11011 lines elided ...
2019-11-06 03:18:21 [jobmngr] Need 8192 MB of memory to start the next job (1948 available).  Waiting for jobs to complete.
2019-11-06 03:18:24 [jobmngr] 168.1GB less memory than expected was free
2019-11-06 03:18:24 [jobmngr] cgroup rss usage is 201.9GB, vs. limit of 8589934592.0
2019-11-06 03:18:28 [jobmngr] Need 8192 MB of memory to start the next job (1905 available).  Waiting for jobs to complete.
2019-11-06 03:18:31 [jobmngr] Need 8192 MB of memory to start the next job (1903 available).  Waiting for jobs to complete.
2019-11-06 03:18:41 [jobmngr] Need 8192 MB of memory to start the next job (1896 available).  Waiting for jobs to complete.
2019-11-06 03:18:53 [jobmngr] Need 8192 MB of memory to start the next job (1850 available).  Waiting for jobs to complete.
2019-11-06 03:18:59 [jobmngr] Need 8192 MB of memory to start the next job (2117 available).  Waiting for jobs to complete.
2019-11-06 03:19:05 [jobmngr] 168.0GB less memory than expected was free
2019-11-06 03:19:05 [jobmngr] cgroup rss usage is 202.3GB, vs. limit of 8589934592.0
2019-11-06 03:19:21 [jobmngr] 168.1GB less memory than expected was free
2019-11-06 03:19:21 [jobmngr] cgroup rss usage is 203.7GB, vs. limit of 8589934592.0
2019-11-06 03:19:30 [jobmngr] Need 8192 MB of memory to start the next job (1833 available).  Waiting for jobs to complete.
2019-11-06 03:19:33 [jobmngr] Need 8192 MB of memory to start the next job (1839 available).  Waiting for jobs to complete.
2019-11-06 03:19:45 [jobmngr] Need 8192 MB of memory to start the next job (1836 available).  Waiting for jobs to complete.
2019-11-06 03:19:51 [jobmngr] 168.4GB less memory than expected was free
2019-11-06 03:19:51 [jobmngr] cgroup rss usage is 205.8GB, vs. limit of 8589934592.0
2019-11-06 03:19:59 [jobmngr] Need 8192 MB of memory to start the next job (1605 available).  Waiting for jobs to complete.
2019-11-06 03:20:02 [jobmngr] Need 8192 MB of memory to start the next job (1574 available).  Waiting for jobs to complete.
2019-11-06 03:20:12 [jobmngr] 168.6GB less memory than expected was free
2019-11-06 03:20:12 [jobmngr] cgroup rss usage is 204.2GB, vs. limit of 8589934592.0
2019-11-06 03:20:30 [runtime] (update)          ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.REPORT_MOLECULES.fork0...
2019-11-06 03:20:36 [jobmngr] Need 8192 MB of memory to start the next job (1434 available).  Waiting for jobs to complete.
2019-11-06 03:20:46 [jobmngr] 168.6GB less memory than expected was free
2019-11-06 03:20:46 [jobmngr] cgroup rss usage is 205.0GB, vs. limit of 8589934592.0
2019-11-06 03:20:52 [jobmngr] Need 8192 MB of memory to start the next job (1387 available).  Waiting for jobs to complete.
2019-11-06 03:21:07 [jobmngr] Need 8192 MB of memory to start the next job (1686 available).  Waiting for jobs to complete.
2019-11-06 03:21:10 [jobmngr] 168.4GB less memory than expected was free
2019-11-06 03:21:10 [jobmngr] cgroup rss usage is 205.2GB, vs. limit of 8589934592.0
2019-11-06 03:21:17 [jobmngr] 168.7GB less memory than expected was free
2019-11-06 03:21:17 [jobmngr] cgroup rss usage is 205.5GB, vs. limit of 8589934592.0
2019-11-06 03:21:23 [jobmngr] Need 8192 MB of memory to start the next job (1559 available).  Waiting for jobs to complete.
2019-11-06 03:21:29 [jobmngr] 168.6GB less memory than expected was free
2019-11-06 03:21:29 [jobmngr] cgroup rss usage is 205.5GB, vs. limit of 8589934592.0
2019-11-06 03:21:44 [jobmngr] Need 8192 MB of memory to start the next job (1299 available).  Waiting for jobs to complete.
2019-11-06 03:21:54 [jobmngr] 168.8GB less memory than expected was free
2019-11-06 03:21:54 [jobmngr] cgroup rss usage is 206.2GB, vs. limit of 8589934592.0
2019-11-06 03:22:06 [jobmngr] Need 8192 MB of memory to start the next job (1265 available).  Waiting for jobs to complete.
2019-11-06 03:22:12 [jobmngr] Need 8192 MB of memory to start the next job (1253 available).  Waiting for jobs to complete.
2019-11-06 03:22:37 [jobmngr] Need 8192 MB of memory to start the next job (1194 available).  Waiting for jobs to complete.
2019-11-06 03:22:43 [jobmngr] 168.9GB less memory than expected was free
2019-11-06 03:22:43 [jobmngr] cgroup rss usage is 202.7GB, vs. limit of 8589934592.0
2019-11-06 03:22:44 [jobmngr] Need 8192 MB of memory to start the next job (1150 available).  Waiting for jobs to complete.
2019-11-06 03:22:44 [jobmngr] Need 8192 MB of memory to start the next job (1139 available).  Waiting for jobs to complete.
2019-11-06 03:22:48 [jobmngr] Need 8192 MB of memory to start the next job (1135 available).  Waiting for jobs to complete.
2019-11-06 03:22:51 [jobmngr] Need 8192 MB of memory to start the next job (1141 available).  Waiting for jobs to complete.
2019-11-06 03:22:54 [jobmngr] Need 8192 MB of memory to start the next job (1147 available).  Waiting for jobs to complete.
2019-11-06 03:22:57 [jobmngr] Need 8192 MB of memory to start the next job (1151 available).  Waiting for jobs to complete.
2019-11-06 03:23:00 [jobmngr] Need 8192 MB of memory to start the next job (1154 available).  Waiting for jobs to complete.
2019-11-06 03:23:04 [jobmngr] 168.9GB less memory than expected was free
2019-11-06 03:23:04 [jobmngr] cgroup rss usage is 203.0GB, vs. limit of 8589934592.0
2019-11-06 03:23:07 [jobmngr] Need 8192 MB of memory to start the next job (1475 available).  Waiting for jobs to complete.
2019-11-06 03:23:28 [jobmngr] 168.8GB less memory than expected was free
2019-11-06 03:23:28 [jobmngr] cgroup rss usage is 203.6GB, vs. limit of 8589934592.0
2019-11-06 03:23:31 [jobmngr] 168.9GB less memory than expected was free
2019-11-06 03:23:31 [jobmngr] cgroup rss usage is 203.2GB, vs. limit of 8589934592.0
2019-11-06 03:23:33 [jobmngr] Need 8192 MB of memory to start the next job (1133 available).  Waiting for jobs to complete.
2019-11-06 03:23:40 [jobmngr] 169.0GB less memory than expected was free
2019-11-06 03:23:40 [jobmngr] cgroup rss usage is 202.8GB, vs. limit of 8589934592.0
2019-11-06 03:23:55 [jobmngr] 169.2GB less memory than expected was free
2019-11-06 03:23:55 [jobmngr] cgroup rss usage is 204.5GB, vs. limit of 8589934592.0
2019-11-06 03:23:58 [jobmngr] 169.3GB less memory than expected was free
2019-11-06 03:23:58 [jobmngr] cgroup rss usage is 204.7GB, vs. limit of 8589934592.0
2019-11-06 03:24:17 [jobmngr] 169.4GB less memory than expected was free
2019-11-06 03:24:17 [jobmngr] cgroup rss usage is 206.3GB, vs. limit of 8589934592.0
2019-11-06 03:24:27 [jobmngr] Need 8192 MB of memory to start the next job (564 available).  Waiting for jobs to complete.
2019-11-06 03:24:30 [jobmngr] 170.0GB less memory than expected was free
2019-11-06 03:24:30 [jobmngr] cgroup rss usage is 202.2GB, vs. limit of 8589934592.0
2019-11-06 03:24:31 [jobmngr] Need 8192 MB of memory to start the next job (8144 available).  Waiting for jobs to complete.
2019-11-06 03:24:31 [jobmngr] Need 8192 MB of memory to start the next job (523 available).  Waiting for jobs to complete.
2019-11-06 03:24:34 [jobmngr] Need 8192 MB of memory to start the next job (544 available).  Waiting for jobs to complete.
2019-11-06 03:24:44 [jobmngr] 169.5GB less memory than expected was free
2019-11-06 03:24:44 [jobmngr] cgroup rss usage is 202.1GB, vs. limit of 8589934592.0
2019-11-06 03:24:45 [jobmngr] Need 8192 MB of memory to start the next job (493 available).  Waiting for jobs to complete.
2019-11-06 03:24:49 [jobmngr] Need 8192 MB of memory to start the next job (451 available).  Waiting for jobs to complete.
2019-11-06 03:24:58 [jobmngr] 169.7GB less memory than expected was free
2019-11-06 03:24:58 [jobmngr] cgroup rss usage is 202.1GB, vs. limit of 8589934592.0
2019-11-06 03:25:13 [jobmngr] Need 8192 MB of memory to start the next job (617 available).  Waiting for jobs to complete.
2019-11-06 03:25:23 [jobmngr] 169.6GB less memory than expected was free
2019-11-06 03:25:23 [jobmngr] cgroup rss usage is 204.0GB, vs. limit of 8589934592.0
2019-11-06 03:25:29 [jobmngr] 169.7GB less memory than expected was free
2019-11-06 03:25:29 [jobmngr] cgroup rss usage is 204.3GB, vs. limit of 8589934592.0
2019-11-06 03:25:35 [runtime] (update)          ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.REPORT_MOLECULES.fork0...
2019-11-06 03:25:35 [jobmngr] 169.8GB less memory than expected was free
2019-11-06 03:25:35 [jobmngr] cgroup rss usage is 204.7GB, vs. limit of 8589934592.0
2019-11-06 03:25:42 [jobmngr] 169.9GB less memory than expected was free
2019-11-06 03:25:42 [jobmngr] cgroup rss usage is 205.6GB, vs. limit of 8589934592.0
2019-11-06 03:25:51 [jobmngr] Need 8192 MB of memory to start the next job (86 available).  Waiting for jobs to complete.
2019-11-06 03:26:22 [jobmngr] 170.0GB less memory than expected was free
2019-11-06 03:26:22 [jobmngr] cgroup rss usage is 208.1GB, vs. limit of 8589934592.0
2019-11-06 03:26:56 [jobmngr] Need 8192 MB of memory to start the next job (8079 available).  Waiting for jobs to complete.
2019-11-06 03:26:57 [jobmngr] 170.2GB less memory than expected was free
2019-11-06 03:26:57 [jobmngr] cgroup rss usage is 205.6GB, vs. limit of 8589934592.0
2019-11-06 03:27:03 [jobmngr] 170.3GB less memory than expected was free
2019-11-06 03:27:03 [jobmngr] cgroup rss usage is 205.7GB, vs. limit of 8589934592.0
2019-11-06 03:27:09 [jobmngr] Need 8192 MB of memory to start the next job (324 available).  Waiting for jobs to complete.
2019-11-06 03:27:12 [jobmngr] 169.8GB less memory than expected was free
2019-11-06 03:27:12 [jobmngr] cgroup rss usage is 205.3GB, vs. limit of 8589934592.0
2019-11-06 03:27:22 [jobmngr] 170.1GB less memory than expected was free
2019-11-06 03:27:22 [jobmngr] cgroup rss usage is 205.9GB, vs. limit of 8589934592.0
2019-11-06 03:27:25 [jobmngr] Need 8192 MB of memory to start the next job (499 available).  Waiting for jobs to complete.
2019-11-06 03:27:31 [jobmngr] 169.6GB less memory than expected was free
2019-11-06 03:27:31 [jobmngr] cgroup rss usage is 205.6GB, vs. limit of 8589934592.0
2019-11-06 03:27:47 [jobmngr] 169.8GB less memory than expected was free
2019-11-06 03:27:47 [jobmngr] cgroup rss usage is 206.1GB, vs. limit of 8589934592.0
2019-11-06 03:28:02 [jobmngr] 169.9GB less memory than expected was free
2019-11-06 03:28:02 [jobmngr] cgroup rss usage is 206.7GB, vs. limit of 8589934592.0
2019-11-06 03:28:05 [jobmngr] Need 8192 MB of memory to start the next job (228 available).  Waiting for jobs to complete.
2019-11-06 03:28:20 [jobmngr] Need 8192 MB of memory to start the next job (387 available).  Waiting for jobs to complete.
2019-11-06 03:28:23 [jobmngr] 169.6GB less memory than expected was free
2019-11-06 03:28:23 [jobmngr] cgroup rss usage is 207.0GB, vs. limit of 8589934592.0
2019-11-06 03:28:34 [jobmngr] Need 8192 MB of memory to start the next job (351 available).  Waiting for jobs to complete.
2019-11-06 03:28:38 [jobmngr] Need 8192 MB of memory to start the next job (336 available).  Waiting for jobs to complete.
2019-11-06 03:28:42 [jobmngr] Need 8192 MB of memory to start the next job (304 available).  Waiting for jobs to complete.
2019-11-06 03:28:45 [jobmngr] Need 8192 MB of memory to start the next job (305 available).  Waiting for jobs to complete.
2019-11-06 03:28:55 [jobmngr] 169.8GB less memory than expected was free
2019-11-06 03:28:55 [jobmngr] cgroup rss usage is 204.4GB, vs. limit of 8589934592.0
2019-11-06 03:29:07 [jobmngr] 169.9GB less memory than expected was free
2019-11-06 03:29:07 [jobmngr] cgroup rss usage is 204.5GB, vs. limit of 8589934592.0
2019-11-06 03:29:10 [jobmngr] Need 8192 MB of memory to start the next job (454 available).  Waiting for jobs to complete.
2019-11-06 03:29:19 [jobmngr] 169.7GB less memory than expected was free
2019-11-06 03:29:19 [jobmngr] cgroup rss usage is 205.0GB, vs. limit of 8589934592.0
2019-11-06 03:29:26 [jobmngr] Need 8192 MB of memory to start the next job (431 available).  Waiting for jobs to complete.
2019-11-06 03:29:32 [jobmngr] 169.7GB less memory than expected was free
2019-11-06 03:29:32 [jobmngr] cgroup rss usage is 205.5GB, vs. limit of 8589934592.0
2019-11-06 03:29:41 [jobmngr] Need 8192 MB of memory to start the next job (295 available).  Waiting for jobs to complete.
2019-11-06 03:29:43 [jobmngr] Need 1024 MB of memory to start the next job (295 available).  Waiting for jobs to complete.
2019-11-06 03:29:47 [jobmngr] Need 1024 MB of memory to start the next job (283 available).  Waiting for jobs to complete.
2019-11-06 03:29:50 [jobmngr] Need 1024 MB of memory to start the next job (340 available).  Waiting for jobs to complete.
2019-11-06 03:29:56 [jobmngr] Need 1024 MB of memory to start the next job (338 available).  Waiting for jobs to complete.
2019-11-06 03:30:05 [jobmngr] Need 1024 MB of memory to start the next job (317 available).  Waiting for jobs to complete.
2019-11-06 03:30:21 [jobmngr] Need 1024 MB of memory to start the next job (284 available).  Waiting for jobs to complete.
2019-11-06 03:30:27 [jobmngr] Need 1024 MB of memory to start the next job (270 available).  Waiting for jobs to complete.
2019-11-06 03:30:33 [jobmngr] 169.8GB less memory than expected was free
2019-11-06 03:30:33 [jobmngr] cgroup rss usage is 206.4GB, vs. limit of 8589934592.0
2019-11-06 03:30:36 [runtime] (update)          ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.REPORT_MOLECULES.fork0...
2019-11-06 03:30:36 [jobmngr] Need 1024 MB of memory to start the next job (257 available).  Waiting for jobs to complete.
2019-11-06 03:30:39 [jobmngr] Need 1024 MB of memory to start the next job (584 available).  Waiting for jobs to complete.
2019-11-06 03:30:42 [jobmngr] 169.5GB less memory than expected was free
2019-11-06 03:30:42 [jobmngr] cgroup rss usage is 206.3GB, vs. limit of 8589934592.0
2019-11-06 03:30:52 [jobmngr] 169.7GB less memory than expected was free
2019-11-06 03:30:52 [jobmngr] cgroup rss usage is 204.2GB, vs. limit of 8589934592.0
2019-11-06 03:30:56 [runtime] (update)          ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.PREPROCESS_MATRIX.fork0 spli...
2019-11-06 03:30:56 [jobmngr] 169.6GB less memory than expected was free
2019-11-06 03:30:56 [jobmngr] cgroup rss usage is 203.8GB, vs. limit of 8589934592.0
2019-11-06 03:30:57 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.PREPROCESS_MATRIX
2019-11-06 03:30:57 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.PREPROCESS_MATRIX.fork0.join
2019-11-06 03:31:00 [jobmngr] 169.8GB less memory than expected was free
2019-11-06 03:31:00 [jobmngr] cgroup rss usage is 202.4GB, vs. limit of 8589934592.0
2019-11-06 03:31:03 [jobmngr] 170.0GB less memory than expected was free
2019-11-06 03:31:03 [jobmngr] cgroup rss usage is 201.3GB, vs. limit of 8589934592.0
2019-11-06 03:31:23 [jobmngr] 170.2GB less memory than expected was free
2019-11-06 03:31:23 [jobmngr] cgroup rss usage is 201.6GB, vs. limit of 8589934592.0
2019-11-06 03:31:26 [jobmngr] 170.3GB less memory than expected was free
2019-11-06 03:31:26 [jobmngr] cgroup rss usage is 201.7GB, vs. limit of 8589934592.0
2019-11-06 03:31:50 [jobmngr] 170.4GB less memory than expected was free
2019-11-06 03:31:50 [jobmngr] cgroup rss usage is 201.5GB, vs. limit of 8589934592.0
2019-11-06 03:31:54 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.PREPROCESS_MATRIX
2019-11-06 03:31:54 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_MULTIGENOME_ANALYSIS
2019-11-06 03:31:55 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_MULTIGENOME_ANALYSIS.for...
2019-11-06 03:31:56 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_FBPCA
2019-11-06 03:31:56 [runtime] (disabled)        ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_FBPCA
2019-11-06 03:31:56 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_PCA
2019-11-06 03:31:56 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_PCA.fork0.split
2019-11-06 03:31:57 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.CORRECT_CHEMISTRY_BATCH
2019-11-06 03:31:57 [runtime] (disabled)        ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.CORRECT_CHEMISTRY_BATCH
2019-11-06 03:31:59 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_MULTIGENOME_ANALYSIS
2019-11-06 03:31:59 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_MULTIGENOME_ANALYSIS.for...
2019-11-06 03:32:00 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_PCA
2019-11-06 03:32:00 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_PCA.fork0.chnk0.main
2019-11-06 03:32:03 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_MULTIGENOME_ANALYSIS
2019-11-06 03:32:10 [jobmngr] 170.6GB less memory than expected was free
2019-11-06 03:32:10 [jobmngr] cgroup rss usage is 199.8GB, vs. limit of 8589934592.0
2019-11-06 03:32:22 [jobmngr] 170.6GB less memory than expected was free
2019-11-06 03:32:22 [jobmngr] cgroup rss usage is 200.0GB, vs. limit of 8589934592.0
2019-11-06 03:32:34 [jobmngr] 170.6GB less memory than expected was free
2019-11-06 03:32:34 [jobmngr] cgroup rss usage is 201.4GB, vs. limit of 8589934592.0
2019-11-06 03:32:44 [jobmngr] 170.4GB less memory than expected was free
2019-11-06 03:32:44 [jobmngr] cgroup rss usage is 201.0GB, vs. limit of 8589934592.0
2019-11-06 03:32:59 [jobmngr] 170.5GB less memory than expected was free
2019-11-06 03:32:59 [jobmngr] cgroup rss usage is 201.4GB, vs. limit of 8589934592.0
2019-11-06 03:33:36 [jobmngr] 170.4GB less memory than expected was free
2019-11-06 03:33:36 [jobmngr] cgroup rss usage is 198.9GB, vs. limit of 8589934592.0
2019-11-06 03:34:06 [jobmngr] 170.0GB less memory than expected was free
2019-11-06 03:34:06 [jobmngr] cgroup rss usage is 199.5GB, vs. limit of 8589934592.0
2019-11-06 03:34:18 [jobmngr] 170.0GB less memory than expected was free
2019-11-06 03:34:18 [jobmngr] cgroup rss usage is 196.2GB, vs. limit of 8589934592.0
2019-11-06 03:34:18 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_PCA
2019-11-06 03:34:19 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_PCA.fork0.join
2019-11-06 03:34:24 [jobmngr] 170.0GB less memory than expected was free
2019-11-06 03:34:24 [jobmngr] cgroup rss usage is 195.8GB, vs. limit of 8589934592.0
2019-11-06 03:34:24 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_PCA
2019-11-06 03:34:25 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.CHOOSE_DIMENSION_REDUCTION_O...
2019-11-06 03:34:26 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.CHOOSE_DIMENSION_REDUCTION_O...
2019-11-06 03:34:30 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.CHOOSE_DIMENSION_REDUCTION_O...
2019-11-06 03:34:32 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_UMAP
2019-11-06 03:34:32 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_UMAP.fork0.split
2019-11-06 03:34:32 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS
2019-11-06 03:34:33 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.split
2019-11-06 03:34:33 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_GRAPH_CLUSTERING
2019-11-06 03:34:33 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_GRAPH_CLUSTERING.fork0.s...
2019-11-06 03:34:33 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_TSNE
2019-11-06 03:34:34 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_TSNE.fork0.split
2019-11-06 03:34:35 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.REPORT_MOLECULES
2019-11-06 03:34:36 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.REPORT_MOLECULES.fork0...
2019-11-06 03:34:38 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS
2019-11-06 03:34:40 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.chnk0.main
2019-11-06 03:34:41 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.chnk1.main
2019-11-06 03:34:41 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.chnk2.main
2019-11-06 03:34:41 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.chnk3.main
2019-11-06 03:34:42 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.chnk4.main
2019-11-06 03:34:42 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.chnk5.main
2019-11-06 03:34:43 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.chnk6.main
2019-11-06 03:34:43 [jobmngr] Need 3072 MB of memory to start the next job (953 available).  Waiting for jobs to complete.
2019-11-06 03:34:43 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.chnk7.main
2019-11-06 03:34:43 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.chnk8.main
2019-11-06 03:34:44 [jobmngr] Need 3072 MB of memory to start the next job (965 available).  Waiting for jobs to complete.
2019-11-06 03:34:44 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_GRAPH_CLUSTERING
2019-11-06 03:34:45 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_GRAPH_CLUSTERING.fork0.c...
2019-11-06 03:34:45 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_GRAPH_CLUSTERING.fork0.c...
2019-11-06 03:34:45 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_TSNE
2019-11-06 03:34:46 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_TSNE.fork0.chnk0.main
2019-11-06 03:34:46 [jobmngr] Need 3072 MB of memory to start the next job (1989 available).  Waiting for jobs to complete.
2019-11-06 03:34:46 [jobmngr] Need 3072 MB of memory to start the next job (1994 available).  Waiting for jobs to complete.
2019-11-06 03:34:46 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_UMAP
2019-11-06 03:34:47 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_UMAP.fork0.chnk0.main
2019-11-06 03:34:53 [jobmngr] Need 3072 MB of memory to start the next job (1969 available).  Waiting for jobs to complete.
2019-11-06 03:34:54 [jobmngr] Need 3072 MB of memory to start the next job (1969 available).  Waiting for jobs to complete.
2019-11-06 03:34:54 [jobmngr] Need 3072 MB of memory to start the next job (1969 available).  Waiting for jobs to complete.
2019-11-06 03:34:54 [jobmngr] 170.1GB less memory than expected was free
2019-11-06 03:34:54 [jobmngr] cgroup rss usage is 193.8GB, vs. limit of 8589934592.0
2019-11-06 03:34:54 [jobmngr] Need 3072 MB of memory to start the next job (1896 available).  Waiting for jobs to complete.
2019-11-06 03:34:56 [jobmngr] Need 1024 MB of memory to start the next job (872 available).  Waiting for jobs to complete.
2019-11-06 03:34:56 [jobmngr] Need 3072 MB of memory to start the next job (2920 available).  Waiting for jobs to complete.
2019-11-06 03:34:57 [jobmngr] 170.3GB less memory than expected was free
2019-11-06 03:34:57 [jobmngr] cgroup rss usage is 192.2GB, vs. limit of 8589934592.0
2019-11-06 03:34:58 [jobmngr] Need 3072 MB of memory to start the next job (2799 available).  Waiting for jobs to complete.
2019-11-06 03:35:08 [jobmngr] 170.3GB less memory than expected was free
2019-11-06 03:35:08 [jobmngr] cgroup rss usage is 194.2GB, vs. limit of 8589934592.0
2019-11-06 03:35:14 [jobmngr] 170.4GB less memory than expected was free
2019-11-06 03:35:14 [jobmngr] cgroup rss usage is 193.4GB, vs. limit of 8589934592.0
2019-11-06 03:35:14 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS
2019-11-06 03:35:14 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS.fork0.join
2019-11-06 03:35:24 [jobmngr] 170.5GB less memory than expected was free
2019-11-06 03:35:24 [jobmngr] cgroup rss usage is 193.3GB, vs. limit of 8589934592.0
2019-11-06 03:35:24 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_GRAPH_CLUSTERING
2019-11-06 03:35:24 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_GRAPH_CLUSTERING.fork0.join
2019-11-06 03:35:24 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_KMEANS
2019-11-06 03:35:35 [jobmngr] 170.6GB less memory than expected was free
2019-11-06 03:35:35 [jobmngr] cgroup rss usage is 193.8GB, vs. limit of 8589934592.0
2019-11-06 03:35:53 [jobmngr] 170.8GB less memory than expected was free
2019-11-06 03:35:53 [jobmngr] cgroup rss usage is 194.1GB, vs. limit of 8589934592.0
2019-11-06 03:36:20 [jobmngr] 170.7GB less memory than expected was free
2019-11-06 03:36:20 [jobmngr] cgroup rss usage is 194.5GB, vs. limit of 8589934592.0
2019-11-06 03:37:08 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_GRAPH_CLUSTERING
2019-11-06 03:37:09 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.MERGE_CLUSTERS
2019-11-06 03:37:09 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.MERGE_CLUSTERS.fork0.split
2019-11-06 03:37:14 [jobmngr] 170.2GB less memory than expected was free
2019-11-06 03:37:14 [jobmngr] cgroup rss usage is 193.4GB, vs. limit of 8589934592.0
2019-11-06 03:37:14 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.MERGE_CLUSTERS
2019-11-06 03:37:14 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.MERGE_CLUSTERS.fork0.chnk0.main
2019-11-06 03:37:18 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.MERGE_CLUSTERS
2019-11-06 03:37:18 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.MERGE_CLUSTERS.fork0.join
2019-11-06 03:37:33 [jobmngr] 170.3GB less memory than expected was free
2019-11-06 03:37:33 [jobmngr] cgroup rss usage is 194.4GB, vs. limit of 8589934592.0
2019-11-06 03:37:48 [jobmngr] 170.4GB less memory than expected was free
2019-11-06 03:37:48 [jobmngr] cgroup rss usage is 193.8GB, vs. limit of 8589934592.0
2019-11-06 03:37:48 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_UMAP
2019-11-06 03:37:48 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_UMAP.fork0.join
2019-11-06 03:37:56 [jobmngr] 170.6GB less memory than expected was free
2019-11-06 03:37:56 [jobmngr] cgroup rss usage is 194.1GB, vs. limit of 8589934592.0
2019-11-06 03:37:56 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_UMAP
2019-11-06 03:38:09 [jobmngr] 170.6GB less memory than expected was free
2019-11-06 03:38:09 [jobmngr] cgroup rss usage is 194.3GB, vs. limit of 8589934592.0
2019-11-06 03:38:21 [jobmngr] 170.8GB less memory than expected was free
2019-11-06 03:38:21 [jobmngr] cgroup rss usage is 194.5GB, vs. limit of 8589934592.0
2019-11-06 03:38:31 [jobmngr] 170.9GB less memory than expected was free
2019-11-06 03:38:31 [jobmngr] cgroup rss usage is 194.5GB, vs. limit of 8589934592.0
2019-11-06 03:38:58 [jobmngr] 169.9GB less memory than expected was free
2019-11-06 03:38:58 [jobmngr] cgroup rss usage is 192.5GB, vs. limit of 8589934592.0
2019-11-06 03:38:58 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.MERGE_CLUSTERS
2019-11-06 03:38:59 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.COMBINE_CLUSTERING
2019-11-06 03:38:59 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.COMBINE_CLUSTERING.fork0.chn...
2019-11-06 03:39:05 [jobmngr] 169.9GB less memory than expected was free
2019-11-06 03:39:05 [jobmngr] cgroup rss usage is 192.5GB, vs. limit of 8589934592.0
2019-11-06 03:39:05 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.COMBINE_CLUSTERING
2019-11-06 03:39:06 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION
2019-11-06 03:39:07 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:11 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION
2019-11-06 03:39:11 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:12 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:12 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:12 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:13 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:13 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:13 [jobmngr] Need 3072 MB of memory to start the next job (2102 available).  Waiting for jobs to complete.
2019-11-06 03:39:14 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:14 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:14 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:14 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:39:21 [jobmngr] Need 3072 MB of memory to start the next job (2320 available).  Waiting for jobs to complete.
2019-11-06 03:39:24 [jobmngr] Need 3072 MB of memory to start the next job (2334 available).  Waiting for jobs to complete.
2019-11-06 03:39:27 [jobmngr] Need 3072 MB of memory to start the next job (2376 available).  Waiting for jobs to complete.
2019-11-06 03:39:34 [jobmngr] Need 3072 MB of memory to start the next job (2357 available).  Waiting for jobs to complete.
2019-11-06 03:39:35 [jobmngr] Need 3072 MB of memory to start the next job (2357 available).  Waiting for jobs to complete.
2019-11-06 03:39:35 [jobmngr] 169.8GB less memory than expected was free
2019-11-06 03:39:35 [jobmngr] cgroup rss usage is 199.8GB, vs. limit of 8589934592.0
2019-11-06 03:39:35 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_TSNE
2019-11-06 03:39:36 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_TSNE.fork0.join
2019-11-06 03:39:40 [jobmngr] Need 3072 MB of memory to start the next job (2335 available).  Waiting for jobs to complete.
2019-11-06 03:39:43 [jobmngr] 169.8GB less memory than expected was free
2019-11-06 03:39:43 [jobmngr] cgroup rss usage is 196.8GB, vs. limit of 8589934592.0
2019-11-06 03:39:49 [jobmngr] Need 3072 MB of memory to start the next job (2251 available).  Waiting for jobs to complete.
2019-11-06 03:40:04 [jobmngr] Need 3072 MB of memory to start the next job (2568 available).  Waiting for jobs to complete.
2019-11-06 03:40:08 [jobmngr] 169.5GB less memory than expected was free
2019-11-06 03:40:08 [jobmngr] cgroup rss usage is 197.0GB, vs. limit of 8589934592.0
2019-11-06 03:40:26 [jobmngr] Need 3072 MB of memory to start the next job (2673 available).  Waiting for jobs to complete.
2019-11-06 03:40:35 [jobmngr] Need 3072 MB of memory to start the next job (2636 available).  Waiting for jobs to complete.
2019-11-06 03:40:39 [runtime] (update)          ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.REPORT_MOLECULES.fork0...
2019-11-06 03:40:39 [jobmngr] Need 3072 MB of memory to start the next job (2643 available).  Waiting for jobs to complete.
2019-11-06 03:40:48 [jobmngr] Need 3072 MB of memory to start the next job (2599 available).  Waiting for jobs to complete.
2019-11-06 03:40:51 [jobmngr] 169.6GB less memory than expected was free
2019-11-06 03:40:51 [jobmngr] cgroup rss usage is 198.1GB, vs. limit of 8589934592.0
2019-11-06 03:40:54 [jobmngr] 169.6GB less memory than expected was free
2019-11-06 03:40:54 [jobmngr] cgroup rss usage is 197.5GB, vs. limit of 8589934592.0
2019-11-06 03:41:06 [jobmngr] 169.8GB less memory than expected was free
2019-11-06 03:41:06 [jobmngr] cgroup rss usage is 197.1GB, vs. limit of 8589934592.0
2019-11-06 03:41:22 [jobmngr] 169.9GB less memory than expected was free
2019-11-06 03:41:22 [jobmngr] cgroup rss usage is 197.0GB, vs. limit of 8589934592.0
2019-11-06 03:41:25 [jobmngr] Need 3072 MB of memory to start the next job (2141 available).  Waiting for jobs to complete.
2019-11-06 03:41:31 [jobmngr] 170.0GB less memory than expected was free
2019-11-06 03:41:31 [jobmngr] cgroup rss usage is 196.8GB, vs. limit of 8589934592.0
2019-11-06 03:41:44 [jobmngr] 170.1GB less memory than expected was free
2019-11-06 03:41:44 [jobmngr] cgroup rss usage is 197.7GB, vs. limit of 8589934592.0
2019-11-06 03:41:59 [jobmngr] 170.3GB less memory than expected was free
2019-11-06 03:41:59 [jobmngr] cgroup rss usage is 197.5GB, vs. limit of 8589934592.0
2019-11-06 03:42:02 [jobmngr] Need 3072 MB of memory to start the next job (1806 available).  Waiting for jobs to complete.
2019-11-06 03:42:08 [jobmngr] 170.3GB less memory than expected was free
2019-11-06 03:42:08 [jobmngr] cgroup rss usage is 197.6GB, vs. limit of 8589934592.0
2019-11-06 03:42:14 [jobmngr] Need 3072 MB of memory to start the next job (1749 available).  Waiting for jobs to complete.
2019-11-06 03:42:17 [jobmngr] Need 3072 MB of memory to start the next job (2145 available).  Waiting for jobs to complete.
2019-11-06 03:42:19 [jobmngr] Need 3072 MB of memory to start the next job (2145 available).  Waiting for jobs to complete.
2019-11-06 03:42:19 [jobmngr] 170.0GB less memory than expected was free
2019-11-06 03:42:19 [jobmngr] cgroup rss usage is 196.5GB, vs. limit of 8589934592.0
2019-11-06 03:42:22 [jobmngr] Need 3072 MB of memory to start the next job (2063 available).  Waiting for jobs to complete.
2019-11-06 03:42:26 [jobmngr] 170.0GB less memory than expected was free
2019-11-06 03:42:26 [jobmngr] cgroup rss usage is 196.9GB, vs. limit of 8589934592.0
2019-11-06 03:42:41 [jobmngr] Need 3072 MB of memory to start the next job (2010 available).  Waiting for jobs to complete.
2019-11-06 03:42:50 [jobmngr] Need 3072 MB of memory to start the next job (1974 available).  Waiting for jobs to complete.
2019-11-06 03:42:53 [jobmngr] Need 3072 MB of memory to start the next job (1981 available).  Waiting for jobs to complete.
2019-11-06 03:43:05 [jobmngr] Need 3072 MB of memory to start the next job (1966 available).  Waiting for jobs to complete.
2019-11-06 03:43:09 [jobmngr] Need 3072 MB of memory to start the next job (3056 available).  Waiting for jobs to complete.
2019-11-06 03:43:15 [jobmngr] Need 3072 MB of memory to start the next job (3057 available).  Waiting for jobs to complete.
2019-11-06 03:43:24 [jobmngr] Need 3072 MB of memory to start the next job (3045 available).  Waiting for jobs to complete.
2019-11-06 03:43:32 [jobmngr] 169.1GB less memory than expected was free
2019-11-06 03:43:32 [jobmngr] cgroup rss usage is 195.7GB, vs. limit of 8589934592.0
2019-11-06 03:43:36 [jobmngr] 169.2GB less memory than expected was free
2019-11-06 03:43:36 [jobmngr] cgroup rss usage is 195.8GB, vs. limit of 8589934592.0
2019-11-06 03:43:36 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_TSNE
2019-11-06 03:43:46 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.REPORT_MOLECULES
2019-11-06 03:44:02 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUBSAMPLE_READS
2019-11-06 03:44:02 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUBSAMPLE_READS.fork0....
2019-11-06 03:44:09 [jobmngr] 169.2GB less memory than expected was free
2019-11-06 03:44:09 [jobmngr] cgroup rss usage is 196.0GB, vs. limit of 8589934592.0
2019-11-06 03:44:09 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUBSAMPLE_READS
2019-11-06 03:44:09 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUBSAMPLE_READS.fork0....
2019-11-06 03:44:09 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUBSAMPLE_READS.fork0....
2019-11-06 03:44:11 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUBSAMPLE_READS.fork0....
2019-11-06 03:44:21 [runtime] (update)          ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:44:47 [jobmngr] 169.2GB less memory than expected was free
2019-11-06 03:44:47 [jobmngr] cgroup rss usage is 200.8GB, vs. limit of 8589934592.0
2019-11-06 03:45:17 [jobmngr] 168.9GB less memory than expected was free
2019-11-06 03:45:17 [jobmngr] cgroup rss usage is 200.5GB, vs. limit of 8589934592.0
2019-11-06 03:45:36 [jobmngr] 169.0GB less memory than expected was free
2019-11-06 03:45:36 [jobmngr] cgroup rss usage is 201.2GB, vs. limit of 8589934592.0
2019-11-06 03:46:03 [jobmngr] 169.1GB less memory than expected was free
2019-11-06 03:46:03 [jobmngr] cgroup rss usage is 200.9GB, vs. limit of 8589934592.0
2019-11-06 03:46:17 [jobmngr] 169.3GB less memory than expected was free
2019-11-06 03:46:17 [jobmngr] cgroup rss usage is 200.5GB, vs. limit of 8589934592.0
2019-11-06 03:46:24 [jobmngr] 169.4GB less memory than expected was free
2019-11-06 03:46:24 [jobmngr] cgroup rss usage is 199.4GB, vs. limit of 8589934592.0
2019-11-06 03:46:39 [jobmngr] 169.5GB less memory than expected was free
2019-11-06 03:46:39 [jobmngr] cgroup rss usage is 199.5GB, vs. limit of 8589934592.0
2019-11-06 03:46:49 [jobmngr] 169.6GB less memory than expected was free
2019-11-06 03:46:49 [jobmngr] cgroup rss usage is 201.0GB, vs. limit of 8589934592.0
2019-11-06 03:46:55 [jobmngr] 169.1GB less memory than expected was free
2019-11-06 03:46:55 [jobmngr] cgroup rss usage is 199.2GB, vs. limit of 8589934592.0
2019-11-06 03:47:04 [jobmngr] 169.2GB less memory than expected was free
2019-11-06 03:47:04 [jobmngr] cgroup rss usage is 199.2GB, vs. limit of 8589934592.0
2019-11-06 03:48:01 [jobmngr] 168.5GB less memory than expected was free
2019-11-06 03:48:01 [jobmngr] cgroup rss usage is 198.2GB, vs. limit of 8589934592.0
2019-11-06 03:48:07 [jobmngr] 168.5GB less memory than expected was free
2019-11-06 03:48:07 [jobmngr] cgroup rss usage is 198.1GB, vs. limit of 8589934592.0
2019-11-06 03:48:56 [jobmngr] 168.6GB less memory than expected was free
2019-11-06 03:48:56 [jobmngr] cgroup rss usage is 198.3GB, vs. limit of 8589934592.0
2019-11-06 03:49:15 [jobmngr] 168.8GB less memory than expected was free
2019-11-06 03:49:15 [jobmngr] cgroup rss usage is 198.5GB, vs. limit of 8589934592.0
2019-11-06 03:49:36 [runtime] (update)          ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:50:02 [jobmngr] 168.8GB less memory than expected was free
2019-11-06 03:50:02 [jobmngr] cgroup rss usage is 197.6GB, vs. limit of 8589934592.0
2019-11-06 03:50:09 [jobmngr] 168.8GB less memory than expected was free
2019-11-06 03:50:09 [jobmngr] cgroup rss usage is 197.7GB, vs. limit of 8589934592.0
2019-11-06 03:50:15 [runtime] (update)          ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUBSAMPLE_READS.fork0 ...
2019-11-06 03:50:36 [jobmngr] 168.6GB less memory than expected was free
2019-11-06 03:50:36 [jobmngr] cgroup rss usage is 197.6GB, vs. limit of 8589934592.0
2019-11-06 03:51:13 [jobmngr] 168.5GB less memory than expected was free
2019-11-06 03:51:13 [jobmngr] cgroup rss usage is 197.0GB, vs. limit of 8589934592.0
2019-11-06 03:51:46 [jobmngr] 168.6GB less memory than expected was free
2019-11-06 03:51:46 [jobmngr] cgroup rss usage is 197.2GB, vs. limit of 8589934592.0
2019-11-06 03:52:10 [jobmngr] 168.8GB less memory than expected was free
2019-11-06 03:52:10 [jobmngr] cgroup rss usage is 197.2GB, vs. limit of 8589934592.0
2019-11-06 03:53:43 [jobmngr] 168.3GB less memory than expected was free
2019-11-06 03:53:43 [jobmngr] cgroup rss usage is 193.7GB, vs. limit of 8589934592.0
2019-11-06 03:53:43 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION
2019-11-06 03:53:43 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION....
2019-11-06 03:53:49 [jobmngr] 168.3GB less memory than expected was free
2019-11-06 03:53:49 [jobmngr] cgroup rss usage is 191.9GB, vs. limit of 8589934592.0
2019-11-06 03:53:57 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.RUN_DIFFERENTIAL_EXPRESSION
2019-11-06 03:53:59 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.SUMMARIZE_ANALYSIS
2019-11-06 03:53:59 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.SUMMARIZE_ANALYSIS.fork0.split
2019-11-06 03:54:02 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.SUMMARIZE_ANALYSIS
2019-11-06 03:54:02 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.SUMMARIZE_ANALYSIS.fork0.chn...
2019-11-06 03:54:04 [jobmngr] 168.4GB less memory than expected was free
2019-11-06 03:54:04 [jobmngr] cgroup rss usage is 190.0GB, vs. limit of 8589934592.0
2019-11-06 03:54:04 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUBSAMPLE_READS
2019-11-06 03:54:04 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUBSAMPLE_READS.fork0....
2019-11-06 03:54:13 [jobmngr] 168.4GB less memory than expected was free
2019-11-06 03:54:13 [jobmngr] cgroup rss usage is 190.1GB, vs. limit of 8589934592.0
2019-11-06 03:54:13 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUBSAMPLE_READS
2019-11-06 03:54:14 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUMMARIZE_BASIC_REPORTS
2019-11-06 03:54:14 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUMMARIZE_BASIC_REPORT...
2019-11-06 03:54:18 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUMMARIZE_BASIC_REPORTS
2019-11-06 03:54:19 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUMMARIZE_BASIC_REPORT...
2019-11-06 03:54:21 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.SUMMARIZE_ANALYSIS
2019-11-06 03:54:21 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.SUMMARIZE_ANALYSIS.fork0.join
2019-11-06 03:54:23 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUMMARIZE_BASIC_REPORTS
2019-11-06 03:54:24 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUMMARIZE_BASIC_REPORT...
2019-11-06 03:54:28 [jobmngr] 168.5GB less memory than expected was free
2019-11-06 03:54:28 [jobmngr] cgroup rss usage is 190.2GB, vs. limit of 8589934592.0
2019-11-06 03:54:30 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._BASIC_SC_RNA_COUNTER.SUMMARIZE_BASIC_REPORTS
2019-11-06 03:54:40 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SC_RNA_ANALYZER.SUMMARIZE_ANALYSIS
2019-11-06 03:54:44 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._CRISPR_ANALYZER.CALL_PROTOSPACERS
2019-11-06 03:54:45 [runtime] (disabled)        ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._CRISPR_ANALYZER.CALL_PROTOSPACERS
2019-11-06 03:54:51 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._CRISPR_ANALYZER._PERTURBATIONS_BY_TARGET
2019-11-06 03:54:51 [runtime] (disabled)        ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._CRISPR_ANALYZER._PERTURBATIONS_BY_TARGET
2019-11-06 03:54:51 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._CRISPR_ANALYZER._PERTURBATIONS_BY_FEATURE
2019-11-06 03:54:51 [runtime] (disabled)        ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._CRISPR_ANALYZER._PERTURBATIONS_BY_FEATURE
2019-11-06 03:54:51 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SUMMARIZE_REPORTS
2019-11-06 03:54:52 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SUMMARIZE_REPORTS.fork0.chnk0.main
2019-11-06 03:54:52 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._CRISPR_ANALYZER.SUMMARIZE_CRISPR_ANALYSIS
2019-11-06 03:54:53 [runtime] (disabled)        ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._CRISPR_ANALYZER.SUMMARIZE_CRISPR_ANALYSIS
2019-11-06 03:54:53 [runtime] (disabled)        ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER._CRISPR_ANALYZER
2019-11-06 03:55:44 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.SC_RNA_COUNTER.SUMMARIZE_REPORTS
2019-11-06 03:55:46 [runtime] (ready)           ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.CLOUPE_PREPROCESS
2019-11-06 03:55:47 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.CLOUPE_PREPROCESS.fork0.split
2019-11-06 03:55:50 [runtime] (split_complete)  ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.CLOUPE_PREPROCESS
2019-11-06 03:55:50 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.CLOUPE_PREPROCESS.fork0.chnk0.main
2019-11-06 03:56:18 [jobmngr] 168.3GB less memory than expected was free
2019-11-06 03:56:18 [jobmngr] cgroup rss usage is 193.6GB, vs. limit of 8589934592.0
2019-11-06 03:56:30 [jobmngr] 168.4GB less memory than expected was free
2019-11-06 03:56:30 [jobmngr] cgroup rss usage is 194.1GB, vs. limit of 8589934592.0
2019-11-06 03:57:22 [jobmngr] 168.5GB less memory than expected was free
2019-11-06 03:57:22 [jobmngr] cgroup rss usage is 194.3GB, vs. limit of 8589934592.0
2019-11-06 03:58:13 [jobmngr] 171.2GB less memory than expected was free
2019-11-06 03:58:13 [jobmngr] cgroup rss usage is 192.9GB, vs. limit of 8589934592.0
2019-11-06 03:58:15 [runtime] (chunks_complete) ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.CLOUPE_PREPROCESS
2019-11-06 03:58:15 [runtime] (run:local)       ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.CLOUPE_PREPROCESS.fork0.join
2019-11-06 03:58:21 [jobmngr] 168.6GB less memory than expected was free
2019-11-06 03:58:21 [jobmngr] cgroup rss usage is 190.2GB, vs. limit of 8589934592.0
2019-11-06 03:58:22 [runtime] (join_complete)   ID.10X_TCPO_premRNA_Transcript.SC_RNA_COUNTER_CS.CLOUPE_PREPROCESS
2019-11-06 03:58:25 [runtime] VDR killed 2970 files, 328 GB.

Outputs:
- Run summary HTML:                         /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/web_summary.html
- Run summary CSV:                          /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/metrics_summary.csv
- BAM:                                      /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/possorted_genome_bam.bam
- BAM index:                                /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/possorted_genome_bam.b...
- Filtered feature-barcode matrices MEX:    /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/filtered_feature_bc_ma...
- Filtered feature-barcode matrices HDF5:   /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/filtered_feature_bc_ma...
- Unfiltered feature-barcode matrices MEX:  /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/raw_feature_bc_matrix
- Unfiltered feature-barcode matrices HDF5: /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/raw_feature_bc_matrix.h5
- Secondary analysis output CSV:            /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/analysis
- Per-molecule read information:            /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/molecule_info.h5
- CRISPR-specific analysis:                 null
- Loupe Cell Browser file:                  /net/waxman-server/mnt/data/waxmanlabvm_home/kkarri/G172/10X_TCPO_premRNA_Transcript/outs/cloupe.cloupe
- Feature Reference:                        null

2019-11-06 03:58:26 [perform] Serializing pipestance performance data.
Waiting 6 seconds for UI to do final refresh.
Pipestance completed successfully!

2019-11-06 03:58:33 Shutting down.
2019-11-06 03:58:33 [jobmngr] Highest memory usage observed: {
  "rss": 149383065600,
  "shared": 1093718016,
  "vmem": 164846845952,
  "text": 181764096,
  "stack": 158354800640,
  "proc_count": 692
}
