[jira] [Created] (UIMA-4811) DUCC Job Driver (JD) consumes large memory near EOJ?

["Lou DeGenaro (JIRA)" <de...@uima.apache.org>]

Lou DeGenaro created UIMA-4811:
----------------------------------

             Summary: DUCC Job Driver (JD) consumes large memory near EOJ?
                 Key: UIMA-4811
                 URL: https://issues.apache.org/jira/browse/UIMA-4811
             Project: UIMA
          Issue Type: Bug
          Components: DUCC
            Reporter: Lou DeGenaro
            Assignee: Lou DeGenaro
             Fix For: future-DUCC


while true; do echo -n `date`" "; echo -n $(cat 
/cgroup/ducc/5749.5749.0/memory.usage_in_bytes)" "; echo -n $(cat /cgroup/ducc/5749.5749.0/memory.max_usage_in_bytes); echo; sleep 60; done
Mon Feb 29 20:42:55 EST 2016 490405888 493027328
Mon Feb 29 20:43:55 EST 2016 553910272 703987712
Mon Feb 29 20:44:55 EST 2016 532152320 703987712
Mon Feb 29 20:45:55 EST 2016 539492352 703987712
Mon Feb 29 20:46:55 EST 2016 537722880 703987712
Mon Feb 29 20:47:55 EST 2016 571932672 703987712
Mon Feb 29 20:48:55 EST 2016 552796160 703987712
Mon Feb 29 20:49:55 EST 2016 561709056 703987712
Mon Feb 29 20:50:55 EST 2016 cat: /cgroup/ducc/5749.5749.0/memory.usage_in_bytes: No such file or directory
 cat: /cgroup/ducc/5749.5749.0/memory.max_usage_in_bytes: No such file or directory

Note the big jump in max_memory from ~493MB to ~703MB in a one minute period. Luckily the cgroup had been pushed up to 734003200 (700MB) and so it just made it.

The previous job went much slower, running for 105 minutes. The max memory use was fairly flat until the end and then had a bump too:

Mon Feb 29 20:21:21 EST 2016 482934784 494206976
Mon Feb 29 20:22:21 EST 2016 485687296 494206976
Mon Feb 29 20:23:21 EST 2016 506396672 538968064
Mon Feb 29 20:24:21 EST 2016 490864640 538968064
Mon Feb 29 20:25:21 EST 2016 522190848 538968064
Mon Feb 29 20:26:21 EST 2016 539033600 545062912
Mon Feb 29 20:27:21 EST 2016 558301184 559087616
Mon Feb 29 20:28:21 EST 2016 545325056 562429952
Mon Feb 29 20:29:21 EST 2016 566296576 567672832
Mon Feb 29 20:30:21 EST 2016 572653568 572653568
Mon Feb 29 20:31:21 EST 2016 555024384 592904192
Mon Feb 29 20:32:21 EST 2016 562888704 592904192
Mon Feb 29 20:33:21 EST 2016 564133888 592904192
Mon Feb 29 20:34:21 EST 2016 568131584 592904192

I think there is something going on in the JD that is using a [relatively] large amount of memory near the end of jobs. Should be fairly easy to simulate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)