You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christopher Schultz <ch...@christopherschultz.net> on 2020/07/06 15:20:31 UTC
[OT] Trying to determine the minimum heap required for an operation
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
Definitely off-topic, but it's the kind of weird thing someone here
might have experience with.
I have an offline operation I'm considering bringing "inside" my
web-based application. My only concern is memory usage: it requires
that a bunch of data be loaded from a db into memory and then
analyzed. It doesn't take long to execute -- maybe 10 seconds or so,
so the memory can be released back to the rest of the application.
I've instrumented the command-line process with a thread which runs
every .5sec and captures the used-memory, maintaining a high-water
mark and reporting it after the whole operation is done. The first
time I ran it (with no specific JVM memory-related settings), it
reported that the high-water mark was ~450MiB.
I figured that was higher than necessary, and probably just
represented a lazy GC with loads of memory, so I constrained the
process using -Xmx64M. That resulted in a 16MiB high-water mark. I
tried again with -Xmx8M and the high-water mark became 5MiB.
Is there a particularly good way to force the GC to be as aggressive
as possible to see how low I can go, or should I just keep
playing-around with the -Xmx setting.
Another option is to serialize my in-memory structure to the disk to
get a sense of the size in-memory, though it's really not the same --
it will at least get me in the right ballpark.
Any suggestions?
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DQT8ACgkQHPApP6U8
pFhvJA/+N8CfjjWvBwkXSpAW6gbozyqgcxx0zt5z9TEbC4viCZNAQchlh0WE1jxF
dQL2NS6138VNOn45QfVLru7jcVQdk6loRSK4Dxe02neAD6sEwe0v3/zsuu7CDu4x
Ln3fwohp+5YNxHAUGc4ssGtw8cilShSSLnJCHwG3mxA+grxg6QvVRVqCxV0b+sCE
ocH0MirON5G7b7zETZohtm5lPcghwDy5SBQ4fVo3mLDjUGR8woGr8SL820pQ3BuY
rjGrJ7SHxq+rVnhOrtX6c4YdEebhjR963385kwPf1ND0GoeCp8Yk/LgySxBRPAbh
2Kt0UTlbK7wYSDii6kVag1Ayrt5gCyUSrHndvVIl6SdI5gLWfZDbTB3+fvHNg+k5
x/+Xx/YPvDbXv+b7CtO663uIKV+24iaVq94W+0NVacp3P0YmAmK1CZ9ggs7HQ/SC
uu3R1wRo4yp7eWszhgfpwPHJBvb9Krtfsr8P6rhs5Ry03pzblkmzzTRCvsE85UEZ
96RN1OGx2YfPEM4+EN9+rxB1hcElLT8V420MAZd9Jx2n8JmJqdZl6DxJ7vgtvKKj
0Y60VaC211M7tzq2zZ5Sh3th3X2tePPoeJQH/vYrreM4snlM8Mt22eQ7jVFri4bY
F+mu+8DGP3csWmY16nZ0SQ+ZDUS4E9yEplOHq1YKKyHSYGHjn88=
=u12k
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] Trying to determine the minimum heap required for an operation
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Hello Christopher,
On Mon, Jul 6, 2020, 8:50 PM Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> All,
>
> Definitely off-topic, but it's the kind of weird thing someone here
> might have experience with.
>
> I have an offline operation I'm considering bringing "inside" my
> web-based application. My only concern is memory usage: it requires
> that a bunch of data be loaded from a db into memory and then
> analyzed. It doesn't take long to execute -- maybe 10 seconds or so,
> so the memory can be released back to the rest of the application.
>
> I've instrumented the command-line process with a thread which runs
> every .5sec and captures the used-memory, maintaining a high-water
> mark and reporting it after the whole operation is done. The first
> time I ran it (with no specific JVM memory-related settings), it
> reported that the high-water mark was ~450MiB.
>
> I figured that was higher than necessary, and probably just
> represented a lazy GC with loads of memory, so I constrained the
> process using -Xmx64M. That resulted in a 16MiB high-water mark. I
> tried again with -Xmx8M and the high-water mark became 5MiB.
>
> Is there a particularly good way to force the GC to be as aggressive
> as possible to see how low I can go, or should I just keep
> playing-around with the -Xmx setting.
>
Looks like you wanted to track live objects generated by your program. I
don't recall a JMX metrics of JVM to get that. If someway you can trigger
couple of Full GCs during processing and logging everything on a GC log,
then you'll get that. So, reducing -Xmx will help after some iterations.
One thing you can try using is -Dsun.rmi.dgc.client.gcInterval and server
flags and set them to 1 sec. That way you will get Full GC every 1 sec. It
worked properly with Parallel GC and doesn't work with G1 in JDK 1.8. You
can try and see if serves your purpose.
More on that property can be found at
https://docs.oracle.com/javase/8/docs/technotes/guides/rmi/sunrmiproperties.html
>
> Another option is to serialize my in-memory structure to the disk to
> get a sense of the size in-memory, though it's really not the same --
> it will at least get me in the right ballpark.
>
> Any suggestions?
>
> Thanks,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DQT8ACgkQHPApP6U8
> pFhvJA/+N8CfjjWvBwkXSpAW6gbozyqgcxx0zt5z9TEbC4viCZNAQchlh0WE1jxF
> dQL2NS6138VNOn45QfVLru7jcVQdk6loRSK4Dxe02neAD6sEwe0v3/zsuu7CDu4x
> Ln3fwohp+5YNxHAUGc4ssGtw8cilShSSLnJCHwG3mxA+grxg6QvVRVqCxV0b+sCE
> ocH0MirON5G7b7zETZohtm5lPcghwDy5SBQ4fVo3mLDjUGR8woGr8SL820pQ3BuY
> rjGrJ7SHxq+rVnhOrtX6c4YdEebhjR963385kwPf1ND0GoeCp8Yk/LgySxBRPAbh
> 2Kt0UTlbK7wYSDii6kVag1Ayrt5gCyUSrHndvVIl6SdI5gLWfZDbTB3+fvHNg+k5
> x/+Xx/YPvDbXv+b7CtO663uIKV+24iaVq94W+0NVacp3P0YmAmK1CZ9ggs7HQ/SC
> uu3R1wRo4yp7eWszhgfpwPHJBvb9Krtfsr8P6rhs5Ry03pzblkmzzTRCvsE85UEZ
> 96RN1OGx2YfPEM4+EN9+rxB1hcElLT8V420MAZd9Jx2n8JmJqdZl6DxJ7vgtvKKj
> 0Y60VaC211M7tzq2zZ5Sh3th3X2tePPoeJQH/vYrreM4snlM8Mt22eQ7jVFri4bY
> F+mu+8DGP3csWmY16nZ0SQ+ZDUS4E9yEplOHq1YKKyHSYGHjn88=
> =u12k
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: [OT] Trying to determine the minimum heap required for an
operation
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
John,
On 7/6/20 16:12, John.E.Gregg@wellsfargo.com.INVALID wrote:
> Chris,
>> John,
>>
>> On 7/6/20 11:48, John.E.Gregg@wellsfargo.com.INVALID wrote:
>>> Chris,
>>>
>>>
>>>> -----Original Message----- From: Christopher Schultz
>>>> <ch...@christopherschultz.net> Sent: Monday, July 06, 2020
>>>> 10:21 AM To: Tomcat Users List <us...@tomcat.apache.org>
>>>> Subject: [OT] Trying to determine the minimum heap required
>>>> for an operation
>>>>
>>> All,
>>>
>>> Definitely off-topic, but it's the kind of weird thing someone
>>> here might have experience with.
>>>
>>> I have an offline operation I'm considering bringing "inside"
>>> my web-based application. My only concern is memory usage: it
>>> requires that a bunch of data be loaded from a db into memory
>>> and then analyzed. It doesn't take long to execute -- maybe 10
>>> seconds or so, so the memory can be released back to the rest
>>> of the application.
>>>
>>> I've instrumented the command-line process with a thread which
>>> runs every .5sec and captures the used-memory, maintaining a
>>> high-water mark and reporting it after the whole operation is
>>> done. The first time I ran it (with no specific JVM
>>> memory-related settings), it reported that the high-water mark
>>> was ~450MiB.
>>>
>>> I figured that was higher than necessary, and probably just
>>> represented a lazy GC with loads of memory, so I constrained
>>> the process using -Xmx64M. That resulted in a 16MiB high-water
>>> mark. I tried again with -Xmx8M and the high-water mark became
>>> 5MiB.
>>>
>>> Is there a particularly good way to force the GC to be as
>>> aggressive as possible to see how low I can go, or should I
>>> just keep playing-around with the -Xmx setting.
>>>
>>> Another option is to serialize my in-memory structure to the
>>> disk to get a sense of the size in-memory, though it's really
>>> not the same -- it will at least get me in the right ballpark.
>>>
>>> Any suggestions?
>>>
>>> Thanks, -chris
>>>>
>>>> -------------------------------------------------------------------
- --
>>>>
>>>>
>>
>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail:
>>>> users-help@tomcat.apache.org
>>>
>>>
>>> I guess I’m that person with the weird experiences.
>>>
>>> Is memory or CPU in short supply? If not, I don't think you'll
>>> have
>> a problem. This isn't 1997 anymore. I do think you should run a
>> realistic load test, however.
>>
>> No specific problem exists, but this is a multi-user web
>> application. Usually somewhere around 500 - 1000 users logged-in
>> at once. Session size is typically quite low -- only a handful of
>> small objects present with lots of sharing of "large" objects and
>> structures. Heap size is set to max 1GiB on each server and
>> memory usage shows a beautiful sawtooth pattern hovering around
>> ~400MiB for days at a time.
>>
>> I will certainly limit the number of these operations that can
>> occur at once, and they should be relatively rare. My test
>> example was using a small data set, but the size of the data-set
>> varies wildly with the client, so I have to be careful for the
>> larger ones.
>>
>> Busting the heap isn't something I'd like to have happen.
>>
>>> To me the most important GC metric is time spent per
>> minute/hour/etc. The next most important metric is individual
>> pause durations. Through testing you'll see what those numbers
>> are. I work with some large apps that have multi-GB heaps and
>> it's rare to see GC time being over 1-2%. IOW, 600-1200ms per
>> minute. Often it's a fraction of a percent. With those small
>> numbers you're talking about, I don't think you'll have any
>> trouble in this area unless the server is very heavily loaded.
>>
>> Actually, I'm not super concerned about performance of the GC
>> itself. I was just wondering if there was a way to ask the JVM
>> "if you *had* to accomplish this task with the smallest possible
>> heap, what would it be?"
>>
>>> Be sure to enable verbose GC. In java 8, it's something like
>>> this:
>>>
>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>>> -XX:+PrintGCDateStamps
>> - -Xloggc:/path/to/gc.log
>>>
>>> Run tests with and without the changes. You can analyze the GC
>>> output with tools like GCEasy and GCViewer.
>> Sure.
>>
>> Again, I'm more concerned with the overhead that will be required
>> for a particular operation, so I can predict when running such an
>> operation might end up endangering the application server's heap
>> -- and therefore the logged-in users.
>>
>> Theoretically, if the thread hits a heap-full error, the thread
>> will experience an OOME, release it's (temporary) large object
>> tree, and the GC will be able to recover, but after an OOME it's
>> never a great plan to trust the JVM for very long.
>>
>> - -chris -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with
>> Thunderbird - https://www.enigmail.net/
>>
>> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DTR4ACgkQHPApP
>> 6U8
>> pFiAkQ//cWQ/CL35LJcRIervUhnByPXg/TN1MhfOl66zXx4upJcIpPXgBuIkigbe
>> 9d9y/jFnRCyHsFodSEsjtT/q2CxD7k30DIAwRrTaGxzrz60QlD/+t8l3getT9xot
>> s0bAxvlpjZTvvhTtpAAv9hkSwJuMxxECksbqmYXaO/rtoBu/N9R8MCjPz4cihTa
>> B dLZZ32Ibhg7tn0VBhwaJYz8AlYK3qJLUfISBU8h3WXXpmrbw48wDmkMYtdwU
>> PcoL
>> aWw9UbOolLj0EZiCh4QjCXb404pLbqTe0Hbuy7FBNv1rB8RDOFj8vWo8eotshqi
>> J
>> c8fVY5jxHV9dqc0S2A6cCjoTrwsvUPimiykhCnrvCUpnif1I90H2b181sCtEyIyV
>> XTADe/ore5K07DJGhFSenAYBvrU19TuiRL7NGH980DndYGns8woQ9BvR4WHE
>> BDuf 15qkHdwBtjD6o5D6X5Mmyf6x6WbafhI6gb3hBPUGMEtxAhgvHNmIGX7+QV
>> QHdtat
>> 9+QSuu2viArftOKHfFpM85O25QuyfkPi4nGz1S86vcgd6q+N1Lk/qVTyJo2VJSm
>> g Jq6DEEko/YaLIujgjU6rQdPoF8oeRAtvWXsnQ3Yw/x7Wv3rRH4l3NMhwv139nC
>> Pb 2Jt/LYAC4v48O1COt0huuCAu0dWIJqky3VIeIcI9lOpAZ1Jm3Do= =svYj
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> What did you mean by "limit the number of these operations that
> can occur at once?" Initially I got the impression that this is
> just one batch operation that will run every minute, hour, or
> something like that.
No. This is something that might be user-initiated. Hence my concern
for it to be using a lot of heap space. I don't want 500 users all
requesting this heavy operation at once, so the requests will be
serialized and/or queued.
> As a starting point, you could take the heap size of your existing
> "external" operation and add it to your existing web traffic jvm.
The current external operation has no explicit limits, so it ggets
like a 4GiB heap or whatever the default when there's plenty of RAM to
go around. This exercise was intended to determine that required size.
I've been playing-around with some large client data sets and it looks
like one of the the largest (currently) requires 48MiB to chew-through
their data. It also took two hours. So I'm starting to think that
bringing this "inside" my web application is not a good idea: the web
application will just be used to kick-off the process and maybe
observe its progress, but the actual data-processing must occur elsewher
e.
> I don't want to write a treatise on how to tune the heap size, but
> Charlie Hunt and Binu John's Java Performance book covers GC well.
> It's not rocket surgery. You can do a lot with a few basic
> principles.
Again, I wasn't looking to tune my GC. I was looking to find out how
much memory a process "really" takes so that I can make sure I don't
hose my existing process.
But I think the point, now, is moot as I'm very unlikely to move this
process inside my web application.
Thanks!
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8Dn3oACgkQHPApP6U8
pFiKuA//Wnn/LmgbNzf7xEKTRdgyLA6O8cAUKpD8eBB8iQIh3gSfvq0ZVZ7c+WCZ
NhLjSu4d0V44oc8xhz/k2GDEIdjgjCh71K9nWVuDUQL1p5R8N9wY8gcDQQ7OpKuO
B08WeWay6KMc1XzExot0WmVfL1qI0dEvgZA8ycIDQlDjPfoCwoQIQJoepfpYWXxi
rnOsZE7pugmnSJ2w193YJYHZcdFrquFgIXakDprAjaTkg51xALoOyH5gD47+8jBc
q5fw4VJUwxNTnKjiRPu+yjEmCxCvScNE0yFwY14fzoDmcr5FdMkzJOzhnvD1aZbV
k3oyjRJxSw+5/rgAki0B3xFCqMRXfwZQpGhPqN38yNTb1j3WGjfUs6aDTWPqGZXU
jeqYCL1MPLcvi3zClkWNdFIcS5Zam+/JSteKGadgFsAvsKOGsnot7CqUv1VKBa5W
pW23U0Zgc+M9b5XcANhGlDil4Xh90QcxBp0/koR0Bz4fxMw2llVCA2efNcnerYD4
UKcC66ZW7sd4qmf8Fjb/Dh6s32p1F3WvL8WZNUDlezyD8he+z0keF3C1haJ0yZAt
axggn0E+aUGMjUUBm70ySlg3ihF/EJJZKVOLfSBLibmnjXrxB7BuluXjFFbTT6H5
mbgRbk27uiNcvRbJzeNGAOKbhq7+S1BdpVamE0ZAfjPwPMSnOlE=
=/CuV
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [OT] Trying to determine the minimum heap required for an
operation
Posted by Jo...@wellsfargo.com.INVALID.
Chris,
> John,
>
> On 7/6/20 11:48, John.E.Gregg@wellsfargo.com.INVALID wrote:
> > Chris,
> >
> >
> >> -----Original Message----- From: Christopher Schultz
> >> <ch...@christopherschultz.net> Sent: Monday, July 06, 2020 10:21 AM
> >> To: Tomcat Users List <us...@tomcat.apache.org> Subject: [OT] Trying
> >> to determine the minimum heap required for an operation
> >>
> > All,
> >
> > Definitely off-topic, but it's the kind of weird thing someone here
> > might have experience with.
> >
> > I have an offline operation I'm considering bringing "inside" my
> > web-based application. My only concern is memory usage: it requires
> > that a bunch of data be loaded from a db into memory and then
> > analyzed. It doesn't take long to execute -- maybe 10 seconds or so,
> > so the memory can be released back to the rest of the application.
> >
> > I've instrumented the command-line process with a thread which runs
> > every .5sec and captures the used-memory, maintaining a high-water
> > mark and reporting it after the whole operation is done. The first
> > time I ran it (with no specific JVM memory-related settings), it
> > reported that the high-water mark was ~450MiB.
> >
> > I figured that was higher than necessary, and probably just
> > represented a lazy GC with loads of memory, so I constrained the
> > process using -Xmx64M. That resulted in a 16MiB high-water mark. I
> > tried again with -Xmx8M and the high-water mark became 5MiB.
> >
> > Is there a particularly good way to force the GC to be as aggressive
> > as possible to see how low I can go, or should I just keep
> > playing-around with the -Xmx setting.
> >
> > Another option is to serialize my in-memory structure to the disk to
> > get a sense of the size in-memory, though it's really not the same --
> > it will at least get me in the right ballpark.
> >
> > Any suggestions?
> >
> > Thanks, -chris
> >>
> >> ---------------------------------------------------------------------
> >>
> >>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
> > I guess I’m that person with the weird experiences.
> >
> > Is memory or CPU in short supply? If not, I don't think you'll have
> a problem. This isn't 1997 anymore. I do think you should run a realistic load
> test, however.
>
> No specific problem exists, but this is a multi-user web application.
> Usually somewhere around 500 - 1000 users logged-in at once. Session size is
> typically quite low -- only a handful of small objects present with lots of
> sharing of "large" objects and structures. Heap size is set to max 1GiB on each
> server and memory usage shows a beautiful sawtooth pattern hovering
> around ~400MiB for days at a time.
>
> I will certainly limit the number of these operations that can occur at once,
> and they should be relatively rare. My test example was using a small data
> set, but the size of the data-set varies wildly with the client, so I have to be
> careful for the larger ones.
>
> Busting the heap isn't something I'd like to have happen.
>
> > To me the most important GC metric is time spent per
> minute/hour/etc. The next most important metric is individual pause
> durations. Through testing you'll see what those numbers are. I work with
> some large apps that have multi-GB heaps and it's rare to see GC time being
> over 1-2%. IOW, 600-1200ms per minute. Often it's a fraction of a percent.
> With those small numbers you're talking about, I don't think you'll have any
> trouble in this area unless the server is very heavily loaded.
>
> Actually, I'm not super concerned about performance of the GC itself.
> I was just wondering if there was a way to ask the JVM "if you *had* to
> accomplish this task with the smallest possible heap, what would it be?"
>
> > Be sure to enable verbose GC. In java 8, it's something like
> > this:
> >
> > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
> - -Xloggc:/path/to/gc.log
> >
> > Run tests with and without the changes. You can analyze the GC output
> > with tools like GCEasy and GCViewer.
> Sure.
>
> Again, I'm more concerned with the overhead that will be required for a
> particular operation, so I can predict when running such an operation might
> end up endangering the application server's heap -- and therefore the
> logged-in users.
>
> Theoretically, if the thread hits a heap-full error, the thread will experience
> an OOME, release it's (temporary) large object tree, and the GC will be able
> to recover, but after an OOME it's never a great plan to trust the JVM for
> very long.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DTR4ACgkQHPApP
> 6U8
> pFiAkQ//cWQ/CL35LJcRIervUhnByPXg/TN1MhfOl66zXx4upJcIpPXgBuIkigbe
> 9d9y/jFnRCyHsFodSEsjtT/q2CxD7k30DIAwRrTaGxzrz60QlD/+t8l3getT9xot
> s0bAxvlpjZTvvhTtpAAv9hkSwJuMxxECksbqmYXaO/rtoBu/N9R8MCjPz4cihTa
> B
> dLZZ32Ibhg7tn0VBhwaJYz8AlYK3qJLUfISBU8h3WXXpmrbw48wDmkMYtdwU
> PcoL
> aWw9UbOolLj0EZiCh4QjCXb404pLbqTe0Hbuy7FBNv1rB8RDOFj8vWo8eotshqi
> J
> c8fVY5jxHV9dqc0S2A6cCjoTrwsvUPimiykhCnrvCUpnif1I90H2b181sCtEyIyV
> XTADe/ore5K07DJGhFSenAYBvrU19TuiRL7NGH980DndYGns8woQ9BvR4WHE
> BDuf
> 15qkHdwBtjD6o5D6X5Mmyf6x6WbafhI6gb3hBPUGMEtxAhgvHNmIGX7+QV
> QHdtat
> 9+QSuu2viArftOKHfFpM85O25QuyfkPi4nGz1S86vcgd6q+N1Lk/qVTyJo2VJSm
> g
> Jq6DEEko/YaLIujgjU6rQdPoF8oeRAtvWXsnQ3Yw/x7Wv3rRH4l3NMhwv139nC
> Pb
> 2Jt/LYAC4v48O1COt0huuCAu0dWIJqky3VIeIcI9lOpAZ1Jm3Do=
> =svYj
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
What did you mean by "limit the number of these operations that can occur at once?" Initially I got the impression that this is just one batch operation that will run every minute, hour, or something like that.
As a starting point, you could take the heap size of your existing "external" operation and add it to your existing web traffic jvm.
I don't want to write a treatise on how to tune the heap size, but Charlie Hunt and Binu John's Java Performance book covers GC well. It's not rocket surgery. You can do a lot with a few basic principles.
Re: [OT] Trying to determine the minimum heap required for an
operation
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
John,
On 7/6/20 11:48, John.E.Gregg@wellsfargo.com.INVALID wrote:
> Chris,
>
>
>> -----Original Message----- From: Christopher Schultz
>> <ch...@christopherschultz.net> Sent: Monday, July 06, 2020 10:21
>> AM To: Tomcat Users List <us...@tomcat.apache.org> Subject: [OT]
>> Trying to determine the minimum heap required for an operation
>>
> All,
>
> Definitely off-topic, but it's the kind of weird thing someone here
> might have experience with.
>
> I have an offline operation I'm considering bringing "inside" my
> web-based application. My only concern is memory usage: it requires
> that a bunch of data be loaded from a db into memory and then
> analyzed. It doesn't take long to execute -- maybe 10 seconds or
> so, so the memory can be released back to the rest of the
> application.
>
> I've instrumented the command-line process with a thread which runs
> every .5sec and captures the used-memory, maintaining a high-water
> mark and reporting it after the whole operation is done. The first
> time I ran it (with no specific JVM memory-related settings), it
> reported that the high-water mark was ~450MiB.
>
> I figured that was higher than necessary, and probably just
> represented a lazy GC with loads of memory, so I constrained the
> process using -Xmx64M. That resulted in a 16MiB high-water mark. I
> tried again with -Xmx8M and the high-water mark became 5MiB.
>
> Is there a particularly good way to force the GC to be as
> aggressive as possible to see how low I can go, or should I just
> keep playing-around with the -Xmx setting.
>
> Another option is to serialize my in-memory structure to the disk
> to get a sense of the size in-memory, though it's really not the
> same -- it will at least get me in the right ballpark.
>
> Any suggestions?
>
> Thanks, -chris
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> I guess I’m that person with the weird experiences.
>
> Is memory or CPU in short supply? If not, I don't think you'll
> have
a problem. This isn't 1997 anymore. I do think you should run a
realistic load test, however.
No specific problem exists, but this is a multi-user web application.
Usually somewhere around 500 - 1000 users logged-in at once. Session
size is typically quite low -- only a handful of small objects present
with lots of sharing of "large" objects and structures. Heap size is
set to max 1GiB on each server and memory usage shows a beautiful
sawtooth pattern hovering around ~400MiB for days at a time.
I will certainly limit the number of these operations that can occur
at once, and they should be relatively rare. My test example was using
a small data set, but the size of the data-set varies wildly with the
client, so I have to be careful for the larger ones.
Busting the heap isn't something I'd like to have happen.
> To me the most important GC metric is time spent per
minute/hour/etc. The next most important metric is individual pause
durations. Through testing you'll see what those numbers are. I work
with some large apps that have multi-GB heaps and it's rare to see GC
time being over 1-2%. IOW, 600-1200ms per minute. Often it's a
fraction of a percent. With those small numbers you're talking about,
I don't think you'll have any trouble in this area unless the server
is very heavily loaded.
Actually, I'm not super concerned about performance of the GC itself.
I was just wondering if there was a way to ask the JVM "if you *had*
to accomplish this task with the smallest possible heap, what would it
be?"
> Be sure to enable verbose GC. In java 8, it's something like
> this:
>
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
- -Xloggc:/path/to/gc.log
>
> Run tests with and without the changes. You can analyze the GC
> output with tools like GCEasy and GCViewer.
Sure.
Again, I'm more concerned with the overhead that will be required for
a particular operation, so I can predict when running such an
operation might end up endangering the application server's heap --
and therefore the logged-in users.
Theoretically, if the thread hits a heap-full error, the thread will
experience an OOME, release it's (temporary) large object tree, and
the GC will be able to recover, but after an OOME it's never a great
plan to trust the JVM for very long.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DTR4ACgkQHPApP6U8
pFiAkQ//cWQ/CL35LJcRIervUhnByPXg/TN1MhfOl66zXx4upJcIpPXgBuIkigbe
9d9y/jFnRCyHsFodSEsjtT/q2CxD7k30DIAwRrTaGxzrz60QlD/+t8l3getT9xot
s0bAxvlpjZTvvhTtpAAv9hkSwJuMxxECksbqmYXaO/rtoBu/N9R8MCjPz4cihTaB
dLZZ32Ibhg7tn0VBhwaJYz8AlYK3qJLUfISBU8h3WXXpmrbw48wDmkMYtdwUPcoL
aWw9UbOolLj0EZiCh4QjCXb404pLbqTe0Hbuy7FBNv1rB8RDOFj8vWo8eotshqiJ
c8fVY5jxHV9dqc0S2A6cCjoTrwsvUPimiykhCnrvCUpnif1I90H2b181sCtEyIyV
XTADe/ore5K07DJGhFSenAYBvrU19TuiRL7NGH980DndYGns8woQ9BvR4WHEBDuf
15qkHdwBtjD6o5D6X5Mmyf6x6WbafhI6gb3hBPUGMEtxAhgvHNmIGX7+QVQHdtat
9+QSuu2viArftOKHfFpM85O25QuyfkPi4nGz1S86vcgd6q+N1Lk/qVTyJo2VJSmg
Jq6DEEko/YaLIujgjU6rQdPoF8oeRAtvWXsnQ3Yw/x7Wv3rRH4l3NMhwv139nCPb
2Jt/LYAC4v48O1COt0huuCAu0dWIJqky3VIeIcI9lOpAZ1Jm3Do=
=svYj
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [OT] Trying to determine the minimum heap required for an
operation
Posted by Jo...@wellsfargo.com.INVALID.
Chris,
> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Monday, July 06, 2020 10:21 AM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: [OT] Trying to determine the minimum heap required for an
> operation
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> All,
>
> Definitely off-topic, but it's the kind of weird thing someone here might have
> experience with.
>
> I have an offline operation I'm considering bringing "inside" my web-based
> application. My only concern is memory usage: it requires that a bunch of
> data be loaded from a db into memory and then analyzed. It doesn't take
> long to execute -- maybe 10 seconds or so, so the memory can be released
> back to the rest of the application.
>
> I've instrumented the command-line process with a thread which runs every
> .5sec and captures the used-memory, maintaining a high-water mark and
> reporting it after the whole operation is done. The first time I ran it (with no
> specific JVM memory-related settings), it reported that the high-water mark
> was ~450MiB.
>
> I figured that was higher than necessary, and probably just represented a
> lazy GC with loads of memory, so I constrained the process using -Xmx64M.
> That resulted in a 16MiB high-water mark. I tried again with -Xmx8M and the
> high-water mark became 5MiB.
>
> Is there a particularly good way to force the GC to be as aggressive as
> possible to see how low I can go, or should I just keep playing-around with
> the -Xmx setting.
>
> Another option is to serialize my in-memory structure to the disk to get a
> sense of the size in-memory, though it's really not the same -- it will at least
> get me in the right ballpark.
>
> Any suggestions?
>
> Thanks,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DQT8ACgkQHPApP
> 6U8
> pFhvJA/+N8CfjjWvBwkXSpAW6gbozyqgcxx0zt5z9TEbC4viCZNAQchlh0WE1jx
> F
> dQL2NS6138VNOn45QfVLru7jcVQdk6loRSK4Dxe02neAD6sEwe0v3/zsuu7CDu
> 4x
> Ln3fwohp+5YNxHAUGc4ssGtw8cilShSSLnJCHwG3mxA+grxg6QvVRVqCxV0b+
> sCE
> ocH0MirON5G7b7zETZohtm5lPcghwDy5SBQ4fVo3mLDjUGR8woGr8SL820pQ3
> BuY
> rjGrJ7SHxq+rVnhOrtX6c4YdEebhjR963385kwPf1ND0GoeCp8Yk/LgySxBRPAbh
> 2Kt0UTlbK7wYSDii6kVag1Ayrt5gCyUSrHndvVIl6SdI5gLWfZDbTB3+fvHNg+k5
> x/+Xx/YPvDbXv+b7CtO663uIKV+24iaVq94W+0NVacp3P0YmAmK1CZ9ggs7HQ
> /SC
> uu3R1wRo4yp7eWszhgfpwPHJBvb9Krtfsr8P6rhs5Ry03pzblkmzzTRCvsE85UEZ
> 96RN1OGx2YfPEM4+EN9+rxB1hcElLT8V420MAZd9Jx2n8JmJqdZl6DxJ7vgtvKKj
> 0Y60VaC211M7tzq2zZ5Sh3th3X2tePPoeJQH/vYrreM4snlM8Mt22eQ7jVFri4bY
> F+mu+8DGP3csWmY16nZ0SQ+ZDUS4E9yEplOHq1YKKyHSYGHjn88=
> =u12k
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
I guess I’m that person with the weird experiences.
Is memory or CPU in short supply? If not, I don't think you'll have a problem. This isn't 1997 anymore. I do think you should run a realistic load test, however.
To me the most important GC metric is time spent per minute/hour/etc. The next most important metric is individual pause durations. Through testing you'll see what those numbers are. I work with some large apps that have multi-GB heaps and it's rare to see GC time being over 1-2%. IOW, 600-1200ms per minute. Often it's a fraction of a percent. With those small numbers you're talking about, I don't think you'll have any trouble in this area unless the server is very heavily loaded.
Be sure to enable verbose GC. In java 8, it's something like this:
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/path/to/gc.log
Run tests with and without the changes. You can analyze the GC output with tools like GCEasy and GCViewer.
John