[GitHub] [pulsar] lhotari commented on issue #12974: [Functions] Function workers require more memory on Pulsar 2.8.x+

[GitBox <gi...@apache.org>]

lhotari commented on issue #12974:
URL: https://github.com/apache/pulsar/issues/12974#issuecomment-979216153


   > This is for the Function Worker ? not for the Process that is running the Function. So it is not a big deal, as you usually run only one Worker per machine.
   
   I didn't measure how big of a deal it is. The impact was that on CircleCI I couldn't get DataStax Pulsar Helm Chart to run with Functions Worker with all .nar files. It seems to be a big deal since [CircleCI has 7.5GB RAM by default for machines](https://circleci.com/docs/2.0/executor-types/#using-machine).
   
    > On the Function Process we observe more Heap usage on JDK11 but this is due to how the JVM stores the Jar file contents and we cannot do much.
   
   There is something that can be done. In Pulsar 2.7.x, the NarClassloader was closed after usage. Similar approach could be made possible. It should be possible to resolve the issue that #9413 intended to fix by loading all required information for the built-in connectors once and then closing the NarClassloaders. This would result in lower memory consumption since all NarClassloaders for all connectors wouldn't have to be kept in memory at once.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org