You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2020/09/02 13:30:55 UTC
[GitHub] [camel-quarkus] ppalaga opened a new issue #1672: Check whether it is possible to reduce the amount of RAM spent by the AS2 itest
ppalaga opened a new issue #1672:
URL: https://github.com/apache/camel-quarkus/issues/1672
As seen in this job, https://github.com/apache/camel-quarkus/pull/1670/checks?check_run_id=1061031500#step:9:1345
the native compilation of the AS2 itest takes too much RAM and too much time. I wonder whether there is any potential to strip down the amount of compiled code somehow so that less RAM and less time is needed. @JiriOndrusek would you mind having a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek edited a comment on issue #1672: Check whether it is possible to reduce the amount of RAM spent by the AS2 itest
Posted by GitBox <gi...@apache.org>.
JiriOndrusek edited a comment on issue #1672:
URL: https://github.com/apache/camel-quarkus/issues/1672#issuecomment-687069983
This idea also crossed my mind. Unfortunately ciphers depends entirely on configuration. So values like
>quarkus.camel.bouncycastle.digests=sha1,md5,sha512
quarkus.camel.bouncycastle.asymmetricCiphers=rsa,x509
depends on the security configuration defined by user (integration test in this case)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1672: Check whether it is possible to reduce the amount of RAM spent by the AS2 itest
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1672:
URL: https://github.com/apache/camel-quarkus/issues/1672#issuecomment-687004283
Hi @ppalaga,
I've looked into this problem and I see 2 items:
- Highlighted log line shows that it took 25 mins to finish as2 integration test
> [INFO] Camel Quarkus :: Integration Tests :: AS2 .......... SUCCESS [25:41 min]
On my local computer it takes usually about 5-6 mins or similar. So I'm not sure what could be wrong.
- From my local RAM consumption it seems, that about a half of the consumption is caused by presence of bouncy castle (extension support-bouncycastle). Here are some measurements:
#### Local execution with **current code**:
>[camel-quarkus-integration-test-as2-1.1.0-SNAPSHOT-runner:14157] classlist: 9,643.39 ms, 1.68 GB
...
[camel-quarkus-integration-test-as2-1.1.0-SNAPSHOT-runner:14157] [total]: 304,349.07 ms, 12.07 GB
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Execute [objcopy, --strip-debug, /home/jondruse/git/camel-quarkus/integration-tests/as2/target/camel-quarkus-integration-test-as2-1.1.0-SNAPSHOT-runner]
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in **308998ms**
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: **05:34 min**
#### Local execution **without bouncycastle**:
>[camel-quarkus-integration-test-as2-1.1.0-SNAPSHOT-runner:16243] classlist: 8,989.76 ms, 1.17 GB
...
[camel-quarkus-integration-test-as2-1.1.0-SNAPSHOT-runner:16243] [total]: 154,163.27 ms, 5.72 GB
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Execute [objcopy, --strip-debug, /home/jondruse/git/camel-quarkus/integration-tests/as2/target/camel-quarkus-integration-test-as2-1.1.0-SNAPSHOT-runner]
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in **157363ms**
...
INFO] **_BUILD FAILURE_**
[INFO] ------------------------------------------------------------------------
[INFO] Total time: **03:00 min**
---
---
I've tried one optimization by registering only those algorithms/signatures, which are really used. WIP can be seen here https://github.com/apache/camel-quarkus/pull/1686, but CI job failed with unrelated issue. It would be nice to compare results from this test with the reported results.
#### Local execution with optimization:
>[camel-quarkus-integration-test-as2-1.1.0-SNAPSHOT-runner:13100] classlist: 9,949.54 ms, 0.93 GB
...
[camel-quarkus-integration-test-as2-1.1.0-SNAPSHOT-runner:13100] [total]: 278,546.97 ms, 10.07 GB
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Execute [objcopy, --strip-debug, /home/jondruse/git/camel-quarkus/integration-tests/as2/target/camel-quarkus-integration-test-as2-1.1.0-SNAPSHOT-runner]
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in **284173ms**
...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: **05:11 min**
I can see an improvement (but not so big), what do you think?
I think that it could be possible to add some as2 quarkus properties to define really needed bouncycastle services. In case of the current integration test it would look like:
> quarkus.as2.digest=sha1,md5,sha512
quarkus.as2.asymmetric=rsa,x509
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1672: Check whether it is possible to reduce the amount of RAM spent by the AS2 itest
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1672:
URL: https://github.com/apache/camel-quarkus/issues/1672#issuecomment-687069983
This idea also crossed my mind. Unfortunately ciphers depends entirely on configuration. So values like
>quarkus.bouncycastle.digests=sha1,md5,sha512
quarkus.bouncycastle.asymmetricCiphers=rsa,x509
depends on the security configuration defined by user (integration test in this case)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1672: Check whether it is possible to reduce the amount of RAM spent by the AS2 itest
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1672:
URL: https://github.com/apache/camel-quarkus/issues/1672#issuecomment-685745063
@ppalaga Please, assign to me. I'll investigate it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #1672: Check whether it is possible to reduce the amount of RAM spent by the AS2 itest
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #1672:
URL: https://github.com/apache/camel-quarkus/issues/1672#issuecomment-687060966
BTW, is as2 using always the same fixed set of BC algos? If so, we could perhaps expose a dedicated `BuildItem` in the BC extension and let as2 processor produce it, so that the users do not need to bother.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #1672: Check whether it is possible to reduce the amount of RAM spent by the AS2 itest
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #1672:
URL: https://github.com/apache/camel-quarkus/issues/1672#issuecomment-687050209
> * Highlighted log line shows that it took 25 mins to finish as2 integration test
>
> > [INFO] Camel Quarkus :: Integration Tests :: AS2 .......... SUCCESS [25:41 min]
>
> On my local computer it takes usually about 5-6 mins or similar. So I'm not sure what could be wrong.
Yeah, it's fast on my computer too. There are two factors to take into account:
* GH Actions nodes have just two cores which are probably slower than what we have on our desk/laptops.
* Native compilation gets much slower with limited RAM. Have you activated the `ci` profile that sets `<quarkus.native.native-image-xmx>6g</quarkus.native.native-image-xmx>` https://github.com/apache/camel-quarkus/blob/master/integration-tests/as2/pom.xml#L146 ?
The 25mins on the CI are real and too much at the same time. We should try to do something about that.
Limiting the bouncy castle algos sounds like a good idea, and I'd be interested in seeing numbers in the CI. BTW, we have `camel-quarkus-support-bouncycastle` extension, so I think the code should go there and the new config options should be namespaced under ` quarkus.camel.bouncycastle.*`
If reducing bouncy castle algos is going to help here, we may try experimenting with it in other tests too where BC is included.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1672: Check whether it is possible to reduce the amount of RAM spent by the AS2 itest
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1672:
URL: https://github.com/apache/camel-quarkus/issues/1672#issuecomment-687056275
Thanks, reducing bouncy castle ciphers seems to be "doable". I'll try to prepare PR with it and yes, properties will be `quarkus.camel.bouncycastle.*`
I'll also try to limiting my RAM to see local impact .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org