You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/04/08 02:11:50 UTC
[GitHub] [druid] martin-g opened a new pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
martin-g opened a new pull request #10958:
URL: https://github.com/apache/druid/pull/10958
Documentation:
- https://blog.travis-ci.com/2020-09-11-arm-on-aws
- https://aws.amazon.com/blogs/opensource/getting-started-with-travis-ci-com-on-aws-graviton2/
Trying to fix the problem described at https://github.com/apache/druid/pull/10953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
suneet-s commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-815395498
Closed and re-opened to see if it would re-trigger travis. It appears that it hasn't :(
@martin-g could you merge master back in to this PR so we can see what the test run on Travis would look like? Since this test job only runs unit tests, we still can't validate that Druid would work correctly on ARM64 machines. Was this the intention of this PR? Or did you just want to validate that Druid compiles on these types of machines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g edited a comment on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g edited a comment on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-799397025
I've reworked the PR to not build the Web Console, so node-sass is no more a problem, but now it fails because of Hyperic Sigar library. This library has no native binaries for aarch64. I really wonder how the old builds passed at all.
And Hyperic Sigar is not maintained since 2016, so there is no chance for adding support for ARM64. There are 3 Pull Requests and all stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-818494034
Closing in favor of https://github.com/apache/druid/pull/11109
@suneet-s The problem was that .travis.yml got corrupted due to a wrong indentation. I've created a diff from master to my branch and created a new PR. This also didn't help and they I realized that it must be something YAML related. Please review #11109
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-799397025
I've reworked the PR to not build the Web Console, so node-sass is no more a problem, but now it fails because of Hyperic Sigar library. This library has no native binaries for aarch64. I really wonder how the old builds passed at all.
And Hyperic Sigar is not maintained since 2016, so there is no chance for adding support for ARM4. There are 3 Pull Requests and all stale.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-817753646
I've created a new Github user (martin-g2) and a new PR (https://github.com/apache/druid/pull/11095) but still no TravisCI build ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s closed pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
suneet-s closed pull request #10958:
URL: https://github.com/apache/druid/pull/10958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-793002898
Please check the CI failure for ARM build. It seems legit.
```
[INFO] Downloading binary from https://github.com/sass/node-sass/releases/download/v4.13.1/linux-arm64-64_binding.node
[ERROR] Cannot download "https://github.com/sass/node-sass/releases/download/v4.13.1/linux-arm64-64_binding.node":
[ERROR]
[ERROR] HTTP error 404 Not Found
[ERROR]
[ERROR] Hint: If github.com is not accessible in your location
[ERROR] try setting a proxy via HTTP_PROXY, e.g.
[ERROR]
[ERROR] export HTTP_PROXY=http://example.com:1234
[ERROR]
[ERROR] or configure npm proxy via
[ERROR]
[ERROR] npm config set proxy http://example.com:8080
...
[ERROR] gyp verb check python checking for Python executable "python2" in the PATH
[ERROR] gyp verb `which` failed Error: not found: python2
[ERROR] gyp verb `which` failed at getNotFoundError (/home/travis/build/apache/druid/web-console/node_modules/which/which.js:13:12)
[ERROR] gyp verb `which` failed at F (/home/travis/build/apache/druid/web-console/node_modules/which/which.js:68:19)
[ERROR] gyp verb `which` failed at E (/home/travis/build/apache/druid/web-console/node_modules/which/which.js:80:29)
[ERROR] gyp verb `which` failed at /home/travis/build/apache/druid/web-console/node_modules/which/which.js:89:16
[ERROR] gyp verb `which` failed at /home/travis/build/apache/druid/web-console/node_modules/isexe/index.js:42:5
[ERROR] gyp verb `which` failed at /home/travis/build/apache/druid/web-console/node_modules/isexe/mode.js:8:5
[ERROR] gyp verb `which` failed at FSReqWrap.oncomplete (fs.js:154:21)
...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-812359111
Any feedback ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-808209370
The TravisCI job for testing on ARM64 has been reworked and passes successfully now: https://travis-ci.com/github/apache/druid/builds/221285346
The new job builds and tests just the main sub-modules because:
- running all modules does not finish in 50 mins (Travis max time for a job)
- `sql` module fails because SqlResourceTest#testTooManyRequests() succeeds for all 3 requests. This is because TravisCI ARM64 nodes have reduced CPU power and can run up to 2 threads
- the distribution job is not executed on ARM64 because of the flacky failures at https://github.com/apache/druid/pull/10953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
suneet-s commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-817859855
@martin-g I think you don't need any travis credits to run the tests. The tests should be running on Apache's account. I don't have any suggestions on how to trick travis into kicking off the tests other than what you've already tried. Let me think about it some more, and if I can come up with a way, I'll comment on this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-815506651
@suneet-s I've rebased the branch with the latest master and force pushed but this also didn't trigger a build at Travis for some reason.
I see that the last build for this PR has been 13 days ago - https://travis-ci.com/github/apache/druid/pull_requests
I just pushed an empty commit (non-force) but again nothing ...
I needed to reduce the built projects for ARM64 because otherwise I have to create a separate job for each "feature" (processing, indexing, server, ...).
Another option is to add something like https://github.com/apache/velocity-engine/blob/master/.travis.yml.disabled#L21-L23 but then the build matrix will explode and the build time will double.
Please let me know if you think it would be good to add more test jobs for the not covered "features"/modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g closed pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g closed pull request #10958:
URL: https://github.com/apache/druid/pull/10958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g closed pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g closed pull request #10958:
URL: https://github.com/apache/druid/pull/10958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-799242742
There is no node-sass binary for Linux Aarch64 - https://github.com/sass/node-sass/releases/tag/v4.14.1
It is interesting why this didn't fail before (on the non-Graviton2 ARM64 TravisCI nodes) ?!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-817740815
@suneet-s I've created a new PR (https://github.com/apache/druid/pull/11094) but it didn't trigger Travis build too. It might be related to the fact that my personal account at TravisCI is out of free credits :-/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-804013016
On my ARM64 VM I am able to run org.apache.druid.segment.virtual.ExpressionVectorSelectorsTest even with `-Xmx384m`.
At TravisCI the tests run with -Xmx3g` - I have the feeling that there might be a memory leak from the previous tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on pull request #10958: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on pull request #10958:
URL: https://github.com/apache/druid/pull/10958#issuecomment-804008296
With the changes in this PR I am able to run the full build and tests on my Linux ARM64 VM!
But at TravisCI org.apache.druid.segment.virtual.ExpressionVectorSelectorsTest fails with OutOfMemoryError in the heap space.
According to Async Profiler the most memory allocations are caused by org.apache.druid.java.util.common.StringUtils#format(). I think this method is overused - most of the time simple String concatenation could be used. String#format() is both slow and memory intensive.
I've simplified a bit this test to not use StringUtils#format() but this is not enough - it still fails with OOME.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org