You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/04/13 06:56:05 UTC
[GitHub] [druid] martin-g opened a new pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
martin-g opened a new pull request #11109:
URL: https://github.com/apache/druid/pull/11109
Documentation:
- https://blog.travis-ci.com/2020-09-11-arm-on-aws
- https://aws.amazon.com/blogs/opensource/getting-started-with-travis-ci-com-on-aws-graviton2/
Fixes the problem described at #10953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #11109:
URL: https://github.com/apache/druid/pull/11109#discussion_r614584818
##########
File path: processing/src/test/java/org/apache/druid/segment/virtual/ExpressionVectorSelectorsTest.java
##########
@@ -96,25 +96,31 @@
private static QueryableIndex INDEX;
private static Closer CLOSER;
+ private static final String CPU_ARCH = System.getProperty("os.arch");
+
@BeforeClass
public static void setupClass()
{
CLOSER = Closer.create();
- final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("expression-testbench");
+ // Do not run the tests on ARM64.
+ // SegmentGenerator#generate() fails with OutOfMemoryError on TravisCI ARM64
Review comment:
maybe it would be worth trying with a smaller segment size (`ROWS_PER_SEGMENT`) instead of skipping the test entirely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on a change in pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on a change in pull request #11109:
URL: https://github.com/apache/druid/pull/11109#discussion_r614740252
##########
File path: .travis.yml
##########
@@ -303,6 +303,17 @@ jobs:
after_success:
- (cd web-console && travis_retry npm run codecov) # retry in case of network error
+ - name: "Build and test on ARM64 CPU architecture"
+ stage: Tests - phase 2
+ arch: arm64-graviton2
+ dist: focal
+ virt: vm
+ group: edge
+ jdk: openjdk11
+ env:
+ - MAVEN_PROJECTS='core,indexing-hadoop,indexing-service,processing,server,services'
Review comment:
I've split the ARM64 job into two jobs with two different sets of modules to test.
This is needed because otherwise TravisCI will kill the job if it takes more than 50mins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on a change in pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on a change in pull request #11109:
URL: https://github.com/apache/druid/pull/11109#discussion_r615670338
##########
File path: processing/src/test/java/org/apache/druid/segment/virtual/ExpressionVectorSelectorsTest.java
##########
@@ -96,25 +96,31 @@
private static QueryableIndex INDEX;
private static Closer CLOSER;
+ private static final String CPU_ARCH = System.getProperty("os.arch");
+
@BeforeClass
public static void setupClass()
{
CLOSER = Closer.create();
- final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("expression-testbench");
+ // Do not run the tests on ARM64.
+ // SegmentGenerator#generate() fails with OutOfMemoryError on TravisCI ARM64
Review comment:
@clintropolis With this change the unit test passed: https://travis-ci.com/github/apache/druid/builds/223430180
Some other non-related TravisCI jobs failed for reasons not related to this PR, but the new ARM64 ones passed successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on a change in pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on a change in pull request #11109:
URL: https://github.com/apache/druid/pull/11109#discussion_r614741500
##########
File path: processing/src/test/java/org/apache/druid/segment/virtual/ExpressionVectorSelectorsTest.java
##########
@@ -96,25 +96,31 @@
private static QueryableIndex INDEX;
private static Closer CLOSER;
+ private static final String CPU_ARCH = System.getProperty("os.arch");
+
@BeforeClass
public static void setupClass()
{
CLOSER = Closer.create();
- final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("expression-testbench");
+ // Do not run the tests on ARM64.
+ // SegmentGenerator#generate() fails with OutOfMemoryError on TravisCI ARM64
Review comment:
Tried with 2 smaller ROWS_PER_SEGMENT with https://github.com/apache/druid/pull/11109/commits/48ab08c0c8ea1c76f2d5a706ada563ba897f9720. Let's see!
##########
File path: processing/src/test/java/org/apache/druid/segment/virtual/ExpressionVectorSelectorsTest.java
##########
@@ -96,25 +96,31 @@
private static QueryableIndex INDEX;
private static Closer CLOSER;
+ private static final String CPU_ARCH = System.getProperty("os.arch");
+
@BeforeClass
public static void setupClass()
{
CLOSER = Closer.create();
- final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("expression-testbench");
+ // Do not run the tests on ARM64.
+ // SegmentGenerator#generate() fails with OutOfMemoryError on TravisCI ARM64
Review comment:
Tried with twice smaller ROWS_PER_SEGMENT with https://github.com/apache/druid/pull/11109/commits/48ab08c0c8ea1c76f2d5a706ada563ba897f9720. Let's see!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on a change in pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on a change in pull request #11109:
URL: https://github.com/apache/druid/pull/11109#discussion_r616384978
##########
File path: processing/src/test/java/org/apache/druid/segment/virtual/ExpressionVectorSelectorsTest.java
##########
@@ -96,25 +96,31 @@
private static QueryableIndex INDEX;
private static Closer CLOSER;
+ private static final String CPU_ARCH = System.getProperty("os.arch");
+
@BeforeClass
public static void setupClass()
{
CLOSER = Closer.create();
- final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("expression-testbench");
+ // Do not run the tests on ARM64.
+ // SegmentGenerator#generate() fails with OutOfMemoryError on TravisCI ARM64
Review comment:
Done!
Also rebased to latest `master`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on a change in pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on a change in pull request #11109:
URL: https://github.com/apache/druid/pull/11109#discussion_r614739435
##########
File path: .travis.yml
##########
@@ -303,6 +303,17 @@ jobs:
after_success:
- (cd web-console && travis_retry npm run codecov) # retry in case of network error
+ - name: "Build and test on ARM64 CPU architecture"
+ stage: Tests - phase 2
+ arch: arm64-graviton2
+ dist: focal
+ virt: vm
+ group: edge
+ jdk: openjdk11
+ env:
+ - MAVEN_PROJECTS='core,indexing-hadoop,indexing-service,processing,server,services'
Review comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #11109:
URL: https://github.com/apache/druid/pull/11109#discussion_r616264454
##########
File path: processing/src/test/java/org/apache/druid/segment/virtual/ExpressionVectorSelectorsTest.java
##########
@@ -96,25 +96,31 @@
private static QueryableIndex INDEX;
private static Closer CLOSER;
+ private static final String CPU_ARCH = System.getProperty("os.arch");
+
@BeforeClass
public static void setupClass()
{
CLOSER = Closer.create();
- final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("expression-testbench");
+ // Do not run the tests on ARM64.
+ // SegmentGenerator#generate() fails with OutOfMemoryError on TravisCI ARM64
Review comment:
:+1:, in that case please remove this commented section and the no longer used `CPU_ARCH`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on a change in pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on a change in pull request #11109:
URL: https://github.com/apache/druid/pull/11109#discussion_r614628083
##########
File path: processing/src/test/java/org/apache/druid/segment/virtual/ExpressionVectorSelectorsTest.java
##########
@@ -96,25 +96,31 @@
private static QueryableIndex INDEX;
private static Closer CLOSER;
+ private static final String CPU_ARCH = System.getProperty("os.arch");
+
@BeforeClass
public static void setupClass()
{
CLOSER = Closer.create();
- final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("expression-testbench");
+ // Do not run the tests on ARM64.
+ // SegmentGenerator#generate() fails with OutOfMemoryError on TravisCI ARM64
Review comment:
I am not 100% certain but I think there is a memory leak in an earlier test(s).
ExpressionVectorSelectorsTest does not fail on my Ubuntu 20.04 ARM64 VM. I have tried even with much lower max memory - it passes with -Xmx384m. The test suite at TravisCI runs with -Xmx3000m.
Locally with -Xmx256m it fails with OOME Too much garbage collections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #11109:
URL: https://github.com/apache/druid/pull/11109#discussion_r614584818
##########
File path: processing/src/test/java/org/apache/druid/segment/virtual/ExpressionVectorSelectorsTest.java
##########
@@ -96,25 +96,31 @@
private static QueryableIndex INDEX;
private static Closer CLOSER;
+ private static final String CPU_ARCH = System.getProperty("os.arch");
+
@BeforeClass
public static void setupClass()
{
CLOSER = Closer.create();
- final GeneratorSchemaInfo schemaInfo = GeneratorBasicSchemas.SCHEMA_MAP.get("expression-testbench");
+ // Do not run the tests on ARM64.
+ // SegmentGenerator#generate() fails with OutOfMemoryError on TravisCI ARM64
Review comment:
maybe it would be trying with a smaller segment size (`ROWS_PER_SEGMENT`) instead of skipping the test entirely?
##########
File path: .travis.yml
##########
@@ -303,6 +303,17 @@ jobs:
after_success:
- (cd web-console && travis_retry npm run codecov) # retry in case of network error
+ - name: "Build and test on ARM64 CPU architecture"
+ stage: Tests - phase 2
+ arch: arm64-graviton2
+ dist: focal
+ virt: vm
+ group: edge
+ jdk: openjdk11
+ env:
+ - MAVEN_PROJECTS='core,indexing-hadoop,indexing-service,processing,server,services'
Review comment:
should probably consider adding `sql` to this set too so we at least have coverage of most core Druid things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] martin-g commented on a change in pull request #11109: Issue #10953: ARM64 - Build and test on AWS Graviton2 node
Posted by GitBox <gi...@apache.org>.
martin-g commented on a change in pull request #11109:
URL: https://github.com/apache/druid/pull/11109#discussion_r614628266
##########
File path: .travis.yml
##########
@@ -303,6 +303,17 @@ jobs:
after_success:
- (cd web-console && travis_retry npm run codecov) # retry in case of network error
+ - name: "Build and test on ARM64 CPU architecture"
+ stage: Tests - phase 2
+ arch: arm64-graviton2
+ dist: focal
+ virt: vm
+ group: edge
+ jdk: openjdk11
+ env:
+ - MAVEN_PROJECTS='core,indexing-hadoop,indexing-service,processing,server,services'
Review comment:
OK, I will add it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org