You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/01/06 17:56:55 UTC
[GitHub] [arrow] arw2019 opened a new pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
arw2019 opened a new pull request #9115:
URL: https://github.com/apache/arrow/pull/9115
cc @BryanCutler
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] BryanCutler commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
BryanCutler commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-761270391
@arw2019 sorry, I probably should have been more clear in the JIRA description, but there are more things needed to properly test against a Spark maintenance branch like branch-3.0. In the interest of time, I made another PR https://github.com/apache/arrow/pull/9210 with what I think needs to happen. I'll try out the tests there to hopefully get them passing soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-756129382
@github-actions crossbow submit test-conda-python-3.7-spark-branch-3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] BryanCutler commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
BryanCutler commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-760410483
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-758096975
@BryanCutler any idea about that failure?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-760917973
We definitely should test against spark releases.
I executed the build against 3.0.1 but it fails with:
```
- max records in batch conf *** FAILED ***
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 21.0 failed 1 times, most recent failure: Lost task 0.0 in stage 21.0 (TID 25, 5fc0f8cfe8d2, executor driver): java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.sql.util.ArrowUtils$
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-766852482
Closing in favour of #9210
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] BryanCutler commented on a change in pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
BryanCutler commented on a change in pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#discussion_r553045953
##########
File path: ci/docker/conda-python-spark.dockerfile
##########
@@ -20,21 +20,15 @@ ARG arch=amd64
ARG python=3.6
FROM ${repo}:${arch}-conda-python-${python}
-ARG jdk=8
-ARG maven=3.5
+ARG spark=3.0.1
# The Spark tests currently break with pandas >= 1.0
Review comment:
This shouldn't be the case any more
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 closed pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
arw2019 closed pull request #9115:
URL: https://github.com/apache/arrow/pull/9115
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-755784738
@github-actions crossbow submit test-conda-python-3.7-spark-branch-3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-760700442
@github-actions crossbow submit test-conda-python-3.7-spark-branch-3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs edited a comment on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-760917973
We definitely should test against spark releases.
I executed the build against 3.0.1 but it fails with:
```
- max records in batch conf *** FAILED ***
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 21.0 failed 1 times, most recent failure: Lost task 0.0 in stage 21.0 (TID 25, 5fc0f8cfe8d2, executor driver): java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.sql.util.ArrowUtils$
```
cc @BryanCutler
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-756130615
Revision: 2a94f93e85e159ac41a4176e7cbd932f1b6e6005
Submitted crossbow builds: [ursa-labs/crossbow @ actions-831](https://github.com/ursa-labs/crossbow/branches/all?query=actions-831)
|Task|Status|
|----|------|
|test-conda-python-3.7-spark-branch-3.0|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-831-github-test-conda-python-3.7-spark-branch-3.0)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-756215627
@github-actions crossbow submit test-conda-python-3.7-spark-branch-3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-756216668
Revision: 2a94f93e85e159ac41a4176e7cbd932f1b6e6005
Submitted crossbow builds: [ursa-labs/crossbow @ actions-833](https://github.com/ursa-labs/crossbow/branches/all?query=actions-833)
|Task|Status|
|----|------|
|test-conda-python-3.7-spark-branch-3.0|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-833-github-test-conda-python-3.7-spark-branch-3.0)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-755871620
> My intention was to test pyarrow against spark branch-3.0 in addition to the current tests that also include arrow Java for the master branch. We still want to maintain that to be sure spark will be able to build with the latest arrow.
Ok! In that case do we want to add another Docker build to the suite with spark branch-3.0 + pyarrow?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-760500200
Revision: 2a94f93e85e159ac41a4176e7cbd932f1b6e6005
Submitted crossbow builds: [ursa-labs/crossbow @ actions-893](https://github.com/ursa-labs/crossbow/branches/all?query=actions-893)
|Task|Status|
|----|------|
|test-conda-python-3.7-spark-branch-3.0|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-893-github-test-conda-python-3.7-spark-branch-3.0)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-756516217
This build is currently failing because of a problem with the Java implementation:
```
[INFO] ------------------------------------------------------------------------
Error: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project arrow-plasma: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test failed: Plugin org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-surefire-plugin:jar:3.0.0-M3 -> org.apache.maven.surefire:maven-surefire-common:jar:3.0.0-M3: Failed to read artifact descriptor for org.apache.maven.surefire:maven-surefire-common:jar:3.0.0-M3: Could not transfer artifact org.apache.maven.surefire:maven-surefire-common:pom:3.0.0-M3 from/to central (https://repo.maven.apache.org/maven2): Connection reset -> [Help 1]
Error:
Error: To see the full stack trace of the errors, re-run Maven with the -e switch.
Error: Re-run Maven using the -X switch to enable full debug logging.
Error:
Error: For more information about the errors and possible solutions, please read the following articles:
Error: [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
Error:
Error: After correcting the problems, you can resume the build with the command
Error: mvn <goals> -rf :arrow-plasma
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs edited a comment on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-761000046
```
Caused by: java.lang.RuntimeException: No DefaultAllocationManager found on classpath. Can't allocate Arrow buffers. Please consider adding arrow-memory-netty or arrow-memory-unsafe as a dependency.
```
This change was introduced by https://github.com/apache/arrow/commit/2092e18752a9c0494799493b12eb1830052217a2
Which is part of the previous release, so I assume a testing issue (or something which should be modified on spark's side).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on a change in pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
arw2019 commented on a change in pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#discussion_r553099694
##########
File path: ci/docker/conda-python-spark.dockerfile
##########
@@ -20,21 +20,15 @@ ARG arch=amd64
ARG python=3.6
FROM ${repo}:${arch}-conda-python-${python}
-ARG jdk=8
-ARG maven=3.5
+ARG spark=3.0.1
# The Spark tests currently break with pandas >= 1.0
Review comment:
Ok! I reverted the changes here and bumped pandas to the latest release (1.2) and hopefully the build will go green
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-755672614
https://issues.apache.org/jira/browse/ARROW-10457
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-755872744
@github-actions crossbow submit test-conda-python-3.7-spark-branch-3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-760700763
Revision: 2a94f93e85e159ac41a4176e7cbd932f1b6e6005
Submitted crossbow builds: [ursacomputing/crossbow @ actions-10](https://github.com/ursacomputing/crossbow/branches/all?query=actions-10)
|Task|Status|
|----|------|
|test-conda-python-3.7-spark-branch-3.0|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-10-github-test-conda-python-3.7-spark-branch-3.0)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-761000046
```
Caused by: java.lang.RuntimeException: No DefaultAllocationManager found on classpath. Can't allocate Arrow buffers. Please consider adding arrow-memory-netty or arrow-memory-unsafe as a dependency.
```
This change was introduced by https://github.com/apache/arrow/commit/2092e18752a9c0494799493b12eb1830052217a2
Which is part of the previous release, so I assume a testing issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-755461385
@github-actions crossbow submit test-conda-python-3.7-spark-branch-3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9115:
URL: https://github.com/apache/arrow/pull/9115#issuecomment-766852482
Closing in favour of #9210
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 closed pull request #9115: ARROW-10457: [CI] Fix Spark branch-3.0 integration tests
Posted by GitBox <gi...@apache.org>.
arw2019 closed pull request #9115:
URL: https://github.com/apache/arrow/pull/9115
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org