You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/08/05 00:40:27 UTC
[spark] branch master updated: [SPARK-36068][BUILD][TEST] No tests
in hadoop-cloud run unless hadoop-3.2 profile is activated explicitly
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0f5c3a4 [SPARK-36068][BUILD][TEST] No tests in hadoop-cloud run unless hadoop-3.2 profile is activated explicitly
0f5c3a4 is described below
commit 0f5c3a4fd642dcfcdfcf1ccfba4556acd333b764
Author: Kousuke Saruta <sa...@oss.nttdata.com>
AuthorDate: Thu Aug 5 09:39:28 2021 +0900
[SPARK-36068][BUILD][TEST] No tests in hadoop-cloud run unless hadoop-3.2 profile is activated explicitly
### What changes were proposed in this pull request?
This PR fixes an issue that no tests in `hadoop-cloud` are compiled and run unless `hadoop-3.2` profile is activated explicitly.
The root cause seems similar to SPARK-36067 (#33276) so the solution is to activate `hadoop-3.2` profile in `hadoop-cloud/pom.xml` by default.
This PR introduced an empty profile for `hadoop-2.7`. Without this, building with `hadoop-2.7` fails.
### Why are the changes needed?
`hadoop-3.2` profile should be activated by default so tests in `hadoop-cloud` also should be compiled and run without activating `hadoop-3.2` profile explicitly.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Confirmed tests in `hadoop-cloud` ran with both SBT and Maven.
```
build/sbt -Phadoop-cloud "hadoop-cloud/test"
...
[info] CommitterBindingSuite:
[info] - BindingParquetOutputCommitter binds to the inner committer (258 milliseconds)
[info] - committer protocol can be serialized and deserialized (11 milliseconds)
[info] - local filesystem instantiation (3 milliseconds)
[info] - reject dynamic partitioning (1 millisecond)
[info] Run completed in 1 second, 234 milliseconds.
[info] Total number of tests run: 4
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
build/mvn -Phadoop-cloud -pl hadoop-cloud test
...
CommitterBindingSuite:
- BindingParquetOutputCommitter binds to the inner committer
- committer protocol can be serialized and deserialized
- local filesystem instantiation
- reject dynamic partitioning
Run completed in 560 milliseconds.
Total number of tests run: 4
Suites: completed 2, aborted 0
Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
I also confirmed building with `-Phadoop-2.7` successfully finishes with both SBT and Maven.
```
build/sbt -Phadoop-cloud -Phadoop-2.7 "hadoop-cloud/Test/compile"
build/mvn -Phadoop-cloud -Phadoop-2.7 -pl hadoop-cloud testCompile
```
Closes #33277 from sarutak/fix-hadoop-3.2-cloud.
Authored-by: Kousuke Saruta <sa...@oss.nttdata.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
hadoop-cloud/pom.xml | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/hadoop-cloud/pom.xml b/hadoop-cloud/pom.xml
index 49b0b34..fe62443 100644
--- a/hadoop-cloud/pom.xml
+++ b/hadoop-cloud/pom.xml
@@ -201,13 +201,22 @@
</dependencies>
<profiles>
-
+ <!--
+ hadoop-3.2 profile is activated by default so hadoop-2.7 profile
+ also needs to be declared here for building with -Phadoop-2.7.
+ -->
+ <profile>
+ <id>hadoop-2.7</id>
+ </profile>
<!--
Hadoop 3 simplifies the classpath, and adds a new committer base class which
enables store-specific committers.
-->
<profile>
<id>hadoop-3.2</id>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
<properties>
<extra.source.dir>src/hadoop-3/main/scala</extra.source.dir>
<extra.testsource.dir>src/hadoop-3/test/scala</extra.testsource.dir>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org