You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/14 18:28:35 UTC
(datafusion-comet) 01/01: test: tag the Azure ABFS suites with AzureTest (#6942)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6942-e4babc46bb608510707c281c72c60e8e1a6e7d9b
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit b513921290f95ffc7021a4a9a8ae92ea507141b4
Author: Oleks V <co...@users.noreply.github.com>
AuthorDate: Wed Oct 14 18:28:09 2026 +0000
test: tag the Azure ABFS suites with AzureTest (#6942)
* test: tag the Azure ABFS suites with AzureTest
AbfsAuthParitySuite, AbfsAuthResolverSuite and AbfsScanFallbackSuite resolve
ABFS authentication through the hadoop-azure on the test classpath, so they
pass only when that hadoop-azure matches the Hadoop the build runs on. A
build against a different Hadoop line, for example a vendor Hadoop with no
matching hadoop-azure pin, had no way to skip them: the root pom hard-codes
tagsToExclude, so -DtagsToExclude on the command line was ignored.
Add an org.apache.comet.tags.AzureTest tag, annotate the three suites with
it, and append a tagsToExclude property (empty by default) to the
scalatest-maven-plugin exclusions so -DtagsToExclude=org.apache.comet.tags.AzureTest
skips them. CI keeps running them, since every Spark profile pins a matching
hadoop-azure.
* test: tag only the AzureBlobFileSystem cross-check with AzureTest
On a mismatched classpath only the FileSystem.newInstance cross-check in
AbfsAuthParitySuite fails. AzureBlobFileSystem links against Hadoop classes
an older Hadoop lacks, while the resolver reaches nothing newer than
AbfsConfiguration, so the resolver cases, AbfsAuthResolverSuite and
AbfsScanFallbackSuite still pass there.
Tag just that test, with a ScalaTest Tag of the same name, so a build that
passes -DtagsToExclude=org.apache.comet.tags.AzureTest keeps running the
resolver cases.
---
docs/source/contributor-guide/development.md | 6 +++++
pom.xml | 5 +++-
.../comet/objectstore/AbfsAuthParitySuite.scala | 5 +++-
.../scala/org/apache/comet/tags/AzureTest.scala | 30 ++++++++++++++++++++++
4 files changed, 44 insertions(+), 2 deletions(-)
diff --git a/docs/source/contributor-guide/development.md b/docs/source/contributor-guide/development.md
index ea4a65e308..eb12723bdd 100644
--- a/docs/source/contributor-guide/development.md
+++ b/docs/source/contributor-guide/development.md
@@ -606,6 +606,12 @@ still assert while recording, so a recorded file cannot contradict what hadoop-a
file is written only when every case ran. Review the diff of the regenerated files before
committing them: each line that changes is a behaviour change on that hadoop-azure release.
+The FileSystem cross-check in `AbfsAuthParitySuite` is tagged `org.apache.comet.tags.AzureTest`.
+It initializes a real `AzureBlobFileSystem`, so it passes only when the test-scoped hadoop-azure
+matches the Hadoop the build runs on. A build against another Hadoop, where no matching
+hadoop-azure is pinned, can skip it with `-DtagsToExclude=org.apache.comet.tags.AzureTest` and
+still run the resolver cases.
+
## Benchmark
There's a `make` command to run micro benchmarks in the repo. For
diff --git a/pom.xml b/pom.xml
index 2a737c9c20..4bd65dc53e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -123,6 +123,9 @@ under the License.
reflection only). Each Spark profile pins it to its Hadoop line; override with
-Dhadoop-azure.version to test another release. -->
<hadoop-azure.version>3.4.2</hadoop-azure.version>
+ <!-- Extra scalatest tags to exclude, comma separated, for example
+ -DtagsToExclude=org.apache.comet.tags.AzureTest. -->
+ <tagsToExclude></tagsToExclude>
<jni.dir>${project.basedir}/../native/target/debug</jni.dir>
<platform>darwin</platform>
<arch>x86_64</arch>
@@ -1124,7 +1127,7 @@ under the License.
<filereports>SparkTestSuite.txt</filereports>
<stdout>D</stdout>
<stderr/>
- <tagsToExclude>org.apache.comet.IntegrationTestSuite</tagsToExclude>
+ <tagsToExclude>org.apache.comet.IntegrationTestSuite,${tagsToExclude}</tagsToExclude>
<systemProperties>
<!-- emit test logs to target/unit-tests.log -->
<log4j.configurationFile>file:src/test/resources/log4j2.properties</log4j.configurationFile>
diff --git a/spark/src/test/scala/org/apache/comet/objectstore/AbfsAuthParitySuite.scala b/spark/src/test/scala/org/apache/comet/objectstore/AbfsAuthParitySuite.scala
index 3d95864130..eeca4d0fb9 100644
--- a/spark/src/test/scala/org/apache/comet/objectstore/AbfsAuthParitySuite.scala
+++ b/spark/src/test/scala/org/apache/comet/objectstore/AbfsAuthParitySuite.scala
@@ -43,6 +43,7 @@ import com.fasterxml.jackson.databind.{JsonNode, ObjectMapper}
import org.apache.comet.objectstore.AbfsAuthParityCases._
import org.apache.comet.objectstore.AbfsAuthResolver._
import org.apache.comet.objectstore.AbfsReflection.Handles
+import org.apache.comet.tags.AzureTest
/**
* Pins the resolver's behaviour per hadoop-azure version and cross-checks it against the real
@@ -184,7 +185,9 @@ class AbfsAuthParitySuite extends AnyFunSuite with Matchers with BeforeAndAfterA
}
}
- test("FileSystem.newInstance agrees with the resolver on every case") {
+ // Tagged AzureTest: AzureBlobFileSystem links against Hadoop classes that only a matching
+ // Hadoop has, while the resolver cases above need nothing newer than AbfsConfiguration.
+ test("FileSystem.newInstance agrees with the resolver on every case", AzureTest) {
val timings = mutable.ArrayBuffer[(String, Long)]()
val disagreements = cases.flatMap { c =>
val outcome = resolve(hadoopConf(c), c.uri)
diff --git a/spark/src/test/scala/org/apache/comet/tags/AzureTest.scala b/spark/src/test/scala/org/apache/comet/tags/AzureTest.scala
new file mode 100644
index 0000000000..aacf5b87e4
--- /dev/null
+++ b/spark/src/test/scala/org/apache/comet/tags/AzureTest.scala
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.tags
+
+import org.scalatest.Tag
+
+/**
+ * Marks tests that initialize a real `AzureBlobFileSystem` from the hadoop-azure on the test
+ * classpath. They pass only when that hadoop-azure matches the Hadoop the build runs on, so a
+ * build against another Hadoop can skip them with
+ * `-DtagsToExclude=org.apache.comet.tags.AzureTest`.
+ */
+object AzureTest extends Tag("org.apache.comet.tags.AzureTest")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org