[GitHub] [hudi] rahil-c commented on a diff in pull request #5786: [HUDI-2955] Support Hadoop 3.x Hive 3.x and Spark 3.2.x default (rebase)

[GitBox <gi...@apache.org>]

rahil-c commented on code in PR #5786:
URL: https://github.com/apache/hudi/pull/5786#discussion_r917450559


##########
pom.xml:
##########
@@ -1669,15 +1762,17 @@
         <fasterxml.jackson.dataformat.yaml.version>${fasterxml.spark3.version}
         </fasterxml.jackson.dataformat.yaml.version>
         <skip.hudi-spark2.unit.tests>true</skip.hudi-spark2.unit.tests>
-        <skipITs>true</skipITs>
+        <skipITs>false</skipITs>
       </properties>
       <modules>
         <module>hudi-spark-datasource/hudi-spark3</module>
         <module>hudi-spark-datasource/hudi-spark3-common</module>
       </modules>
       <activation>
+        <activeByDefault>true</activeByDefault>
         <property>
           <name>spark3</name>
+          <value>!disabled</value>

Review Comment:
   This is still an ongoing issue and possibly might be a bug with `mvn`. For example in current hudi master branch we have this value tag in spark 2 profile https://github.com/apache/hudi/blob/master/pom.xml#L1645. If you see the comment it will add the modules for `spark2` to all profiles(even if you specified spark3). This value tag is what allows these modules to be included from my testing. 
   ```
          <module>hudi-spark-datasource/hudi-spark2</module>
           <module>hudi-spark-datasource/hudi-spark2-common</module>
         </modules>
   ```
   but the issue is without this value tag above it will not include these two modules. 
   
   For our pr we follow a similar approach but this causes an issue for spark2 if you dont also explicitly pass the scala version as well. 
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org