You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prabodh1194 (via GitHub)" <gi...@apache.org> on 2024/07/13 17:21:33 UTC

[PR] fix the target location for auxlib download in hudi CLI [hudi]

prabodh1194 opened a new pull request, #11628:
URL: https://github.com/apache/hudi/pull/11628

   ### Change Logs
   
   Using Hudi CLI to access tables on S3 has some limitations as the relevant `hadoop` jars are not defined in the path by default. I have updated the CLI utility to facilitate adding the hadoop s3 jars as well. For compatibility purpose, I have made this facility behind a flag called `IS_S3_ENABLED` which can be set to `true`. Enabling this flag, will add the hadoop jars to the `auxlib` as well.
   
   ### Impact
   
   NA
   
   ### Risk level (write none, low medium or high below)
   
   none
   
   ### Documentation Update
   
   This page can be updated to highlight that the flag can be set to access `s3a` buckets path:
   https://hudi.apache.org/docs/next/cli/#using-hudi-cli
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "prabodh1194 (via GitHub)" <gi...@apache.org>.
prabodh1194 commented on code in PR #11628:
URL: https://github.com/apache/hudi/pull/11628#discussion_r1678187890


##########
packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh:
##########
@@ -38,9 +38,15 @@ HUDI_CONF_DIR="${DIR}"/conf
 HUDI_AUX_LIB="${DIR}"/auxlib
 
 if [ ! -d $HUDI_AUX_LIB ]; then
-  echo "Downloading necessary auxiliary jars for Hudi CLI"
-  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P auxlib
-  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P auxlib
+  echo "Downloading necessary auxiliary jars for Hudi CLI to $HUDI_AUX_LIB"
+  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+
+  if [ "$IS_S3_ENABLED" = "true" ]; then
+    echo "Adding hadoop-aws jars to auxlib"
+    wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar -P $HUDI_AUX_LIB
+    wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar -P $HUDI_AUX_LIB
+  fi

Review Comment:
   Agreed @CTTY . I tried with the `hudi-aws` bundle before creating the above fix and that actually didn't work. `s3a` paths require `org.apache.hadoop.fs.s3a.*` package which is defined in the `hadoop-aws` module. `hudi-aws` modules only defines dependency on `hadoop-common`.
   Hence, an explicit dependency is required in the `java` classpath.
   
   Lemme know if I should explore this in some other manner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "prabodh1194 (via GitHub)" <gi...@apache.org>.
prabodh1194 commented on PR #11628:
URL: https://github.com/apache/hudi/pull/11628#issuecomment-2254342326

   @yihua , I'll make updates later today IST 😃. Thanks for following up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11628:
URL: https://github.com/apache/hudi/pull/11628#issuecomment-2227047795

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d57782938c183d8d9ba3039e4be32c3a284fc89e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24846",
       "triggerID" : "d57782938c183d8d9ba3039e4be32c3a284fc89e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d57782938c183d8d9ba3039e4be32c3a284fc89e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24846) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #11628:
URL: https://github.com/apache/hudi/pull/11628#discussion_r1680160012


##########
packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh:
##########
@@ -38,9 +38,15 @@ HUDI_CONF_DIR="${DIR}"/conf
 HUDI_AUX_LIB="${DIR}"/auxlib
 
 if [ ! -d $HUDI_AUX_LIB ]; then
-  echo "Downloading necessary auxiliary jars for Hudi CLI"
-  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P auxlib
-  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P auxlib
+  echo "Downloading necessary auxiliary jars for Hudi CLI to $HUDI_AUX_LIB"
+  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+
+  if [ "$IS_S3_ENABLED" = "true" ]; then
+    echo "Adding hadoop-aws jars to auxlib"
+    wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar -P $HUDI_AUX_LIB
+    wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar -P $HUDI_AUX_LIB
+  fi

Review Comment:
   @prabodh1194 : hadoop-aws version needs to align withe HADOOP version installed on the the site where the cli is run. Also, aws-java-sdk-bundle's version needs to align with hadoop-aws version. So, we cannot hard-code the package version here. I would suggest to use an ENV variable HADOOP_VERSION and then instead of using wget, use mvn dependency:get of the specific hadoop-aws version which would also transitively download aws-java-sdk-bundle and get the jar file paths to use.
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #11628:
URL: https://github.com/apache/hudi/pull/11628#discussion_r1678168918


##########
packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh:
##########
@@ -38,9 +38,15 @@ HUDI_CONF_DIR="${DIR}"/conf
 HUDI_AUX_LIB="${DIR}"/auxlib
 
 if [ ! -d $HUDI_AUX_LIB ]; then
-  echo "Downloading necessary auxiliary jars for Hudi CLI"
-  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P auxlib
-  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P auxlib
+  echo "Downloading necessary auxiliary jars for Hudi CLI to $HUDI_AUX_LIB"
+  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+
+  if [ "$IS_S3_ENABLED" = "true" ]; then
+    echo "Adding hadoop-aws jars to auxlib"
+    wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar -P $HUDI_AUX_LIB
+    wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar -P $HUDI_AUX_LIB
+  fi

Review Comment:
   I'm not sure if this makes sense as Hudi already has `hudi-aws` bundle that has those dependencies



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11628:
URL: https://github.com/apache/hudi/pull/11628#issuecomment-2227015741

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d57782938c183d8d9ba3039e4be32c3a284fc89e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d57782938c183d8d9ba3039e4be32c3a284fc89e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d57782938c183d8d9ba3039e4be32c3a284fc89e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "prabodh1194 (via GitHub)" <gi...@apache.org>.
prabodh1194 commented on code in PR #11628:
URL: https://github.com/apache/hudi/pull/11628#discussion_r1678187890


##########
packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh:
##########
@@ -38,9 +38,15 @@ HUDI_CONF_DIR="${DIR}"/conf
 HUDI_AUX_LIB="${DIR}"/auxlib
 
 if [ ! -d $HUDI_AUX_LIB ]; then
-  echo "Downloading necessary auxiliary jars for Hudi CLI"
-  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P auxlib
-  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P auxlib
+  echo "Downloading necessary auxiliary jars for Hudi CLI to $HUDI_AUX_LIB"
+  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+
+  if [ "$IS_S3_ENABLED" = "true" ]; then
+    echo "Adding hadoop-aws jars to auxlib"
+    wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar -P $HUDI_AUX_LIB
+    wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar -P $HUDI_AUX_LIB
+  fi

Review Comment:
   Agreed @CTTY . I tried with the `hudi-aws` bundle before creating the above fix and that actually didn't work. `s3a` paths required `org.apache.hadoop.fs.s3a.*` package which is defined in the `hadoop-aws` module. `hudi-aws` modules only defines dependency on `hadoop-common`.
   Hence, an explicit dependency is required in the `java` classpath.
   
   Lemme know if I should explore this in some other manner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "prabodh1194 (via GitHub)" <gi...@apache.org>.
prabodh1194 commented on code in PR #11628:
URL: https://github.com/apache/hudi/pull/11628#discussion_r1680301602


##########
packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh:
##########
@@ -38,9 +38,15 @@ HUDI_CONF_DIR="${DIR}"/conf
 HUDI_AUX_LIB="${DIR}"/auxlib
 
 if [ ! -d $HUDI_AUX_LIB ]; then
-  echo "Downloading necessary auxiliary jars for Hudi CLI"
-  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P auxlib
-  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P auxlib
+  echo "Downloading necessary auxiliary jars for Hudi CLI to $HUDI_AUX_LIB"
+  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+
+  if [ "$IS_S3_ENABLED" = "true" ]; then
+    echo "Adding hadoop-aws jars to auxlib"
+    wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar -P $HUDI_AUX_LIB
+    wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar -P $HUDI_AUX_LIB
+  fi

Review Comment:
   @bvaradar  ACK.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "prabodh1194 (via GitHub)" <gi...@apache.org>.
prabodh1194 commented on PR #11628:
URL: https://github.com/apache/hudi/pull/11628#issuecomment-2333304944

   this is not much needed. will close this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11628:
URL: https://github.com/apache/hudi/pull/11628#issuecomment-2227033491

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d57782938c183d8d9ba3039e4be32c3a284fc89e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24846",
       "triggerID" : "d57782938c183d8d9ba3039e4be32c3a284fc89e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d57782938c183d8d9ba3039e4be32c3a284fc89e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24846) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "prabodh1194 (via GitHub)" <gi...@apache.org>.
prabodh1194 closed pull request #11628: [MINOR] fix the target location for auxlib download in hudi CLI
URL: https://github.com/apache/hudi/pull/11628


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix the target location for auxlib download in hudi CLI [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11628:
URL: https://github.com/apache/hudi/pull/11628#discussion_r1693873977


##########
packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh:
##########
@@ -38,9 +38,15 @@ HUDI_CONF_DIR="${DIR}"/conf
 HUDI_AUX_LIB="${DIR}"/auxlib
 
 if [ ! -d $HUDI_AUX_LIB ]; then
-  echo "Downloading necessary auxiliary jars for Hudi CLI"
-  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P auxlib
-  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P auxlib
+  echo "Downloading necessary auxiliary jars for Hudi CLI to $HUDI_AUX_LIB"
+  wget https://repo1.maven.org/maven2/org/glassfish/jakarta.el/$JAKARTA_EL_VERSION/jakarta.el-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+  wget https://repo1.maven.org/maven2/jakarta/el/jakarta.el-api/$JAKARTA_EL_VERSION/jakarta.el-api-$JAKARTA_EL_VERSION.jar -P $HUDI_AUX_LIB
+
+  if [ "$IS_S3_ENABLED" = "true" ]; then
+    echo "Adding hadoop-aws jars to auxlib"
+    wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.262/aws-java-sdk-bundle-1.12.262.jar -P $HUDI_AUX_LIB
+    wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar -P $HUDI_AUX_LIB
+  fi

Review Comment:
   @prabodh1194 could you make the change according to @bvaradar 's suggestion?  Also give an example in the script on how the ENV variables can be set and update the website https://hudi.apache.org/docs/cli.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org