You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "PhantomHunt (via GitHub)" <gi...@apache.org> on 2023/05/09 12:49:35 UTC
[GitHub] [hudi] PhantomHunt opened a new issue, #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
PhantomHunt opened a new issue, #8676:
URL: https://github.com/apache/hudi/issues/8676
**ERROR**
We tried to install HUDI CLI on our EC2 Ubuntu instance by following the steps mentioned in Apache HUDI CLI documentation. After all the installation, we ran this command - `./hudi-cli.sh` as instructed in the documentation but we are getting an error and CLI is not working.
**Expected behavior**
Hudi CLI should run.
**Environment Description**
* Hudi version : 0.13.0
* Spark version : 3.3.1
* Storage (HDFS/S3/GCS..) : S3
**Stacktrace**
```ls: cannot access '/home/ubuntu/hudi-0.12.3/hudi-cli/target/hudi-cli-*.jar': No such file or directory
Client jar location not set, please set it in conf/hudi-env.sh
ls: cannot access '/home/ubuntu/hudi-0.12.3/hudi-cli/target/lib/*': No such file or directory
Running : java -cp /etc/hadoop/conf:/etc/spark/conf::: -DSPARK_CONF_DIR=/etc/spark/conf -DHADOOP_CONF_DIR=/etc/hadoop/conf org.apache.hudi.cli.Main
Error: Could not find or load main class org.apache.hudi.cli.Main
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.cli.Main```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8676:
URL: https://github.com/apache/hudi/issues/8676#issuecomment-1542297697
@PhantomHunt As discussed we need to build the code first. Set JAVA_HOME to idk 1.8 and then try to build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8676:
URL: https://github.com/apache/hudi/issues/8676#issuecomment-1549653842
@PhantomHunt
You can export the s3a env variables as discussed on call.
similar to this you can set all fs variables.
export HOODIE_ENV_fs_DOT_s3a_DOT_credential_DOT_manager=$credentialmanager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] PhantomHunt commented on issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
Posted by "PhantomHunt (via GitHub)" <gi...@apache.org>.
PhantomHunt commented on issue #8676:
URL: https://github.com/apache/hudi/issues/8676#issuecomment-1551409076
Hi,
This issue is now resolved. Here are the steps we followed to fix this -
1. Downgraded Java from version 11 to version 8 (open JDK 1.8) as Hudi CLI supports only JDK1.8
2. Used this command to build packages as we are having spark 3.3 -` mvn -T 2C clean package -DskipTests -Dspark3.2 -Dscala-2.12`
3. Install these Client jars -
`a. wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.2.2/hadoop-aws-3.2.2.jar`
`b. wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.180/aws-java-sdk-bundle-1.12.180.jar`
4. Set these environment variables - ```
export SPARK_HOME=/usr/local/lib/python3.8/dist-packages/pyspark
export CLIENT_JAR=/home/ubuntu/hudi-release-0.13.0/hudi-cli/aws-java-sdk-bundle-1.12.180.jar:/home/ubuntu/hudi-release-0.13.0/hudi-cli/hadoop-aws-3.2.2.jar
export HOODIE_ENV_fs_DOT_s3a_DOT_impl=org.apache.hadoop.fs.s3a.S3AFileSystem
export HOODIE_ENV_fs_DOT_s3a_DOT_aws_DOT_credentials_DOT_provider=com.amazonaws.auth.InstanceProfileCredentialsProvider,com.amazonaws.auth.DefaultAWSCredentialsProviderChain
export HOODIE_ENV_fs_DOT_AbstractFileSystem_DOT_s3a_DOT_impl=org.apache.hadoop.fs.s3a.S3A
```
**Suggestion:**
1. @ad1happy2go and team, We would suggest enhancing the HUDI CLI documentation to provide more clarity on the installation process for future users. Also, there is a version mismatch issue [here](https://hudi.apache.org/docs/cli/#using-hudi-cli-in-s3)
which can cause errors.
2. Kindly provide CLI installation steps for spark 3.3 in [this](https://www.onehouse.ai/blog/getting-started-manage-your-hudi-tables-with-the-admin-hudi-cli-tool) OneHouse documentation.
3. The Environment variables we had to set for connecting CLI to S3/S3a were not there in any document. Request to add those as well
Thanks @ad1happy2go for your all help and time. Really appreciate the dedication you showed to help us resolve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] PhantomHunt closed issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
Posted by "PhantomHunt (via GitHub)" <gi...@apache.org>.
PhantomHunt closed issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
URL: https://github.com/apache/hudi/issues/8676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] PhantomHunt commented on issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
Posted by "PhantomHunt (via GitHub)" <gi...@apache.org>.
PhantomHunt commented on issue #8676:
URL: https://github.com/apache/hudi/issues/8676#issuecomment-1545600681
Thanks @ad1happy2go! I have now installed hudi cli successfully but while running it, got this error -
`hudi->create --path s3://gn-video-richmedia-nonprod-hudi-tables/nonprod_hudi_tables/assets/asset --tableName asset --tableType MERGE_ON_READ
2023-05-12 11:29:50.367 WARN 2799 --- [ main] o.a.h.u.NativeCodeLoader : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-05-12 11:29:50.766 WARN 2799 --- [ main] o.a.h.f.FileSystem : Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem could not be instantiated
2023-05-12 11:29:50.766 WARN 2799 --- [ main] o.a.h.f.FileSystem : java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkState(ZLjava/lang/String;J)V
2023-05-12 11:29:51.097 INFO 2799 --- [ main] o.a.h.c.t.HoodieTableMetaClient : Loading HoodieTableMetaClient from s3://gn-video-richmedia-nonprod-hudi-tables/nonprod_hudi_tables/assets/asset
Failed to get instance of org.apache.hadoop.fs.FileSystem
Details of the error have been omitted. You can use the stacktrace command to print the full stacktrace.
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] PhantomHunt commented on issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
Posted by "PhantomHunt (via GitHub)" <gi...@apache.org>.
PhantomHunt commented on issue #8676:
URL: https://github.com/apache/hudi/issues/8676#issuecomment-1541368018
Any updates? If not do we have any alternative method for checking the configuration (Compaction and cleaner schedule) of an existing HUDI Mor table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] PhantomHunt commented on issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
Posted by "PhantomHunt (via GitHub)" <gi...@apache.org>.
PhantomHunt commented on issue #8676:
URL: https://github.com/apache/hudi/issues/8676#issuecomment-1549814739
Since the original issue is resolved, closing the ticket. Will open another ticket for the current issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] PhantomHunt commented on issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
Posted by "PhantomHunt (via GitHub)" <gi...@apache.org>.
PhantomHunt commented on issue #8676:
URL: https://github.com/apache/hudi/issues/8676#issuecomment-1551348484
> export HOODIE_ENV_fs_DOT_s3a_DOT_credential_DOT_manager=$credentialmanager
We tried this -
```
export HOODIE_ENV_fs_DOT_s3a_DOT_impl=org_DOT_apache_DOT_hadoop_DOT_fs_DOT_s3a_DOT_S3AFileSystem
export HOODIE_ENV_fs_DOT_s3a_DOT_aws_DOT_credentials_DOT_provider=com_DOT_amazonaws_DOT_auth_DOT_InstanceProfileCredentialsProvider,com_DOT_amazonaws_DOT_auth_DOT_DefaultAWSCredentialsProviderChain
export HOODIE_ENV_fs_DOT_AbstractFileSystem_DOT_s3a_DOT_impl=org_DOT_apache_DOT_hadoop_DOT_fs_DOT_s3a_DOT_S3A
```
But getting error -
` java.lang.ClassNotFoundException: Class org_DOT_apache_DOT_hadoop_DOT_fs_DOT_s3a_DOT_S3AFileSystem not found`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] PhantomHunt closed issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
Posted by "PhantomHunt (via GitHub)" <gi...@apache.org>.
PhantomHunt closed issue #8676: [SUPPORT] Cannot install Hoodie CLI on EC2
URL: https://github.com/apache/hudi/issues/8676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org