You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "peter-mccabe (via GitHub)" <gi...@apache.org> on 2023/03/09 15:04:41 UTC

[GitHub] [hudi] peter-mccabe opened a new issue, #8144: [SUPPORT]Unable to connect to an s3 hudi table

peter-mccabe opened a new issue, #8144:
URL: https://github.com/apache/hudi/issues/8144

   I am unable to connect to an s3 hudi table using the hudi client. I keep getting an error:
   
   Loading HoodieTableMetaClient from s3://test-datalake/datasets//test_table
   Failed to get instance of org.apache.hadoop.fs.FileSystem
   Details of the error have been omitted. You can use the stacktrace command to print the full stacktrace.
   
   I can't see any guidance on how to set up hudi to connect to s3 hudi tables locally and i am completely stuck with this. 
   
   I have tried building locally with:
   
   >  # Checkout code and build
   > git clone https://github.com/apache/hudi.git && cd hudi
   > mvn clean package -DskipTests
   
   I then navigate to the hudi client directory and can start up the client and try to connect to my s3 dataset and get this error. I have also tried to build hudi using the docker quickstart guide and exec to the container:
   
   docker exec -it adhoc-2 /bin/bash
   
   navigate to the hudi client folder:
   
   cd /var/hoodie/ws/hudi-cli
   
   I then try and connect there and i still get the same issue. 
   
   
   
   **Expected behavior**
   
   I expect to connect to the s3 table and view commits so that i can clean/repair failed commits to the hudie table. 
   **Environment Description**
   Mac
   
   * Hudi version :
   0.13.0
   
   * Spark version :
   3.3.2
   
   * Hive version :
   
   * Hadoop version :
   3.3.4
   * Storage (HDFS/S3/GCS..) :
   s3
   * Running on Docker? (yes/no) :
   
   
   **Additional context**
   I have set my hadoop files as follows:
   core-site.xml file looks like
   ```
   <configuration>
     <property>
       <name>fs.defaultFS</name>
       <value>hdfs://localhost:9000</value>
     </property>
   </configuration>
   ```
   hdfs-site.xml file looks like:
   
   ```
   <property>
     <name>dfs.replication</name>
     <value>3</value>
   </property>
   <property>
     <name>dfs.namenode.name.dir</name>
     <value>/hadoop/namenode/</value>
   </property>
   <property>
     <name>dfs.datanode.data.dir</name>   
     <value>/hadoop/namenode</value>
   </property>
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] ad1happy2go commented on issue #8144: [SUPPORT]Unable to connect to an s3 hudi table

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8144:
URL: https://github.com/apache/hudi/issues/8144#issuecomment-1550147860

   @peter-mccabe Were you able to resolve this issue? If yes please share insights and feel free to close the issue. Else can you please provide more details to triage this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] ad1happy2go commented on issue #8144: [SUPPORT]Unable to connect to an s3 hudi table

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8144:
URL: https://github.com/apache/hudi/issues/8144#issuecomment-1504517165

   @peter-mccabe 
   Are you still facing this issue? If yes, can you share complete stack trace of the error? 
   Are you setting up S3 keys properly in Hadoop fs configuration to connect to S3. (props like - fs.s3a.access.key)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] peter-mccabe commented on issue #8144: [SUPPORT]Unable to connect to an s3 hudi table

Posted by "peter-mccabe (via GitHub)" <gi...@apache.org>.
peter-mccabe commented on issue #8144:
URL: https://github.com/apache/hudi/issues/8144#issuecomment-1494974166

   any update on this? i really need to be able to manage this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on issue #8144: [SUPPORT]Unable to connect to an s3 hudi table

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8144:
URL: https://github.com/apache/hudi/issues/8144#issuecomment-1464804750

   Seems a permission setup issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org