[GitHub] [hudi] Archie-selfless commented on issue #6572: [SUPPORT] DeltaStreamer sync hive-table called NPE when adding patition(s)

[GitBox <gi...@apache.org>]

Archie-selfless commented on issue #6572:
URL: https://github.com/apache/hudi/issues/6572#issuecomment-1280713709

   @yihua I'm really sorry for replying the current issue so late! With the help of my colleagues, I completed the Hive-sync by accessing S3 protocol. I think it's because the org.apache.hudi.hive.HiveSyncTool always fetch the s3\s3a as protocol. My current core-site.xml is here:
   
   ```xml
   <configuration>
       <property>
           <name>fs.defaultFS</name>
           <value>cons://bigdata-1252394470</value>
           <description>COS bucket name</description>
       </property>
   
       <property>
           <name>fs.cosn.userinfo.secretId</name>
           <value>XXXXXXXX</value>
           <description>Tencent Cloud Secret Id</description>
       </property>
   
       <property>
           <name>fs.cosn.userinfo.secretKey</name>
           <value>XXXXXXXX</value>
           <description>Tencent Cloud Secret Key</description>
       </property>
   
       <property>
           <name>fs.cosn.bucket.region</name>
           <value>ap-guangzhou</value>
           <description>The region where the bucket is located.</description>
       </property>
   
       <property>
           <name>fs.cosn.bucket.endpoint_suffix</name>
           <value>cos.ap-guangzhou.myqcloud.com</value>
           <description>
               COS endpoint to connect to.
               For public cloud users, it is recommended not to set this option, and only the correct area field is required.
           </description>
       </property>
   
       <property>
           <name>fs.cosn.impl</name>
           <value>org.apache.hadoop.fs.CosFileSystem</value>
           <description>The implementation class of the CosN Filesystem.</description>
       </property>
   
       <property>
           <name>fs.AbstractFileSystem.cosn.impl</name>
           <value>org.apache.hadoop.fs.CosN</value>
           <description>The implementation class of the CosN AbstractFileSystem.</description>
       </property>
   
   
       <!-- ############### If I add below params, everything goes well ############### -->
   
       <property>
           <name>fs.s3.impl</name>
           <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
       </property>
       <property>
           <name>fs.s3a.impl</name>
           <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
       </property>
       
       <property>
           <name>fs.s3n.impl</name>
           <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
       </property>
       <property>
           <name>fs.s3.endpoint</name>
           <value>cos.ap-guangzhou.myqcloud.com</value>
       </property>
       <property>
           <name>fs.s3a.endpoint</name>
           <value>cos.ap-guangzhou.myqcloud.com</value>
       </property>
       <property>
           <name>fs.s3n.endpoint</name>
           <value>cos.ap-guangzhou.myqcloud.com</value>
       </property>
   
       <property>
           <name>fs.s3a.access.key</name>
           <value>XXXXXXXX</value>
       </property>
   
       <property>
           <name>fs.s3a.secret.key</name>
           <value>XXXXXXXX</value>
       </property>
   
   
       <property>
           <name>fs.s3a.aws.credentials.provider</name>
           <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
       </property>
   
   </configuration>
   ```
   
   **Main Summary**: org.apache.hudi.hive.HiveSyncTool always fetch the s3\s3a as protocol.(Based Hudi-0.12.0)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org