[GitHub] [hudi] James601232 commented on issue #3963: [SUPPORT] Use hudi-java-client to create hudi table incorrect

[GitBox <gi...@apache.org>]

James601232 commented on issue #3963:
URL: https://github.com/apache/hudi/issues/3963#issuecomment-966082120


   > @James601232 I'm not sure if i get it. In the code `client` it does not do any upsert/insert/bulkinsert operation, it's just to init a hudi table, which it did by creating `.hoodie/`. You need to use the client to write data to the table as well if you intend to do so.
   
   yes.  you are right.  so i add insert method in my codes. now create table is ok. but no data insert into table. my codes as following:
   
   `
   package com.tx.bigdata;
   
   import org.apache.hadoop.conf.Configuration;
   import org.apache.hadoop.fs.FileSystem;
   import org.apache.hadoop.fs.Path;
   import org.apache.hudi.client.HoodieJavaWriteClient;
   import org.apache.hudi.client.common.HoodieJavaEngineContext;
   import org.apache.hudi.common.fs.FSUtils;
   import org.apache.hudi.common.model.HoodieAvroPayload;
   import org.apache.hudi.common.model.HoodieKey;
   import org.apache.hudi.common.model.HoodieRecord;
   import org.apache.hudi.common.model.HoodieTableType;
   import org.apache.hudi.common.table.HoodieTableMetaClient;
   import org.apache.hudi.config.HoodieCompactionConfig;
   import org.apache.hudi.config.HoodieIndexConfig;
   import org.apache.hudi.config.HoodieWriteConfig;
   import org.apache.hudi.examples.common.HoodieExampleDataGenerator;
   import org.apache.hudi.index.HoodieIndex;
   import org.apache.log4j.LogManager;
   import org.apache.log4j.Logger;
   
   import java.util.ArrayList;
   import java.util.List;
   import java.util.stream.Collectors;
   
   
   public class HoodieJavaWriteClientExample {
   
     private static final Logger LOG = LogManager.getLogger(HoodieJavaWriteClientExample.class);
   
     private static String tableType = HoodieTableType.COPY_ON_WRITE.name();
   
     public static void main(String[] args) throws Exception {
   //    if (args.length < 2) {
   //      System.err.println("Usage: HoodieJavaWriteClientExample <tablePath> <tableName>");
   //      System.exit(1);
   //    }
   //    String tablePath = args[0];
   //    String tableName = args[1];
   
       String tablePath = "/home/work/hudi_catalog4";
       String tableName = "hudi_clyang_table";
   
       // Generator of some records to be loaded in.
       HoodieExampleDataGenerator<HoodieAvroPayload> dataGen = new HoodieExampleDataGenerator<>();
   
       Configuration hadoopConf = new Configuration();
   
       hadoopConf.addResource(new Path("core-site.xml"));
       hadoopConf.addResource(new Path("hdfs-site.xml"));
       System.setProperty("HADOOP_USER_NAME", "work");
   
       // initialize the table, if not done already
       Path path = new Path(tablePath);
       FileSystem fs = FSUtils.getFs(tablePath, hadoopConf);
       if (!fs.exists(path)) {
         HoodieTableMetaClient.withPropertyBuilder()
           .setTableType(tableType)
           .setTableName(tableName)
           .setPayloadClassName(HoodieAvroPayload.class.getName())
           .initTable(hadoopConf, tablePath);
       }
   
       // Create the write client to write some records in
       HoodieWriteConfig cfg = HoodieWriteConfig.newBuilder().withPath(tablePath)
           .withSchema(HoodieExampleDataGenerator.TRIP_EXAMPLE_SCHEMA).withParallelism(2, 2)
           .withDeleteParallelism(2).forTable(tableName)
           .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexType(HoodieIndex.IndexType.INMEMORY).build())
           .withCompactionConfig(HoodieCompactionConfig.newBuilder().archiveCommitsWith(20, 30).build()).build();
       HoodieJavaWriteClient<HoodieAvroPayload> client =
           new HoodieJavaWriteClient<>(new HoodieJavaEngineContext(hadoopConf), cfg);
   
       // inserts
       String newCommitTime = client.startCommit();
       LOG.info("Starting commit " + newCommitTime);
       List<HoodieRecord<HoodieAvroPayload>> records = dataGen.generateInserts(newCommitTime, 10);
       List<HoodieRecord<HoodieAvroPayload>> recordsSoFar = new ArrayList<>(records);
       List<HoodieRecord<HoodieAvroPayload>> writeRecords =
           recordsSoFar.stream().map(r -> new HoodieRecord<HoodieAvroPayload>(r)).collect(Collectors.toList());
       client.insert(writeRecords, newCommitTime);
       client.close();
     }
   }
   `
   these codes from hudi githup test project


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org