[GitHub] [hudi] aznwarmonkey edited a comment on issue #4541: [SUPPORT] NullPointerException while writing Bulk ingest table

[GitBox <gi...@apache.org>]

aznwarmonkey edited a comment on issue #4541:
URL: https://github.com/apache/hudi/issues/4541#issuecomment-1008283680


   > 
   
   Hi, 
   
   After making the suggested changes as mentioned above, I am getting this exception thrown immediately:
   
   ```bash
   py4j.protocol.Py4JJavaError: An error occurred while calling o154.save.
   : org.apache.hudi.exception.HoodieException: Invalid value of Type.
           at org.apache.hudi.common.model.WriteOperationType.fromValue(WriteOperationType.java:86)
           at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:83)
           at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:134)
   ```
   The updated config looks like the following
   
   ```python
           hudi_options = {
               'hoodie.table.name': table_name,
               'hoodie.datasource.write.keygenerator.class':
                   'org.apache.hudi.keygen.ComplexKeyGenerator',
               'hoodie.datasource.write.recordkey.field': keys,
               'hoodie.datasource.write.partitionpath.field': ','.join(partitions),
               'hoodie.datasource.write.hive_style_partitioning': True,
               'hoodie.datasource.write.table.name': table_name,
               'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
               'hoodie.datasource.write.precombine.field': timestamp_col,
               'hoodie.index.type': 'BLOOM',
               'hoodie.consistency.check.enabled': True,
               'hoodie.parquet.small.file.limit': 134217728,
               'hoodie.parquet.max.file.size': 1073741824,
               'write.bulk_insert.shuffle_by_partition': True,
               'hoodie.copyonwrite.insert.auto.split': False,
               'hoodie.copyonwrite.insert.split.size': 10000000,
               'hoodie.datasource.write.row.writer.enable': True,
               'hoodie.bulkinsert.sort.mode': 'PARTITION_SORT',
               'hoodie.bulkinsert.shuffle.parallelism': 50,
               'hoodie.cleaner.commits.retained': '2',
               'hoodie.clean.async': True,
           }
   
           df.write.format('org.apache.hudi') \
               .option('hoodie.datasource.write.operation', 'bulk_ingest') \
               .options(**hudi_options).mode('append').save(output_path)
   ```
   
   If i change `hoodie.datasource.write.operation` to `upsert` then I do not get the error. (I am currently rerunning with `upsert` as i type this to ensure the process fully completes`
   
   A couple of questions and what I am hoping to accomplish:
   - For whatever reason unless, if i use ` 'hoodie.datasource.write.row.writer.enable': True`  with `bulk_insert` the write operation is significantly slower, it is about 2x-3x slower 
   - `write.parquet.block.size` is this in bytes or mb? i believe the default is 120: https://hudi.apache.org/docs/configurations/#writeparquetblocksize
   - when writing, I noticed I am getting a bunch of small parquet files ~20-50mb which i am trying to avoid, the desired size is  at least 128MB. Tuning small.file.limit and max.file.size seem to have no noticeable impact, which is why having clustering is important (as the next time it writes to the same partition the hope is to combine the small files together)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org