You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/12/23 07:50:18 UTC

[GitHub] [hudi] waywtdcc opened a new issue #4432: [SUPPORT] The parquet file size exceeds the configured value

waywtdcc opened a new issue #4432:
URL: https://github.com/apache/hudi/issues/4432


   The parquet file size exceeds the configured value
   
   ![image](https://user-images.githubusercontent.com/59957056/147206536-f7206021-d932-4b2c-affa-6e61af43ed52.png)
   
   hudi:0.10 flink:1.13.5
   
   DDL:
   CREATE TABLE hudi.datagen_hudi2(
   id bigint ,
   name1        VARCHAR(10),
       name2        VARCHAR(10),
       name3        VARCHAR(10),
       name4        VARCHAR(10),
       name5        VARCHAR(10),
       name6        VARCHAR(10),
       name7        VARCHAR(10),
       name8        VARCHAR(10),
       name9        VARCHAR(10),
       name10        VARCHAR(10),
       name11        VARCHAR(10),
       name12        VARCHAR(10),
       name13        VARCHAR(10),
       name14        VARCHAR(10),
       name15        VARCHAR(10),
       name16        VARCHAR(10),
       name17        VARCHAR(10),
       name18        VARCHAR(10),
       name19        VARCHAR(10),
       name20        VARCHAR(10),
       name21        VARCHAR(10),
       name22        VARCHAR(10),
       name23        VARCHAR(10),
       name24        VARCHAR(10),
       name25        VARCHAR(10),
       name26        VARCHAR(10),
       name27        VARCHAR(10),
       name28        VARCHAR(10),
       name29        VARCHAR(10),
       name30        VARCHAR(10),
       name31        VARCHAR(10),
       name32        VARCHAR(10),
       name33        VARCHAR(10),
       name34        VARCHAR(10),
       name35        VARCHAR(10),
       name36        VARCHAR(10),
       name37        VARCHAR(10),
       name38        VARCHAR(10),
       name39        VARCHAR(10),
       name40        VARCHAR(10),
       name41        VARCHAR(10),
       name42        VARCHAR(10),
       name43        VARCHAR(10),
       name44        VARCHAR(10),
       name45        VARCHAR(10),
       name46        VARCHAR(10),
       name47        VARCHAR(10),
       name48        VARCHAR(10),
       name49        VARCHAR(10),
       name50        VARCHAR(10),
       name        VARCHAR(20),
   
   birthday TIMESTAMP(3),
   
   ts TIMESTAMP(3),
   
   `partition_str` VARCHAR(20),
   
   primary key(id) not enforced --必须指定uuid 主键
   
   )
   
   PARTITIONED BY (`partition_str`)
   
   with(
   
   'connector'='hudi',
   
   'path'= 'hdfs://test/user/hive/warehouse/hudi.db/datagen_hudi2'
   
   , 'hoodie.datasource.write.recordkey.field'= 'id'-- 主键
   
   , 'write.tasks'= '1'
   
   , 'compaction.tasks'= '1'
   
   , 'table.type'= 'MERGE_ON_READ'-- 默认COPY_ON_WRITE,可选MERGE_ON_READ 
   
   , 'compaction.async.enabled'= 'true'-- 是否开启异步压缩
   
   , 'compaction.trigger.strategy'= 'num_or_time'
   
   , 'compaction.delta_commits'= '2',   -- 默认为5
   'compaction.delta_seconds' = '120',
     'hive_sync.enable' = 'true', 
     'hive_sync.mode' = 'hms'    ,
     'hive_sync.metastore.uris' = '*',
       'hive_sync.table'='datagen_hudi2',                         
     'hive_sync.db'='hudi'  ,
     'index.global.enabled' = 'true',
     'index.bootstrap.enabled' = 'true',
   'write.log_block.size' = '61'
   );


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1001762429


   yes, hudi by default uses gzip for parquet compression. but you can override the compression if need be. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan closed issue #4432: [SUPPORT] The parquet file size exceeds the configured value

Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4432:
URL: https://github.com/apache/hudi/issues/4432


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1002861656


   please re-open if the proposed solution does not work.thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] danny0405 commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value

Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1000761369


   Yes, the file size control is just an estimation. You can set the record size with option `hoodie.copyonwrite.record.size.estimate` manually to make the file size more precise BTW.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1001807101


   Have filed a tracking jira. Will be looking into it.
   https://issues.apache.org/jira/browse/HUDI-3110
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan edited a comment on issue #4432: [SUPPORT] The parquet file size exceeds the configured value

Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1001807101


   Have filed a tracking jira. Will be looking into it.
   https://issues.apache.org/jira/browse/HUDI-3110
   
   @waywtdcc: may I know whats the operation you are using? and did you try w/ COW and facing the same issue ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1002316249


   Can you also try setting `hoodie.parquet.block.size` in addition to parquet.max.file.size.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan edited a comment on issue #4432: [SUPPORT] The parquet file size exceeds the configured value

Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1001807101


   Have filed a tracking jira. Will be looking into it.
   https://issues.apache.org/jira/browse/HUDI-3110
   
   @waywtdcc: may I know whats the operation you are using? is it after initial bulk_insert operation or even after incremental ingestion, the file keeps growing? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan closed issue #4432: [SUPPORT] The parquet file size exceeds the configured value

Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4432:
URL: https://github.com/apache/hudi/issues/4432


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1002165461


   with bulk_insert, file sizing is not guaranteed as its closely tied to flushing of data to disk. So, if you are seeing issues w/ bulk_insert, it is expected. but if you see it with incremental ingestion. that the file keeps growing, then it needs to be looked into. Let me know where exactly do you see the problem. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org