You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/12/23 07:50:18 UTC
[GitHub] [hudi] waywtdcc opened a new issue #4432: [SUPPORT] The parquet file size exceeds the configured value
waywtdcc opened a new issue #4432:
URL: https://github.com/apache/hudi/issues/4432
The parquet file size exceeds the configured value

hudi:0.10 flink:1.13.5
DDL:
CREATE TABLE hudi.datagen_hudi2(
id bigint ,
name1 VARCHAR(10),
name2 VARCHAR(10),
name3 VARCHAR(10),
name4 VARCHAR(10),
name5 VARCHAR(10),
name6 VARCHAR(10),
name7 VARCHAR(10),
name8 VARCHAR(10),
name9 VARCHAR(10),
name10 VARCHAR(10),
name11 VARCHAR(10),
name12 VARCHAR(10),
name13 VARCHAR(10),
name14 VARCHAR(10),
name15 VARCHAR(10),
name16 VARCHAR(10),
name17 VARCHAR(10),
name18 VARCHAR(10),
name19 VARCHAR(10),
name20 VARCHAR(10),
name21 VARCHAR(10),
name22 VARCHAR(10),
name23 VARCHAR(10),
name24 VARCHAR(10),
name25 VARCHAR(10),
name26 VARCHAR(10),
name27 VARCHAR(10),
name28 VARCHAR(10),
name29 VARCHAR(10),
name30 VARCHAR(10),
name31 VARCHAR(10),
name32 VARCHAR(10),
name33 VARCHAR(10),
name34 VARCHAR(10),
name35 VARCHAR(10),
name36 VARCHAR(10),
name37 VARCHAR(10),
name38 VARCHAR(10),
name39 VARCHAR(10),
name40 VARCHAR(10),
name41 VARCHAR(10),
name42 VARCHAR(10),
name43 VARCHAR(10),
name44 VARCHAR(10),
name45 VARCHAR(10),
name46 VARCHAR(10),
name47 VARCHAR(10),
name48 VARCHAR(10),
name49 VARCHAR(10),
name50 VARCHAR(10),
name VARCHAR(20),
birthday TIMESTAMP(3),
ts TIMESTAMP(3),
`partition_str` VARCHAR(20),
primary key(id) not enforced --必须指定uuid 主键
)
PARTITIONED BY (`partition_str`)
with(
'connector'='hudi',
'path'= 'hdfs://test/user/hive/warehouse/hudi.db/datagen_hudi2'
, 'hoodie.datasource.write.recordkey.field'= 'id'-- 主键
, 'write.tasks'= '1'
, 'compaction.tasks'= '1'
, 'table.type'= 'MERGE_ON_READ'-- 默认COPY_ON_WRITE,可选MERGE_ON_READ
, 'compaction.async.enabled'= 'true'-- 是否开启异步压缩
, 'compaction.trigger.strategy'= 'num_or_time'
, 'compaction.delta_commits'= '2', -- 默认为5
'compaction.delta_seconds' = '120',
'hive_sync.enable' = 'true',
'hive_sync.mode' = 'hms' ,
'hive_sync.metastore.uris' = '*',
'hive_sync.table'='datagen_hudi2',
'hive_sync.db'='hudi' ,
'index.global.enabled' = 'true',
'index.bootstrap.enabled' = 'true',
'write.log_block.size' = '61'
);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1001762429
yes, hudi by default uses gzip for parquet compression. but you can override the compression if need be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4432: [SUPPORT] The parquet file size exceeds the configured value
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4432:
URL: https://github.com/apache/hudi/issues/4432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1002861656
please re-open if the proposed solution does not work.thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1000761369
Yes, the file size control is just an estimation. You can set the record size with option `hoodie.copyonwrite.record.size.estimate` manually to make the file size more precise BTW.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1001807101
Have filed a tracking jira. Will be looking into it.
https://issues.apache.org/jira/browse/HUDI-3110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #4432: [SUPPORT] The parquet file size exceeds the configured value
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1001807101
Have filed a tracking jira. Will be looking into it.
https://issues.apache.org/jira/browse/HUDI-3110
@waywtdcc: may I know whats the operation you are using? and did you try w/ COW and facing the same issue ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1002316249
Can you also try setting `hoodie.parquet.block.size` in addition to parquet.max.file.size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #4432: [SUPPORT] The parquet file size exceeds the configured value
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1001807101
Have filed a tracking jira. Will be looking into it.
https://issues.apache.org/jira/browse/HUDI-3110
@waywtdcc: may I know whats the operation you are using? is it after initial bulk_insert operation or even after incremental ingestion, the file keeps growing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4432: [SUPPORT] The parquet file size exceeds the configured value
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4432:
URL: https://github.com/apache/hudi/issues/4432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4432:
URL: https://github.com/apache/hudi/issues/4432#issuecomment-1002165461
with bulk_insert, file sizing is not guaranteed as its closely tied to flushing of data to disk. So, if you are seeing issues w/ bulk_insert, it is expected. but if you see it with incremental ingestion. that the file keeps growing, then it needs to be looked into. Let me know where exactly do you see the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org