You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "ly7213 (via GitHub)" <gi...@apache.org> on 2025/06/23 07:39:18 UTC
[I] [Bug] The directory item limit of index is exceeded [paimon]
ly7213 opened a new issue, #5788:
URL: https://github.com/apache/paimon/issues/5788
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon1.0.1
### Compute Engine
paimon-flink-1.19-1.0.1.jar
### Minimal reproduce step
I created a new table with pk like this:
caiji_time STRING,
dt STRING,
PRIMARY KEY (mid,dt) NOT ENFORCED
) PARTITIONED BY (dt) WITH (
'compaction.optimization-interval' = '7d',
'sink.committer-memory' = '500m',
'changelog-producer' = 'none', //No changelog file.
'deletion-vectors.enabled' = 'true', //MOW(Merge On Write)
'metastore.partitioned-table' = 'true',
'snapshot.num-retained.min' = '1',
'snapshot.num-retained.max' = '3',
'snapshot.time-retained' = '1h',
'metastore.partitioned-table' = 'true',
'file.format' = 'parquet',
'write-buffer-size' = '100mb',
'write-buffer-spillable' = 'true' ,
'partition.timestamp-formatter' = 'yyyy-MM-dd',
'partition.timestamp-pattern' = '$dt'
);
Then I customized a streaming job using KafkaSource and FlinkSinkBuilder. However, the files in the directory of index keep growing until my Hadoop reported: the item limit of /XXX/index is exceeded, they cause the task failed.
### What doesn't meet your expectations?
1. What are these index files?Why are they keep growing?
2. How can I resolve this problom?
### Anything else?
Nothing
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org