You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/13 13:40:25 UTC
[GitHub] [hudi] LinMingQiang opened a new issue, #6390: [SUPPORT] HoodieCompactor.generateCompactionPlan return the wrong results operations.
LinMingQiang opened a new issue, #6390:
URL: https://github.com/apache/hudi/issues/6390
Caused by: java.io.FileNotFoundException: File file:/hudi_tbl/2/.00000000-575e-4905-85e4-fb62d29a4bea_20220813205548259.log.1_0-1-0 does not exist
HoodieCompactor.generateCompactionPlan
List<HoodieCompactionOperation> operations = context.flatMap(partitionPaths, partitionPath -> fileSystemView
.getLatestFileSlices(partitionPath)
.filter(slice -> !fgIdsInPendingCompactionAndClustering.contains(slice.getFileGroupId()))
.map(s ->
// error : partitionPath.equals(s.getPartitionPath()) -> is false ???
// This will cause the partition to correspond to the wrong fileid .
);
How to reproduce
flink : RuntimeExecutionMode.BATCH
public static String srcTbl = "create table test_data (\n" +
" msg string,\n" +
" name string,\n" +
" rowtime string,\n" +
" dt STRING\n" +
") with (\n" +
" 'connector' = 'filesystem',\n" +
" 'path' = '"+srcPath+"',\n" +
" 'format' = 'csv'\n" +
")";
public static String hudiTbl = "CREATE TABLE hudi_tbl(\n" +
" msg STRING PRIMARY KEY NOT ENFORCED,\n" +
" `name` STRING,\n" +
" rowtime BIGINT,\n" +
" `dt` STRING\n" +
") PARTITIONED BY (`dt`) WITH (\n" +
" 'connector' = 'hudi',\n" +
" 'table.type' = 'MERGE_ON_READ',\n" +
" 'path' = '"+path+"',\n" +
" 'write.payload.class' = 'org.apache.hudi.common.model.OverwriteNonDefaultsWithLatestAvroPayload',\n" +
" 'write.precombine' = 'true',\n" +
" 'write.precombine.field' = 'rowtime',\n" +
" 'compaction.delta_commits' = '1',\n" +
" 'compaction.schedule.enable' = 'true',\n" +
" 'compaction.async.enabled' = 'true',\n" +
" 'changelog.enabled' = 'false',\n" +
" 'index.type' = 'BUCKET', \n" +
" 'hoodie.bucket.index.num.buckets'='1', \n" +
" 'write.tasks' = '1')\n";
String insertSql = "insert into hudi_tbl " +
" select msg,name," +
" UNIX_TIMESTAMP(DATE_FORMAT(rowtime,'yyyy-MM-dd HH:mm:ss')) as rowtime," +
" dt from test_data";
How to create data:
FileWriter s = new FileWriter(f);
Random r = new Random();
for (int i = 0; i < 10000; i++) {
s.write("msg"+r.nextInt(100)+",name,2022-01-03 11:11:11," + r.nextInt(10));
s.write("\n");
}
like this :
`msg19,name,2022-01-03 11:11:11,1`
**Environment Description**
* Hudi version : master or release-0.12.0
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #6390: [SUPPORT] HoodieCompactor.generateCompactionPlan return the wrong results operations.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on issue #6390:
URL: https://github.com/apache/hudi/issues/6390#issuecomment-1214561576
Concurrent requests will improve efficiency. I have tried that non concurrent requests will affect the speed of `Compact`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6390: [SUPPORT] FileNotFoundException: file:/hudi_tbl/2/.00000000-575e-4905-85e4-fb62d29a4bea_20220813205548259.log.1_0-1-0 does not exist
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6390:
URL: https://github.com/apache/hudi/issues/6390#issuecomment-1229260074
can you try w/ latest master. we have fixed a bug around this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang closed issue #6390: [SUPPORT] FileNotFoundException: file:/hudi_tbl/2/.00000000-575e-4905-85e4-fb62d29a4bea_20220813205548259.log.1_0-1-0 does not exist
Posted by GitBox <gi...@apache.org>.
LinMingQiang closed issue #6390: [SUPPORT] FileNotFoundException: file:/hudi_tbl/2/.00000000-575e-4905-85e4-fb62d29a4bea_20220813205548259.log.1_0-1-0 does not exist
URL: https://github.com/apache/hudi/issues/6390
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #6390: [SUPPORT] HoodieCompactor.generateCompactionPlan return the wrong results operations.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #6390:
URL: https://github.com/apache/hudi/issues/6390#issuecomment-1214547742
The code `context.flatMap(partitionPaths, partitionPath` is problematic because it may cause concurrent access to the remote fs view, the fs view is not thread safe now BTW.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org