You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2025/03/21 07:02:29 UTC
[PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
lokeshj1703 opened a new pull request, #13010:
URL: https://github.com/apache/hudi/pull/13010
### Change Logs
The PR adds capability to read inflight instants with HoodieLogRecordReader by adding `allowInflightInstants` flag. The capability is required for updating RLI since RLI reads the log records and finds deleted keys.
While replacing `HoodieMergedLogRecordScanner` with new reader class, we need to use the allowInflightInstants flag.
### Impact
NA
### Risk level (write none, low medium or high below)
low
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2020036550
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -82,20 +82,17 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private final Option<UnaryOperator<T>> outputConverter;
private final HoodieReadStats readStats;
+ // Allows to consider inflight instants while merging log records using HoodieMergedLogRecordReader
+ // The inflight instants need to be considered while updating RLI records. RLI needs to fetch the revived
+ // and deleted keys from the log files written as part of active data commit. During the RLI update,
+ // the allowInflightInstants flag would need to be set to true. This would ensure the HoodieMergedLogRecordReader
Review Comment:
I'm OK with this flag temporarily, but as @danny0405 mentioned we should really get rid of such unorthodox read logic that is only due to bandaid, after rewriting of MDT write path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #13010:
URL: https://github.com/apache/hudi/pull/13010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2014803596
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -288,13 +301,21 @@ private FileSlice getFileSliceToRead(StorageConfiguration<?> storageConf,
return fileSlice;
}
- private List<T> readRecordsFromFileGroup(StorageConfiguration<?> storageConf,
- String tablePath,
- HoodieTableMetaClient metaClient,
- FileSlice fileSlice,
- Schema avroSchema,
- RecordMergeMode recordMergeMode,
- boolean isSkipMerge) throws Exception {
+ private FileSlice getFileSliceToReadIncludingInflight(StorageConfiguration<?> storageConf, String tablePath,
+ HoodieTableMetaClient metaClient, String[] partitionPaths,
+ boolean containsBaseFile, int expectedLogFileNum) {
+ HoodieEngineContext engineContext = new HoodieLocalEngineContext(storageConf);
+ HoodieTableFileSystemView fsView = HoodieTableFileSystemView.fileListingBasedFileSystemView(engineContext, metaClient, metaClient.getActiveTimeline(), false);
Review Comment:
This code has been removed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2758095785
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
}, {
"hash" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444",
"triggerID" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"triggerType" : "PUSH"
}, {
"hash" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4446",
"triggerID" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"triggerType" : "PUSH"
}, {
"hash" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4449",
"triggerID" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"triggerType" : "PUSH"
}, {
"hash" : "988700daaf6c63683b48bf8b7318a857a929305a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4493",
"triggerID" : "988700daaf6c63683b48bf8b7318a857a929305a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910418360e4be2e9edf3464e1afc915516eb85ae Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4449)
* 988700daaf6c63683b48bf8b7318a857a929305a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4493)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2755430793
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
}, {
"hash" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398)
* 3f73cefed6aa7459e1413326ebbaf328fd64c1cb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2755794521
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
}, {
"hash" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444",
"triggerID" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"triggerType" : "PUSH"
}, {
"hash" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4446",
"triggerID" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"triggerType" : "PUSH"
}, {
"hash" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4449",
"triggerID" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910418360e4be2e9edf3464e1afc915516eb85ae Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2752361652
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2019066769
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -82,20 +82,17 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private final Option<UnaryOperator<T>> outputConverter;
private final HoodieReadStats readStats;
+ // Allows to consider inflight instants while merging log records using HoodieMergedLogRecordReader
+ // The inflight instants need to be considered while updating RLI records. RLI needs to fetch the revived
+ // and deleted keys from the log files written as part of active data commit. During the RLI update,
+ // the allowInflightInstants flag would need to be set to true. This would ensure the HoodieMergedLogRecordReader
Review Comment:
yes, you are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2014670266
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderOnSpark.scala:
##########
@@ -176,4 +184,25 @@ class TestHoodieFileGroupReaderOnSpark extends TestHoodieFileGroupReaderBase[Int
assertEquals(expectedOrderingValue,
metadataMap.get(HoodieReaderContext.INTERNAL_META_ORDERING_FIELD))
}
+
+ @ParameterizedTest
+ @EnumSource(classOf[RecordMergeMode])
+ @throws[Exception]
+ def testReadFileGroupInflightData(recordMergeMode: RecordMergeMode): Unit = {
+ val writeConfigs = new util.HashMap[String, String](getCommonConfigs(recordMergeMode))
+ writeConfigs.put(DataSourceWriteOptions.TABLE_TYPE.key(), DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL)
+ try {
+ val dataGen = new HoodieTestDataGenerator(0xDEEF)
+ try {
+ // One commit; reading one file group containing a base file only
+ commitToTable(dataGen.generateInserts("001", 100), INSERT.value, writeConfigs)
+ validateOutputFromFileGroupReader(getStorageConf, getBasePath, dataGen.getPartitionPaths, true, 0, recordMergeMode)
+
+ commitToTable(dataGen.generateUniqueUpdates("003", 100), UPSERT.value, writeConfigs)
Review Comment:
Could you prepare the table instead of transactions? Also, could this validation be added to existing tests, i.e., adding a new step in existing tests to remove completed detlacommit from the timeline so that data files become inflight and should not be read? So that we don't need this new test and we can have more coverage with different permutations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2752175889
@lokeshj1703 , the context is not clear. Can you explain more on "since RLI reads the log records and finds deleted keys."?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2014804106
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderOnSpark.scala:
##########
@@ -176,4 +184,25 @@ class TestHoodieFileGroupReaderOnSpark extends TestHoodieFileGroupReaderBase[Int
assertEquals(expectedOrderingValue,
metadataMap.get(HoodieReaderContext.INTERNAL_META_ORDERING_FIELD))
}
+
+ @ParameterizedTest
+ @EnumSource(classOf[RecordMergeMode])
+ @throws[Exception]
+ def testReadFileGroupInflightData(recordMergeMode: RecordMergeMode): Unit = {
+ val writeConfigs = new util.HashMap[String, String](getCommonConfigs(recordMergeMode))
+ writeConfigs.put(DataSourceWriteOptions.TABLE_TYPE.key(), DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL)
+ try {
+ val dataGen = new HoodieTestDataGenerator(0xDEEF)
+ try {
+ // One commit; reading one file group containing a base file only
+ commitToTable(dataGen.generateInserts("001", 100), INSERT.value, writeConfigs)
+ validateOutputFromFileGroupReader(getStorageConf, getBasePath, dataGen.getPartitionPaths, true, 0, recordMergeMode)
+
+ commitToTable(dataGen.generateUniqueUpdates("003", 100), UPSERT.value, writeConfigs)
+ val metaClient = HoodieTestUtils.createMetaClient(getStorageConf, getBasePath)
+ metaClient.getStorage.deleteFile(new StoragePath(metaClient.getTimelinePath, new DefaultInstantFileNameGenerator().getFileName(metaClient.getActiveTimeline.lastInstant().get())))
+ validateOutputFromFileGroupReaderIncludingInflight(getStorageConf, getBasePath, dataGen.getPartitionPaths, true, 1, recordMergeMode, true)
Review Comment:
I have added a UT now. This test has been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2013058804
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -288,13 +301,21 @@ private FileSlice getFileSliceToRead(StorageConfiguration<?> storageConf,
return fileSlice;
}
- private List<T> readRecordsFromFileGroup(StorageConfiguration<?> storageConf,
- String tablePath,
- HoodieTableMetaClient metaClient,
- FileSlice fileSlice,
- Schema avroSchema,
- RecordMergeMode recordMergeMode,
- boolean isSkipMerge) throws Exception {
+ private FileSlice getFileSliceToReadIncludingInflight(StorageConfiguration<?> storageConf, String tablePath,
+ HoodieTableMetaClient metaClient, String[] partitionPaths,
+ boolean containsBaseFile, int expectedLogFileNum) {
+ HoodieEngineContext engineContext = new HoodieLocalEngineContext(storageConf);
+ HoodieTableFileSystemView fsView = HoodieTableFileSystemView.fileListingBasedFileSystemView(engineContext, metaClient, metaClient.getActiveTimeline(), false);
Review Comment:
where are we closing the FSV ?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderOnSpark.scala:
##########
@@ -176,4 +184,25 @@ class TestHoodieFileGroupReaderOnSpark extends TestHoodieFileGroupReaderBase[Int
assertEquals(expectedOrderingValue,
metadataMap.get(HoodieReaderContext.INTERNAL_META_ORDERING_FIELD))
}
+
+ @ParameterizedTest
+ @EnumSource(classOf[RecordMergeMode])
+ @throws[Exception]
+ def testReadFileGroupInflightData(recordMergeMode: RecordMergeMode): Unit = {
+ val writeConfigs = new util.HashMap[String, String](getCommonConfigs(recordMergeMode))
+ writeConfigs.put(DataSourceWriteOptions.TABLE_TYPE.key(), DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL)
+ try {
+ val dataGen = new HoodieTestDataGenerator(0xDEEF)
+ try {
+ // One commit; reading one file group containing a base file only
+ commitToTable(dataGen.generateInserts("001", 100), INSERT.value, writeConfigs)
+ validateOutputFromFileGroupReader(getStorageConf, getBasePath, dataGen.getPartitionPaths, true, 0, recordMergeMode)
+
+ commitToTable(dataGen.generateUniqueUpdates("003", 100), UPSERT.value, writeConfigs)
+ val metaClient = HoodieTestUtils.createMetaClient(getStorageConf, getBasePath)
+ metaClient.getStorage.deleteFile(new StoragePath(metaClient.getTimelinePath, new DefaultInstantFileNameGenerator().getFileName(metaClient.getActiveTimeline.lastInstant().get())))
+ validateOutputFromFileGroupReaderIncludingInflight(getStorageConf, getBasePath, dataGen.getPartitionPaths, true, 1, recordMergeMode, true)
Review Comment:
I would expect, we have a log file from a concurrent writer or something and if we initialize the FG reader by setting "allowInflightInstants = true". FG reader should return the records from inflight log file as well.
we don't need to add functional test for this.
just write a test directly against a FG reader (one file group essentially).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2016213756
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/BaseHoodieLogRecordReader.java:
##########
@@ -136,16 +136,14 @@ public abstract class BaseHoodieLogRecordReader<T> {
// Use scanV2 method.
private final boolean enableOptimizedLogBlocksScan;
protected FileGroupRecordBuffer<T> recordBuffer;
+ // Allows to consider inflight instants while merging log records
Review Comment:
Can we add more doc to address that this is only used for MDT RLI index, and give an use case how it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2758088559
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
}, {
"hash" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444",
"triggerID" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"triggerType" : "PUSH"
}, {
"hash" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4446",
"triggerID" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"triggerType" : "PUSH"
}, {
"hash" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4449",
"triggerID" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"triggerType" : "PUSH"
}, {
"hash" : "988700daaf6c63683b48bf8b7318a857a929305a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "988700daaf6c63683b48bf8b7318a857a929305a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910418360e4be2e9edf3464e1afc915516eb85ae Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4449)
* 988700daaf6c63683b48bf8b7318a857a929305a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2755447236
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
}, {
"hash" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444",
"triggerID" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"triggerType" : "PUSH"
}, {
"hash" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4446",
"triggerID" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"triggerType" : "PUSH"
}, {
"hash" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5b8a3ce8778a161aadc7a057d39b42279e50604 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4446)
* 910418360e4be2e9edf3464e1afc915516eb85ae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2758607256
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
}, {
"hash" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444",
"triggerID" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"triggerType" : "PUSH"
}, {
"hash" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4446",
"triggerID" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"triggerType" : "PUSH"
}, {
"hash" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4449",
"triggerID" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"triggerType" : "PUSH"
}, {
"hash" : "988700daaf6c63683b48bf8b7318a857a929305a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4493",
"triggerID" : "988700daaf6c63683b48bf8b7318a857a929305a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 988700daaf6c63683b48bf8b7318a857a929305a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4493)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2014807324
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/BaseHoodieLogRecordReader.java:
##########
@@ -136,16 +136,14 @@ public abstract class BaseHoodieLogRecordReader<T> {
// Use scanV2 method.
private final boolean enableOptimizedLogBlocksScan;
protected HoodieFileGroupRecordBuffer<T> recordBuffer;
+ // Allows to consider inflight instants while merging log records
+ protected boolean allowInflightInstants;
- protected BaseHoodieLogRecordReader(HoodieReaderContext readerContext,
- HoodieStorage storage,
- List<String> logFilePaths,
+ protected BaseHoodieLogRecordReader(HoodieReaderContext readerContext, HoodieStorage storage, List<String> logFilePaths,
boolean reverseReader, int bufferSize, Option<InstantRange> instantRange,
- boolean withOperationField, boolean forceFullScan,
- Option<String> partitionNameOverride,
- Option<String> keyFieldOverride,
- boolean enableOptimizedLogBlocksScan,
- HoodieFileGroupRecordBuffer<T> recordBuffer) {
+ boolean withOperationField, boolean forceFullScan, Option<String> partitionNameOverride,
+ Option<String> keyFieldOverride, boolean enableOptimizedLogBlocksScan, HoodieFileGroupRecordBuffer<T> recordBuffer,
+ boolean allowInflightInstants) {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2742727199
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9bb24320873e0e814a8b152b541ea091d564981 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2742525920
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9bb24320873e0e814a8b152b541ea091d564981 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2014804414
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderOnSpark.scala:
##########
@@ -176,4 +184,25 @@ class TestHoodieFileGroupReaderOnSpark extends TestHoodieFileGroupReaderBase[Int
assertEquals(expectedOrderingValue,
metadataMap.get(HoodieReaderContext.INTERNAL_META_ORDERING_FIELD))
}
+
+ @ParameterizedTest
+ @EnumSource(classOf[RecordMergeMode])
+ @throws[Exception]
+ def testReadFileGroupInflightData(recordMergeMode: RecordMergeMode): Unit = {
+ val writeConfigs = new util.HashMap[String, String](getCommonConfigs(recordMergeMode))
+ writeConfigs.put(DataSourceWriteOptions.TABLE_TYPE.key(), DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL)
+ try {
+ val dataGen = new HoodieTestDataGenerator(0xDEEF)
+ try {
+ // One commit; reading one file group containing a base file only
+ commitToTable(dataGen.generateInserts("001", 100), INSERT.value, writeConfigs)
+ validateOutputFromFileGroupReader(getStorageConf, getBasePath, dataGen.getPartitionPaths, true, 0, recordMergeMode)
+
+ commitToTable(dataGen.generateUniqueUpdates("003", 100), UPSERT.value, writeConfigs)
Review Comment:
I have added a UT now. This test has been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2755443354
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
}, {
"hash" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444",
"triggerID" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"triggerType" : "PUSH"
}, {
"hash" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4446",
"triggerID" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5b8a3ce8778a161aadc7a057d39b42279e50604 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2755434207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
}, {
"hash" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444",
"triggerID" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398)
* 3f73cefed6aa7459e1413326ebbaf328fd64c1cb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2771255199
I'm fine with the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2771213398
@danny0405 : are you good w/ the patch. If yes, can we go ahead and land the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2742528053
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9bb24320873e0e814a8b152b541ea091d564981 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2752159769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9bb24320873e0e814a8b152b541ea091d564981 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303)
* dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2752155954
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9bb24320873e0e814a8b152b541ea091d564981 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303)
* dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2016645204
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/BaseHoodieLogRecordReader.java:
##########
@@ -136,16 +136,14 @@ public abstract class BaseHoodieLogRecordReader<T> {
// Use scanV2 method.
private final boolean enableOptimizedLogBlocksScan;
protected FileGroupRecordBuffer<T> recordBuffer;
+ // Allows to consider inflight instants while merging log records
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2771219208

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2017786155
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -82,20 +82,17 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private final Option<UnaryOperator<T>> outputConverter;
private final HoodieReadStats readStats;
+ // Allows to consider inflight instants while merging log records using HoodieMergedLogRecordReader
+ // The inflight instants need to be considered while updating RLI records. RLI needs to fetch the revived
+ // and deleted keys from the log files written as part of active data commit. During the RLI update,
+ // the allowInflightInstants flag would need to be set to true. This would ensure the HoodieMergedLogRecordReader
Review Comment:
So this flag `allowInflightInstants` can be eliminated after the DAG write of DT and MDT? cc @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2755439149
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
}, {
"hash" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444",
"triggerID" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"triggerType" : "PUSH"
}, {
"hash" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398)
* 3f73cefed6aa7459e1413326ebbaf328fd64c1cb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444)
* c5b8a3ce8778a161aadc7a057d39b42279e50604 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13010:
URL: https://github.com/apache/hudi/pull/13010#discussion_r2014680390
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/BaseHoodieLogRecordReader.java:
##########
@@ -136,16 +136,14 @@ public abstract class BaseHoodieLogRecordReader<T> {
// Use scanV2 method.
private final boolean enableOptimizedLogBlocksScan;
protected HoodieFileGroupRecordBuffer<T> recordBuffer;
+ // Allows to consider inflight instants while merging log records
+ protected boolean allowInflightInstants;
- protected BaseHoodieLogRecordReader(HoodieReaderContext readerContext,
- HoodieStorage storage,
- List<String> logFilePaths,
+ protected BaseHoodieLogRecordReader(HoodieReaderContext readerContext, HoodieStorage storage, List<String> logFilePaths,
boolean reverseReader, int bufferSize, Option<InstantRange> instantRange,
- boolean withOperationField, boolean forceFullScan,
- Option<String> partitionNameOverride,
- Option<String> keyFieldOverride,
- boolean enableOptimizedLogBlocksScan,
- HoodieFileGroupRecordBuffer<T> recordBuffer) {
+ boolean withOperationField, boolean forceFullScan, Option<String> partitionNameOverride,
+ Option<String> keyFieldOverride, boolean enableOptimizedLogBlocksScan, HoodieFileGroupRecordBuffer<T> recordBuffer,
+ boolean allowInflightInstants) {
Review Comment:
Could you add javadocs on why `allowInflightInstants` is needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9206] Support reading inflight instants with HoodieLogRecordReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13010:
URL: https://github.com/apache/hudi/pull/13010#issuecomment-2755591445
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9bb24320873e0e814a8b152b541ea091d564981",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4303",
"triggerID" : "c9bb24320873e0e814a8b152b541ea091d564981",
"triggerType" : "PUSH"
}, {
"hash" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4398",
"triggerID" : "dbe54f1ba74ceb1a6d88a66fa3588898df0d29cb",
"triggerType" : "PUSH"
}, {
"hash" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4444",
"triggerID" : "3f73cefed6aa7459e1413326ebbaf328fd64c1cb",
"triggerType" : "PUSH"
}, {
"hash" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4446",
"triggerID" : "c5b8a3ce8778a161aadc7a057d39b42279e50604",
"triggerType" : "PUSH"
}, {
"hash" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4449",
"triggerID" : "910418360e4be2e9edf3464e1afc915516eb85ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5b8a3ce8778a161aadc7a057d39b42279e50604 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4446)
* 910418360e4be2e9edf3464e1afc915516eb85ae Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org