You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "geserdugarov (via GitHub)" <gi...@apache.org> on 2024/12/18 12:09:05 UTC
[PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
geserdugarov opened a new pull request, #12516:
URL: https://github.com/apache/hudi/pull/12516
### Change Logs
Currently, `hoodie.populate.meta.fields` is not supported in Flink. This config should be used for append mode, so this MR adds support for this case.
**Before**, master, 3cb874fd53390b4465a8a8b9acd615b1b7f362cf:
**After**:
| | Before | After |
| -- | -- | -- |
| CPU samples | 79.8K | 67.9K |
| Total time of 60M rows writing | 108 s | 93 s |
TPC-H lineitem table is used for profiling.
14% faster write in append mode for Flink when `hoodie.populate.meta.fields = false`.
The next possible optimization for this scenario is to think about do we really need Bloom filters in this use case, because it costs 16% of CPU after optimization of this MR:
### Impact
No
### Risk level (write none, low medium or high below)
Low
### Documentation Update
No need. There is no mention in "All Configurations" page that `hoodie.populate.meta.fields` is not supported for Flink.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891071761
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
`preserveHoodieMetadata` actually used in the code as an indicator to get metadata from row data or generate it by calling corresponding methods. And it actually a little bit confusing naming then. I believe that values for `preserveHoodieMetadata` could be described by this schema:
Looks like `preserveHoodieMetadata` could be true only for clustering operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1893691006
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
@danny0405
I've added `ITTestHoodieDataSource::testWriteWithoutMetaColumns`. But for proper checking, it would be great to write data by Flink, and then read it by Spark, because in Spark
```SQL
SELECT * FROM table;
```
will return all columns including those with metadata.
And it would be really useful check of engines interoperability. I've created a corresponding task HUDI-8788.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891906063
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
User sets value for `hoodie.populate.meta.fields` option, which is `true` by default. And in description for this config, "append only/immutable data" is mentioned as use case:
https://github.com/apache/hudi/blob/9da3221a79465f3326ae3ac206b08d60864ddcaa/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L261-L265
For this reason, in this MR I supported `hoodie.populate.meta.fields` in Flink only for append mode.
For quick check I use SQL queries like the following ones, which used for append mode:
```SQL
CREATE TABLE hudi_debug (
id INT,
part INT,
desc STRING,
PRIMARY KEY (id) NOT ENFORCED
)
WITH (
'connector' = 'hudi',
'path' = '...',
'table.type' = 'COPY_ON_WRITE',
'write.operation' = 'insert',
'hoodie.populate.meta.fields' = 'false'
);
```
```SQL
INSERT INTO hudi_debug VALUES
(1,100,'aaa'),
(2,200,'bbb');
```
**Expected results**: there is no exceptions during
```SQL
SELECT * FROM hudi_debug;
```
and corresponding parquet files in HDFS don't contain columns with metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12516:
URL: https://github.com/apache/hudi/pull/12516#issuecomment-2556624894
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491",
"triggerID" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"triggerType" : "PUSH"
}, {
"hash" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2536",
"triggerID" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"triggerType" : "PUSH"
}, {
"hash" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2539",
"triggerID" : "2556624099",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f6914db4fbd50ed817aa9308ea36cb8407632207 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2536) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2539)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891906063
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
User sets value for `hoodie.populate.meta.fields` option, which is `true` by default. And in description for this config, "append only/immutable data" is mentioned as use case:
https://github.com/apache/hudi/blob/9da3221a79465f3326ae3ac206b08d60864ddcaa/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L261-L265
For this reason, in this MR I supported `hoodie.populate.meta.fields` in Flink only for append mode.
Checked that everything is working for append mode by:
```SQL
CREATE TABLE hudi_debug (
id INT,
part INT,
desc STRING,
PRIMARY KEY (id) NOT ENFORCED
)
WITH (
'connector' = 'hudi',
'path' = '...',
'table.type' = 'COPY_ON_WRITE',
'write.operation' = 'insert',
'hoodie.populate.meta.fields' = 'false'
);
```
```SQL
INSERT INTO hudi_debug VALUES
(1,100,'aaa'),
(2,200,'bbb');
```
I checked parquet files, that they are not contain columns with metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1892061545
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
I've also found that we could write MOR table in upsert mode without metadata. Call stack in this case will include `HoodieAppendHandle`. But we couldn't read result MOR table by Flink later due to exception thrown during:
```SQL
SELECT * FROM hudi_debug;
```
I've created separate bug for MOR without meta columns read: HUDI-8785, will fix it in a separate MR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12516:
URL: https://github.com/apache/hudi/pull/12516
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891068828
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
Yes, I saw https://github.com/apache/hudi/pull/12404 and was confused by ticket name, which mention Flink table config `hoodie.populate.meta.fields`, and didn't find any changes in `hudi-flink-client` or `hudi-flink-datasource` that will change current behavior. My point of view I described in https://github.com/apache/hudi/pull/12404#issuecomment-2550559944
And to support my comment, I created this MR that shows the lack of support of `hoodie.populate.meta.fields` in Flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1892061545
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
I've also found that we could write MOR table in upsert mode without metadata. Call stack in this case will include `HoodieAppendHandle`. But we couldn't read result MOR table by Flink later due to exception thrown during:
```SQL
SELECT * FROM hudi_debug;
```
Also this query wouldn't return meta columns anyway, so I need to check log files internal data. I've created separate bug for MOR without meta columns read: HUDI-8785, will fix it in a separate MR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1892061545
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
I've also found that we could write MOR table in upsert mode without metadata, but we couldn't read it later by Flink. I've created separate bug for this: HUDI-8785, will fix it in a separate MR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12516:
URL: https://github.com/apache/hudi/pull/12516#issuecomment-2551159308
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f5100f26e86cc92e8a90256bf0dd33413b7b12f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12516:
URL: https://github.com/apache/hudi/pull/12516#issuecomment-2556431624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491",
"triggerID" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"triggerType" : "PUSH"
}, {
"hash" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2536",
"triggerID" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f5100f26e86cc92e8a90256bf0dd33413b7b12f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491)
* f6914db4fbd50ed817aa9308ea36cb8407632207 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2536)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12516:
URL: https://github.com/apache/hudi/pull/12516#issuecomment-2556429874
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491",
"triggerID" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"triggerType" : "PUSH"
}, {
"hash" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f5100f26e86cc92e8a90256bf0dd33413b7b12f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491)
* f6914db4fbd50ed817aa9308ea36cb8407632207 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891061879
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
Isn't the flag `preserveHoodieMetadata` already control this behavior, there is another PR raised by @usberkeley for fixing all the scenarios BTW: https://github.com/apache/hudi/pull/12404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #12516:
URL: https://github.com/apache/hudi/pull/12516#issuecomment-2556624099
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891906063
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
User sets value for `hoodie.populate.meta.fields` option, which is `true` by default. And in description for this config, "append only/immutable data" is mentioned as use case:
https://github.com/apache/hudi/blob/9da3221a79465f3326ae3ac206b08d60864ddcaa/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L261-L265
For this reason, in this MR I supported `hoodie.populate.meta.fields` in Flink only for append mode.
Checked that everything is working for append mode by:
```SQL
CREATE TABLE hudi_debug (
id INT,
part INT,
desc STRING,
PRIMARY KEY (id) NOT ENFORCED
)
WITH (
'connector' = 'hudi',
'path' = '...',
'table.type' = 'COPY_ON_WRITE',
'write.operation' = 'insert',
'hoodie.populate.meta.fields' = 'false'
);
```
```SQL
INSERT INTO hudi_debug VALUES
(1,100,'aaa'),
(2,200,'bbb');
```
```SQL
SELECT * FROM hudi_debug;
```
I got as a result expected:
```Text
id part desc
2 200 bbb
1 100 aaa
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12516:
URL: https://github.com/apache/hudi/pull/12516#issuecomment-2556758068
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491",
"triggerID" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"triggerType" : "PUSH"
}, {
"hash" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2536",
"triggerID" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"triggerType" : "PUSH"
}, {
"hash" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2539",
"triggerID" : "2556624099",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f6914db4fbd50ed817aa9308ea36cb8407632207 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2536) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2539)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891140388
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
It looks like the flag `preserveHoodieMetadata` indicates whether the source row includes the metadata fields already, for table service like clustering, this should be true by default(because clustering is just a rewrite). For regular write, the metadata fields should be generated on the fly.
Let's check in which case the option `hoodie.populate.meta.fields` could be false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891068828
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
Yes, I saw https://github.com/apache/hudi/pull/12404 and was confused by ticket name, which mention Flink table config `hoodie.populate.meta.fields`, and didn't find any changes in `hudi-flink-client` or `hudi-flink-datasource` that will change current behavior. My point of view I described in https://github.com/apache/hudi/pull/12404#issuecomment-2550559944
And to support my comment, I created this MR that shows the lack of support of hoodie.populate.meta.fields` in Flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891071761
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
`preserveHoodieMetadata` actually used in the code as an indicator to get metadata from row data or generate it by calling corresponding methods. And it actually a little bit confusing naming then. I suppose, values for `preserveHoodieMetadata` are defined by this schema:
Looks like `preserveHoodieMetadata` could be true only for clustering operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891068828
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
Yes, I saw https://github.com/apache/hudi/pull/12404 and was confused by ticket name, which mention Flink table config `hoodie.populate.meta.fields`, and didn't find any changes in `hudi-flink-client` or `hudi-flink-datasource` that will change current behavior. My point of view I described in https://github.com/apache/hudi/pull/12404#issuecomment-2550559944
And to support my comment, I've created this MR that shows the lack of support of `hoodie.populate.meta.fields` in Flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1892061545
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
I've also found that we could write MOR table in upsert mode without metadata, but we couldn't read it later by Flink. Exception is thrown during:
```SQL
SELECT * FROM hudi_debug;
```
Also this query wouldn't return meta columns anyway, so I need to check log files internal data. I've created separate bug for MOR without meta columns read: HUDI-8785, will fix it in a separate MR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #12516:
URL: https://github.com/apache/hudi/pull/12516#issuecomment-2556623857
I will restart Azure CI, but we have a problem with timeout:
https://dev.azure.com/apachehudi/hudi-oss-ci/_build/results?buildId=2536&view=results
All jobs are successful, but CI failed. I suppose due to running time of "UT spark-datasource Java Tests & DDL", 1 h 29 min 56 s, which is close to 1 h 30 min timeout. Could we increase timeout a little bit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1891906063
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
User sets value for `hoodie.populate.meta.fields` option, which is `true` by default. And in description for this config, "append only/immutable data" is mentioned as use case:
https://github.com/apache/hudi/blob/9da3221a79465f3326ae3ac206b08d60864ddcaa/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L261-L265
For this reason, in this MR I supported `hoodie.populate.meta.fields` in Flink only for append mode.
Checked that everything is working for append mode by:
```SQL
CREATE TABLE hudi_debug (
id INT,
part INT,
desc STRING,
PRIMARY KEY (id) NOT ENFORCED
)
WITH (
'connector' = 'hudi',
'path' = '...',
'table.type' = 'COPY_ON_WRITE',
'write.operation' = 'insert',
'hoodie.populate.meta.fields' = 'false'
);
```
```SQL
INSERT INTO hudi_debug VALUES
(1,100,'aaa'),
(2,200,'bbb');
```
I checked that there is no exceptions during:
```SQL
SELECT * FROM hudi_debug;
```
and corresponding parquet files in HDFS don't not contain columns with metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1892061545
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
I've also found that we could write MOR table in upsert mode without metadata, but we couldn't read it later by Flink. I've created separate bug for this, will check it: HUDI-8785
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12516:
URL: https://github.com/apache/hudi/pull/12516#discussion_r1893447482
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java:
##########
@@ -90,6 +91,7 @@ public HoodieRowDataCreateHandle(HoodieTable table, HoodieWriteConfig writeConfi
this.fileId = fileId;
this.newRecordLocation = new HoodieRecordLocation(instantTime, fileId);
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ this.skipMetadataWrite = skipMetadataWrite;
Review Comment:
Can we atleast add an integration test in `ITTestHoodieDataSource`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12516:
URL: https://github.com/apache/hudi/pull/12516#issuecomment-2556551254
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491",
"triggerID" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"triggerType" : "PUSH"
}, {
"hash" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2536",
"triggerID" : "f6914db4fbd50ed817aa9308ea36cb8407632207",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f6914db4fbd50ed817aa9308ea36cb8407632207 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2536)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12516:
URL: https://github.com/apache/hudi/pull/12516#issuecomment-2551162655
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491",
"triggerID" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f5100f26e86cc92e8a90256bf0dd33413b7b12f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8631] Support of `hoodie.populate.meta.fields` for Flink append mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12516:
URL: https://github.com/apache/hudi/pull/12516#issuecomment-2551327457
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491",
"triggerID" : "5f5100f26e86cc92e8a90256bf0dd33413b7b12f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f5100f26e86cc92e8a90256bf0dd33413b7b12f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org