You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "flashJd (via GitHub)" <gi...@apache.org> on 2023/06/25 10:03:34 UTC
[GitHub] [hudi] flashJd opened a new pull request, #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
flashJd opened a new pull request, #9048:
URL: https://github.com/apache/hudi/pull/9048
### Change Logs
when do read_optimized read in Flink on a table that contains log files only, illegalArgumentException will be threw；Actually we should get an empty set.
### Impact
N/A
### Risk level (write none, low medium or high below)
N/A
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1606007029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94a99ae8e5281292880710555587c9ebd0f7f567",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "94a99ae8e5281292880710555587c9ebd0f7f567",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94a99ae8e5281292880710555587c9ebd0f7f567 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1606082805
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94a99ae8e5281292880710555587c9ebd0f7f567",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18093",
"triggerID" : "94a99ae8e5281292880710555587c9ebd0f7f567",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94a99ae8e5281292880710555587c9ebd0f7f567 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "flashJd (via GitHub)" <gi...@apache.org>.
flashJd commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1612904526
>
sry to reply late
## engine conflicts:
v0.12.2 when spark insert overwrite a partition after flink write the log files only bucket in this partition,
https://github.com/apache/hudi/blob/b95248e011931f4748a7a9fbb8298cbbb71bda88/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java#L160 throws, but I found it was fixed in the master
## other consideration:
If align the first create base file logic, many codes can be simplified, like:
https://github.com/apache/hudi/blob/b95248e011931f4748a7a9fbb8298cbbb71bda88/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java#L362
https://github.com/apache/hudi/blob/b95248e011931f4748a7a9fbb8298cbbb71bda88/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/CompactionExecutionHelper.java#L63
https://github.com/apache/hudi/blob/b95248e011931f4748a7a9fbb8298cbbb71bda88/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadInputFormat.java#L200
etc.
what's your opinion, looking forward to your reply
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "flashJd (via GitHub)" <gi...@apache.org>.
flashJd commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1606692365
Flink generate log files at the first data written on MOR, I'm puzzled with this as spark generate base file first, I want to know the consideration @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1607138431
You are right, for the first file of each data bucket, we can write parquet direcly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "flashJd (via GitHub)" <gi...@apache.org>.
flashJd commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1607048249
> > I'm puzzled with this as spark generate base file first
>
> Because Spark uses BloomFilter index, with wich a base file metadata is maintained for all the new inserts.
Can we align flink‘s logic to spark to generate base file on the first write, recently I use spark and flink to write to the same hudi table, some conflict occurs as the differents of the first write logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "flashJd (via GitHub)" <gi...@apache.org>.
flashJd commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1607062700
First align flink‘s logic to spark to generate base file on the first write resolve many engine conflicts and makes logic consistent,
Second, the change will not affect flink's current functionality
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1617392915
I would prefer we fix it after 0.14.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "flashJd (via GitHub)" <gi...@apache.org>.
flashJd commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1617403962
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1606025223
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94a99ae8e5281292880710555587c9ebd0f7f567",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18093",
"triggerID" : "94a99ae8e5281292880710555587c9ebd0f7f567",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94a99ae8e5281292880710555587c9ebd0f7f567 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "flashJd (via GitHub)" <gi...@apache.org>.
flashJd commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1617185826
> That's true. Actually it is even more friendly for Hive query engine too, just a little late for 0.14.0 release because I'm scared for introducing protential bug, we can make the first file slice with parquets once we have enough test cases in production for backing up the confidence.
Totally agreed, I modified the logic in my local envirment, seems tests works fine, should I submit a pr or wait for v0.14.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1607033631
The test failure is located in `hudi-utilities`, should not be caused by this change: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=18093&view=logs&j=9273dbcb-a208-5cd0-5541-8b9925cb3da0&t=55ef67e2-6f87-5746-60ad-7ddc97e935ed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1607142155
The `DeltaCommitWriteHandleFactory` can be tweaked for the purpose, I'm wondering what's the engine conflicts you are talking about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #9048:
URL: https://github.com/apache/hudi/pull/9048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1607056021
No, Flink has not supportted `BloomFilter` Index yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "flashJd (via GitHub)" <gi...@apache.org>.
flashJd commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1612907539
> The `DeltaCommitWriteHandleFactory` can be tweaked for the purpose, I'm wondering what's the engine conflicts you are talking about?
sry to reply late
## engine conflicts:
v0.12.2 when spark insert overwrite a partition after flink write the log files only bucket in this partition,
https://github.com/apache/hudi/blob/b95248e011931f4748a7a9fbb8298cbbb71bda88/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java#L160 throws, but I found it was fixed in the master
## other consideration:
If align the first create base file logic, many codes can be simplified, like:
https://github.com/apache/hudi/blob/b95248e011931f4748a7a9fbb8298cbbb71bda88/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java#L362
https://github.com/apache/hudi/blob/b95248e011931f4748a7a9fbb8298cbbb71bda88/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/CompactionExecutionHelper.java#L63
https://github.com/apache/hudi/blob/b95248e011931f4748a7a9fbb8298cbbb71bda88/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadInputFormat.java#L200
etc.
what's your opinion, looking forward to your reply
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1607031702
> I'm puzzled with this as spark generate base file first
Because Spark uses BloomFilter index, with wich a base file metadata is maintained for all the new inserts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "flashJd (via GitHub)" <gi...@apache.org>.
flashJd commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1607027978
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1614517556
That's true. Actually it is even more friendly for Hive query engine too, just a little late for 0.14.0 release because I'm scared for introducing protential bug, we can make the first file slice with parquets once we have enough test cases in production for backing up the confidence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #9048: [HUDI-6434] Fix illegalArgumentException when do read_optimized read in Flink
Posted by "flashJd (via GitHub)" <gi...@apache.org>.
flashJd commented on PR #9048:
URL: https://github.com/apache/hudi/pull/9048#issuecomment-1617185611
> That's true. Actually it is even more friendly for Hive query engine too, just a little late for 0.14.0 release because I'm scared for introducing protential bug, we can make the first file slice with parquets once we have enough test cases in production for backing up the confidence.
Totally agreed, I modified the logic in my local envirment, seems tests works fine, should I submit a pr or wait for v0.14.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org