You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/04 01:58:31 UTC
[GitHub] [hudi] dongkelun opened a new pull request #3746: [HUDI-2515] Add close when producing records failed
dongkelun opened a new pull request #3746:
URL: https://github.com/apache/hudi/pull/3746
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*Add close when producing records failed*
## Brief change log
*(for example:)*
- *Add close when producing records failed*
- *Fix typos*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on a change in pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
dongkelun commented on a change in pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#discussion_r734925672
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/util/TestParquetReaderIterator.java
##########
@@ -59,6 +59,6 @@ public void testParquetIterator() throws IOException {
assertEquals(1, iterator.next());
// no more entries to iterate on
assertFalse(iterator.hasNext());
- assertThrows(HoodieIOException.class, iterator::next, "should throw an exception since there is only 1 record");
+ assertThrows(HoodieException.class, iterator::next, "should throw an exception since there is only 1 record");
Review comment:
@yihua Hello, use the `verify ` method to verify whether the `close` method is called?
First, the HoodieIOExceptionis thrown, then caught, and then the HoodieException is thrown, so it can fix the unit test failure. But for code unification, I also changed the HoodieIOExceptionis to HoodieException . Do you think this is OK?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#discussion_r734243707
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ParquetReaderIterator.java
##########
@@ -49,8 +50,13 @@ public boolean hasNext() {
this.next = parquetReader.read();
}
return this.next != null;
- } catch (IOException io) {
- throw new HoodieIOException("unable to read next record from parquet file ", io);
+ } catch (Exception e) {
+ try {
+ close();
+ } catch (IOException io) {
+ io.printStackTrace();
+ }
Review comment:
Use `FileIOUtils.closeQuietly()`?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ParquetReaderIterator.java
##########
@@ -66,8 +72,13 @@ public T next() {
T retVal = this.next;
this.next = parquetReader.read();
return retVal;
- } catch (IOException io) {
- throw new HoodieIOException("unable to read next record from parquet file ", io);
+ } catch (Exception e) {
+ try {
+ close();
+ } catch (IOException io) {
+ io.printStackTrace();
+ }
Review comment:
Same here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#discussion_r734653163
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/util/TestParquetReaderIterator.java
##########
@@ -59,6 +59,6 @@ public void testParquetIterator() throws IOException {
assertEquals(1, iterator.next());
// no more entries to iterate on
assertFalse(iterator.hasNext());
- assertThrows(HoodieIOException.class, iterator::next, "should throw an exception since there is only 1 record");
+ assertThrows(HoodieException.class, iterator::next, "should throw an exception since there is only 1 record");
Review comment:
Should this be enough to fix the unit test failure without changing `HoodieIOException` to `HoodieException` in `ParquetReaderIterator`?
Regarding unit testing the close behavior, you can add a check here to make sure `close()` is called once on the mocked `reader`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc2c849af2a376bae8c46f10a9386963faf0ccbf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793)
* 5531fdbbb2adf2484ac3ff19f3a404df01ac27c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-957054312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
dongkelun commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-949400995
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2806",
"triggerID" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5531fdbbb2adf2484ac3ff19f3a404df01ac27c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2806)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2806",
"triggerID" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"triggerType" : "PUSH"
}, {
"hash" : "17812e35590546b3bc1eb4059de811f699df6217",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "17812e35590546b3bc1eb4059de811f699df6217",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5531fdbbb2adf2484ac3ff19f3a404df01ac27c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2806)
* 17812e35590546b3bc1eb4059de811f699df6217 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3746:
URL: https://github.com/apache/hudi/pull/3746
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ce19d7f978e39f95ac777a58e99e74af92e66fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792)
* bc2c849af2a376bae8c46f10a9386963faf0ccbf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-949332833
@dongkelun could you also check the Azure CI failure:
`[ERROR] TestParquetReaderIterator.testParquetIterator:62 should throw an exception since there is only 1 record ==> Unexpected exception type thrown ==> expected: <org.apache.hudi.exception.HoodieIOException> but was: <org.apache.hudi.exception.HoodieException>`
Besides, I'm wondering if any unit test can be added to cover the code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-957054312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3746:
URL: https://github.com/apache/hudi/pull/3746
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-957054312
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-957362404
thanks for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
dongkelun commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-949388633
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0ce19d7f978e39f95ac777a58e99e74af92e66fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792)
* bc2c849af2a376bae8c46f10a9386963faf0ccbf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* bc2c849af2a376bae8c46f10a9386963faf0ccbf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#discussion_r735750112
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/util/TestParquetReaderIterator.java
##########
@@ -59,6 +61,7 @@ public void testParquetIterator() throws IOException {
assertEquals(1, iterator.next());
// no more entries to iterate on
assertFalse(iterator.hasNext());
- assertThrows(HoodieIOException.class, iterator::next, "should throw an exception since there is only 1 record");
+ assertThrows(HoodieException.class, iterator::next, "should throw an exception since there is only 1 record");
+ verify(reader, times(1)).close();
Review comment:
Yes, this looks good.
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/util/TestParquetReaderIterator.java
##########
@@ -59,6 +59,6 @@ public void testParquetIterator() throws IOException {
assertEquals(1, iterator.next());
// no more entries to iterate on
assertFalse(iterator.hasNext());
- assertThrows(HoodieIOException.class, iterator::next, "should throw an exception since there is only 1 record");
+ assertThrows(HoodieException.class, iterator::next, "should throw an exception since there is only 1 record");
Review comment:
@nsivabalan do you think changing `HoodieIOException` to `HoodieException` is okay? My thought is that anything related I/O operations should stay with `HoodieIOException`. Not sure if there's any convention in the codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2806",
"triggerID" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"triggerType" : "PUSH"
}, {
"hash" : "17812e35590546b3bc1eb4059de811f699df6217",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3037",
"triggerID" : "17812e35590546b3bc1eb4059de811f699df6217",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 17812e35590546b3bc1eb4059de811f699df6217 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3037)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6dfd7a9da05c858c5246eebc20395c7d7ad62738 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on a change in pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
dongkelun commented on a change in pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#discussion_r734272106
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ParquetReaderIterator.java
##########
@@ -66,8 +72,13 @@ public T next() {
T retVal = this.next;
this.next = parquetReader.read();
return retVal;
- } catch (IOException io) {
- throw new HoodieIOException("unable to read next record from parquet file ", io);
+ } catch (Exception e) {
+ try {
+ close();
+ } catch (IOException io) {
+ io.printStackTrace();
+ }
Review comment:
@yihua ok, modified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
dongkelun commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-949440528
@yihua Hello, the Azure CI failure has been solved, but I don't know how to write unit test cases, but I have tested it on the server before.Do you have any ideas？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ce19d7f978e39f95ac777a58e99e74af92e66fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792)
* bc2c849af2a376bae8c46f10a9386963faf0ccbf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3746:
URL: https://github.com/apache/hudi/pull/3746
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-941904370
@yihua : Can you review this patch please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on a change in pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
dongkelun commented on a change in pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#discussion_r736550976
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ParquetReaderIterator.java
##########
@@ -49,8 +49,9 @@ public boolean hasNext() {
this.next = parquetReader.read();
}
return this.next != null;
- } catch (IOException io) {
- throw new HoodieIOException("unable to read next record from parquet file ", io);
+ } catch (Exception e) {
Review comment:
As I mentioned in [HUDI-2515](https://issues.apache.org/jira/browse/HUDI-2515),It may throw an InvalidRecordException:
```scala
o.a.h.c.u.queue.BoundedInMemoryExecutor error producing records
org.apache.parquet.io.InvalidRecordException: Parquet/Avro schema mismatch: Avro field 'TRANSFER_RESULT' not found
at org.apache.parquet.avro.AvroRecordConverter.getAvroField(AvroRecordConverter.java:225)
at org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:130)
at org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:95)
at org.apache.parquet.avro.AvroRecordMaterializer.<init>(AvroRecordMaterializer.java:33)
at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:138)
at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:183)
at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:156)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135)
at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:49)
at org.apache.hudi.common.util.queue.IteratorBasedQueueProducer.produce(IteratorBasedQueueProducer.java:45)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$0(BoundedInMemoryExecutor.java:92)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#discussion_r736719988
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ParquetReaderIterator.java
##########
@@ -49,8 +49,9 @@ public boolean hasNext() {
this.next = parquetReader.read();
}
return this.next != null;
- } catch (IOException io) {
- throw new HoodieIOException("unable to read next record from parquet file ", io);
+ } catch (Exception e) {
Review comment:
my bad. did not notice that. yeah, this looks like we have to go w/ HoodieException. LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6dfd7a9da05c858c5246eebc20395c7d7ad62738 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500)
* 0ce19d7f978e39f95ac777a58e99e74af92e66fc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#discussion_r736983188
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ParquetReaderIterator.java
##########
@@ -49,8 +49,9 @@ public boolean hasNext() {
this.next = parquetReader.read();
}
return this.next != null;
- } catch (IOException io) {
- throw new HoodieIOException("unable to read next record from parquet file ", io);
+ } catch (Exception e) {
Review comment:
Got it. Then the PR looks good and it's ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* bc2c849af2a376bae8c46f10a9386963faf0ccbf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2806",
"triggerID" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc2c849af2a376bae8c46f10a9386963faf0ccbf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793)
* 5531fdbbb2adf2484ac3ff19f3a404df01ac27c9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2806)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0ce19d7f978e39f95ac777a58e99e74af92e66fc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#discussion_r736511594
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ParquetReaderIterator.java
##########
@@ -49,8 +49,9 @@ public boolean hasNext() {
this.next = parquetReader.read();
}
return this.next != null;
- } catch (IOException io) {
- throw new HoodieIOException("unable to read next record from parquet file ", io);
+ } catch (Exception e) {
Review comment:
Not sure if we needed to change this . May I know whats the necessity to move from IOException to Exception ? I understand higher layers can catch HoodieExeception, but all classes doing I/O prefer throwing IOExceptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6dfd7a9da05c858c5246eebc20395c7d7ad62738 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6dfd7a9da05c858c5246eebc20395c7d7ad62738 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2792",
"triggerID" : "949388633",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"triggerType" : "PUSH"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2794",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "bc2c849af2a376bae8c46f10a9386963faf0ccbf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2793",
"triggerID" : "949400995",
"triggerType" : "MANUAL"
}, {
"hash" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2806",
"triggerID" : "5531fdbbb2adf2484ac3ff19f3a404df01ac27c9",
"triggerType" : "PUSH"
}, {
"hash" : "17812e35590546b3bc1eb4059de811f699df6217",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3037",
"triggerID" : "17812e35590546b3bc1eb4059de811f699df6217",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5531fdbbb2adf2484ac3ff19f3a404df01ac27c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2806)
* 17812e35590546b3bc1eb4059de811f699df6217 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3037)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on a change in pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
dongkelun commented on a change in pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#discussion_r736550976
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ParquetReaderIterator.java
##########
@@ -49,8 +49,9 @@ public boolean hasNext() {
this.next = parquetReader.read();
}
return this.next != null;
- } catch (IOException io) {
- throw new HoodieIOException("unable to read next record from parquet file ", io);
+ } catch (Exception e) {
Review comment:
As I mentioned in [HUDI-2515](https://issues.apache.org/jira/browse/HUDI-2515),It may throw an InvalidRecordException:
```scala
u.queue.BoundedInMemoryExecutor error producing records
org.apache.parquet.io.InvalidRecordException: Parquet/Avro schema mismatch: Avro field 'TRANSFER_RESULT' not found
at org.apache.parquet.avro.AvroRecordConverter.getAvroField(AvroRecordConverter.java:225)
at org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:130)
at org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:95)
at org.apache.parquet.avro.AvroRecordMaterializer.<init>(AvroRecordMaterializer.java:33)
at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:138)
at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:183)
at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:156)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135)
at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:49)
at org.apache.hudi.common.util.queue.IteratorBasedQueueProducer.produce(IteratorBasedQueueProducer.java:45)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$0(BoundedInMemoryExecutor.java:92)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-941904370
@yihua : Can you review this patch please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
dongkelun commented on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-949345143
> @dongkelun could you also check the Azure CI failure: `[ERROR] TestParquetReaderIterator.testParquetIterator:62 should throw an exception since there is only 1 record ==> Unexpected exception type thrown ==> expected: <org.apache.hudi.exception.HoodieIOException> but was: <org.apache.hudi.exception.HoodieException>` Besides, I'm wondering if any unit test can be added to cover the code change.
@yihua OK, I'll try hard
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3746: [HUDI-2515] Add close when producing records failed
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3746:
URL: https://github.com/apache/hudi/pull/3746#issuecomment-933086527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500",
"triggerID" : "6dfd7a9da05c858c5246eebc20395c7d7ad62738",
"triggerType" : "PUSH"
}, {
"hash" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788",
"triggerID" : "0ce19d7f978e39f95ac777a58e99e74af92e66fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6dfd7a9da05c858c5246eebc20395c7d7ad62738 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2500)
* 0ce19d7f978e39f95ac777a58e99e74af92e66fc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2788)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org