You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2023/01/18 23:24:23 UTC
[GitHub] [hudi] alexeykudinkin opened a new pull request, #7702: Fixing Kryo registration to be properly wired into Spark sessions
alexeykudinkin opened a new pull request, #7702:
URL: https://github.com/apache/hudi/pull/7702
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1398012526
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b075e25aa5811f36e83e12bfba11a08bc929bf1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396436526
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 11cceab4fc10a46bb7d169079262b1e4c1ea558e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424)
* 27d54ea8f8f395f2044009d51ba8e969c74c2a4d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1398072303
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1b075e25aa5811f36e83e12bfba11a08bc929bf1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1399156860
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506",
"triggerID" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"triggerType" : "PUSH"
}, {
"hash" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14515",
"triggerID" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"triggerType" : "PUSH"
}, {
"hash" : "c372fe7110b207eb6b0136e0303c4778da10fbb5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14524",
"triggerID" : "c372fe7110b207eb6b0136e0303c4778da10fbb5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14515)
* c372fe7110b207eb6b0136e0303c4778da10fbb5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14524)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1398259582
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1b075e25aa5811f36e83e12bfba11a08bc929bf1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1399291719
CI is green:
<img width="1632" alt="Screenshot 2023-01-21 at 9 15 20 AM" src="https://user-images.githubusercontent.com/428277/213878762-e58de1c0-1aff-47b8-a618-bc39796b97ed.png">
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=14524&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1399182239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506",
"triggerID" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"triggerType" : "PUSH"
}, {
"hash" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14515",
"triggerID" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"triggerType" : "PUSH"
}, {
"hash" : "c372fe7110b207eb6b0136e0303c4778da10fbb5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14524",
"triggerID" : "c372fe7110b207eb6b0136e0303c4778da10fbb5",
"triggerType" : "PUSH"
}, {
"hash" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14487",
"triggerID" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c372fe7110b207eb6b0136e0303c4778da10fbb5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14524)
* ae1bcf3c42da3945c843864cdeac7f8cb89ef088 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14487)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1398731123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b075e25aa5811f36e83e12bfba11a08bc929bf1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491)
* 8218fde3afec8a069de63eac7dda1abf45523f12 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7702:
URL: https://github.com/apache/hudi/pull/7702#discussion_r1080809686
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieCommonKryoRegistrar.java:
##########
@@ -81,7 +83,8 @@ public Class<?>[] registerClasses() {
HoodieRecordLocation.class,
HoodieRecordGlobalLocation.class
- };
+ })
+ .forEachOrdered(kryo::register);
Review Comment:
A stateless function (function that does not take any side effect) is always a better choice especially for tool method, personally I prefer the old way we handle this.
##########
pom.xml:
##########
@@ -440,34 +441,13 @@
<!-- common to all bundles -->
<artifactSet>
<includes>
- <!-- com.esotericsoftware:kryo-shaded -->
- <include>com.esotericsoftware:kryo-shaded</include>
- <include>com.esotericsoftware:minlog</include>
- <include>org.objenesis:objenesis</include>
<!-- org.apache.httpcomponents -->
<include>org.apache.httpcomponents:httpclient</include>
<include>org.apache.httpcomponents:httpcore</include>
<include>org.apache.httpcomponents:fluent-hc</include>
</includes>
</artifactSet>
<relocations>
- <!-- com.esotericsoftware:kryo-shaded -->
- <relocation>
- <pattern>com.esotericsoftware.kryo.</pattern>
- <shadedPattern>org.apache.hudi.com.esotericsoftware.kryo.</shadedPattern>
- </relocation>
Review Comment:
What is the purpose to move the common bundle dependencies to each bundle pom files?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestUtils.java:
##########
@@ -92,10 +93,18 @@ public class HoodieClientTestUtils {
*/
public static SparkConf getSparkConfForTest(String appName) {
SparkConf sparkConf = new SparkConf().setAppName(appName)
- .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer").setMaster("local[4]")
+ .setMaster("local[4]")
+ .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .set("spark.kryo.registrator", "org.apache.spark.HoodieSparkKryoRegistrar")
+ .set("spark.sql.extensions", "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
Review Comment:
Can we also move these common options into a tool method?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/HoodieSparkSqlTestBase.scala:
##########
@@ -51,13 +52,6 @@ class HoodieSparkSqlTestBase extends FunSuite with BeforeAndAfterAll {
DateTimeZone.setDefault(DateTimeZone.UTC)
TimeZone.setDefault(DateTimeUtils.getTimeZone("UTC"))
protected lazy val spark: SparkSession = SparkSession.builder()
- .master("local[1]")
- .appName("hoodie sql test")
- .withExtensions(new HoodieSparkSessionExtension)
- .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
- .config("hoodie.insert.shuffle.parallelism", "4")
- .config("hoodie.upsert.shuffle.parallelism", "4")
- .config("hoodie.delete.shuffle.parallelism", "4")
Review Comment:
Not sure whether we can remove these parallelism options.
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/HoodieSparkKryoRegistrar.scala:
##########
@@ -42,22 +43,31 @@ import org.apache.spark.serializer.KryoRegistrator
* or renamed (w/o correspondingly updating such usages)</li>
* </ol>
*/
-class HoodieSparkKryoProvider extends HoodieCommonKryoProvider {
- override def registerClasses(): Array[Class[_]] = {
+class HoodieSparkKryoRegistrar extends HoodieCommonKryoRegistrar with KryoRegistrator {
+ override def registerClasses(kryo: Kryo): Unit = {
///////////////////////////////////////////////////////////////////////////
// NOTE: DO NOT REORDER REGISTRATIONS
///////////////////////////////////////////////////////////////////////////
- val classes = super[HoodieCommonKryoProvider].registerClasses()
- classes ++ Array(
- classOf[HoodieWriteConfig],
- classOf[HoodieSparkRecord],
- classOf[HoodieInternalRow]
- )
+ super[HoodieCommonKryoRegistrar].registerClasses(kryo)
+
+ kryo.register(classOf[HoodieWriteConfig])
+
+ kryo.register(classOf[HoodieSparkRecord])
+ kryo.register(classOf[HoodieInternalRow])
+
+ // NOTE: Hadoop's configuration is not a serializable object by itself, and hence
+ // we're relying on [[SerializableConfiguration]] wrapper to work it around
+ kryo.register(classOf[SerializableConfiguration], new JavaSerializer())
}
}
-object HoodieSparkKryoProvider {
+object HoodieSparkKryoRegistrar {
+
+ // NOTE: We're copying definition of the config introduced in Spark 3.0
+ // (to stay compatible w/ Spark 2.4)
+ private val KRYO_USER_REGISTRATORS = "spark.kryo.registrator"
+
def register(conf: SparkConf): SparkConf = {
- conf.registerKryoClasses(new HoodieSparkKryoProvider().registerClasses())
+ conf.set(KRYO_USER_REGISTRATORS, Seq(classOf[HoodieSparkKryoRegistrar].getName).mkString(","))
}
Review Comment:
Does `.mkString(",")` make sense here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396272345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b01684325958414250a21143756d98fdbcd5c034 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396333149
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b01684325958414250a21143756d98fdbcd5c034 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421)
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1398001638
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
}, {
"hash" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461",
"triggerID" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"triggerType" : "PUSH"
}, {
"hash" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14466",
"triggerID" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"triggerType" : "PUSH"
}, {
"hash" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14487",
"triggerID" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* ae1bcf3c42da3945c843864cdeac7f8cb89ef088 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14487)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1398082191
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1b075e25aa5811f36e83e12bfba11a08bc929bf1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1398896958
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506",
"triggerID" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8218fde3afec8a069de63eac7dda1abf45523f12 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin merged pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin merged PR #7702:
URL: https://github.com/apache/hudi/pull/7702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396279224
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b01684325958414250a21143756d98fdbcd5c034 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397247078
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* e95e38b33e134cbc74d4c08065fe0c2ae1418937 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397908181
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
}, {
"hash" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461",
"triggerID" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"triggerType" : "PUSH"
}, {
"hash" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14466",
"triggerID" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 45b15748e01531fb144e37f5b04b34b811ab1474 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14466)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #7702:
URL: https://github.com/apache/hudi/pull/7702#discussion_r1080866041
##########
pom.xml:
##########
@@ -440,34 +441,13 @@
<!-- common to all bundles -->
<artifactSet>
<includes>
- <!-- com.esotericsoftware:kryo-shaded -->
- <include>com.esotericsoftware:kryo-shaded</include>
- <include>com.esotericsoftware:minlog</include>
- <include>org.objenesis:objenesis</include>
<!-- org.apache.httpcomponents -->
<include>org.apache.httpcomponents:httpclient</include>
<include>org.apache.httpcomponents:httpcore</include>
<include>org.apache.httpcomponents:fluent-hc</include>
</includes>
</artifactSet>
<relocations>
- <!-- com.esotericsoftware:kryo-shaded -->
- <relocation>
- <pattern>com.esotericsoftware.kryo.</pattern>
- <shadedPattern>org.apache.hudi.com.esotericsoftware.kryo.</shadedPattern>
- </relocation>
Review Comment:
We actually move it just to bundles that will have the Kryo included (Spark and Flink won't have Kryo included)
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestUtils.java:
##########
@@ -92,10 +93,18 @@ public class HoodieClientTestUtils {
*/
public static SparkConf getSparkConfForTest(String appName) {
SparkConf sparkConf = new SparkConf().setAppName(appName)
- .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer").setMaster("local[4]")
+ .setMaster("local[4]")
+ .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .set("spark.kryo.registrator", "org.apache.spark.HoodieSparkKryoRegistrar")
+ .set("spark.sql.extensions", "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
Review Comment:
This is exactly the method you're referring to (used in tests)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieCommonKryoRegistrar.java:
##########
@@ -81,7 +83,8 @@ public Class<?>[] registerClasses() {
HoodieRecordLocation.class,
HoodieRecordGlobalLocation.class
- };
+ })
+ .forEachOrdered(kryo::register);
Review Comment:
Agree in principle, but here we actually aligning it w/ an interface of `KryoRegistrator`
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/HoodieSparkSqlTestBase.scala:
##########
@@ -51,13 +52,6 @@ class HoodieSparkSqlTestBase extends FunSuite with BeforeAndAfterAll {
DateTimeZone.setDefault(DateTimeZone.UTC)
TimeZone.setDefault(DateTimeUtils.getTimeZone("UTC"))
protected lazy val spark: SparkSession = SparkSession.builder()
- .master("local[1]")
- .appName("hoodie sql test")
- .withExtensions(new HoodieSparkSessionExtension)
- .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
- .config("hoodie.insert.shuffle.parallelism", "4")
- .config("hoodie.upsert.shuffle.parallelism", "4")
- .config("hoodie.delete.shuffle.parallelism", "4")
Review Comment:
These are not removed -- they are replaced w/ options set in `getSparkConfForTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397788798
> but this is a very broad usability change. we should have brought this up for highlighting earlier.
Agreed, not ideal, but unavoidable unfortunately -- w/o we'd be passing around ~20-30% more dead-weight data. And in some cases it would actually lead to failures as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396342986
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b01684325958414250a21143756d98fdbcd5c034 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421)
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 11cceab4fc10a46bb7d169079262b1e4c1ea558e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396432469
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b01684325958414250a21143756d98fdbcd5c034 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421)
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 11cceab4fc10a46bb7d169079262b1e4c1ea558e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424)
* 27d54ea8f8f395f2044009d51ba8e969c74c2a4d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397701234
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
}, {
"hash" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461",
"triggerID" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"triggerType" : "PUSH"
}, {
"hash" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14466",
"triggerID" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 61e621587610d74d3d3820d68d1f18ddaa4538d8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461)
* 45b15748e01531fb144e37f5b04b34b811ab1474 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14466)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397694848
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
}, {
"hash" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461",
"triggerID" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"triggerType" : "PUSH"
}, {
"hash" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 61e621587610d74d3d3820d68d1f18ddaa4538d8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461)
* 45b15748e01531fb144e37f5b04b34b811ab1474 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1398007308
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
}, {
"hash" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461",
"triggerID" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"triggerType" : "PUSH"
}, {
"hash" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14466",
"triggerID" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"triggerType" : "PUSH"
}, {
"hash" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14487",
"triggerID" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* ae1bcf3c42da3945c843864cdeac7f8cb89ef088 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14487)
* 1b075e25aa5811f36e83e12bfba11a08bc929bf1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1399220505
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506",
"triggerID" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"triggerType" : "PUSH"
}, {
"hash" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14515",
"triggerID" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"triggerType" : "PUSH"
}, {
"hash" : "c372fe7110b207eb6b0136e0303c4778da10fbb5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14524",
"triggerID" : "c372fe7110b207eb6b0136e0303c4778da10fbb5",
"triggerType" : "PUSH"
}, {
"hash" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14487",
"triggerID" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae1bcf3c42da3945c843864cdeac7f8cb89ef088 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14487)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396286012
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b01684325958414250a21143756d98fdbcd5c034 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7702:
URL: https://github.com/apache/hudi/pull/7702#discussion_r1080806942
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/HoodieSparkKryoRegistrar.scala:
##########
@@ -42,22 +43,31 @@ import org.apache.spark.serializer.KryoRegistrator
* or renamed (w/o correspondingly updating such usages)</li>
* </ol>
*/
-class HoodieSparkKryoProvider extends HoodieCommonKryoProvider {
- override def registerClasses(): Array[Class[_]] = {
+class HoodieSparkKryoRegistrar extends HoodieCommonKryoRegistrar with KryoRegistrator {
+ override def registerClasses(kryo: Kryo): Unit = {
///////////////////////////////////////////////////////////////////////////
// NOTE: DO NOT REORDER REGISTRATIONS
///////////////////////////////////////////////////////////////////////////
- val classes = super[HoodieCommonKryoProvider].registerClasses()
- classes ++ Array(
- classOf[HoodieWriteConfig],
- classOf[HoodieSparkRecord],
- classOf[HoodieInternalRow]
- )
+ super[HoodieCommonKryoRegistrar].registerClasses(kryo)
+
+ kryo.register(classOf[HoodieWriteConfig])
+
+ kryo.register(classOf[HoodieSparkRecord])
+ kryo.register(classOf[HoodieInternalRow])
+
+ // NOTE: Hadoop's configuration is not a serializable object by itself, and hence
+ // we're relying on [[SerializableConfiguration]] wrapper to work it around
+ kryo.register(classOf[SerializableConfiguration], new JavaSerializer())
}
}
-object HoodieSparkKryoProvider {
+object HoodieSparkKryoRegistrar {
+
+ // NOTE: We're copying definition of the config introduced in Spark 3.0
+ // (to stay compatible w/ Spark 2.4)
+ private val KRYO_USER_REGISTRATORS = "spark.kryo.registrator"
+
Review Comment:
Guess we can make it public so that there is no need to hard code the option key `spark.kryo.registrator` everywhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #7702:
URL: https://github.com/apache/hudi/pull/7702#discussion_r1080862515
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/HoodieSparkKryoRegistrar.scala:
##########
@@ -42,22 +43,31 @@ import org.apache.spark.serializer.KryoRegistrator
* or renamed (w/o correspondingly updating such usages)</li>
* </ol>
*/
-class HoodieSparkKryoProvider extends HoodieCommonKryoProvider {
- override def registerClasses(): Array[Class[_]] = {
+class HoodieSparkKryoRegistrar extends HoodieCommonKryoRegistrar with KryoRegistrator {
+ override def registerClasses(kryo: Kryo): Unit = {
///////////////////////////////////////////////////////////////////////////
// NOTE: DO NOT REORDER REGISTRATIONS
///////////////////////////////////////////////////////////////////////////
- val classes = super[HoodieCommonKryoProvider].registerClasses()
- classes ++ Array(
- classOf[HoodieWriteConfig],
- classOf[HoodieSparkRecord],
- classOf[HoodieInternalRow]
- )
+ super[HoodieCommonKryoRegistrar].registerClasses(kryo)
+
+ kryo.register(classOf[HoodieWriteConfig])
+
+ kryo.register(classOf[HoodieSparkRecord])
+ kryo.register(classOf[HoodieInternalRow])
+
+ // NOTE: Hadoop's configuration is not a serializable object by itself, and hence
+ // we're relying on [[SerializableConfiguration]] wrapper to work it around
+ kryo.register(classOf[SerializableConfiguration], new JavaSerializer())
}
}
-object HoodieSparkKryoProvider {
+object HoodieSparkKryoRegistrar {
+
+ // NOTE: We're copying definition of the config introduced in Spark 3.0
+ // (to stay compatible w/ Spark 2.4)
+ private val KRYO_USER_REGISTRATORS = "spark.kryo.registrator"
+
Review Comment:
We actually won't be able to use it everywhere, so i rather stuck w/ the Spark option for consistency (which is the way we handle every other option as well)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1399171989
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506",
"triggerID" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"triggerType" : "PUSH"
}, {
"hash" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14515",
"triggerID" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"triggerType" : "PUSH"
}, {
"hash" : "c372fe7110b207eb6b0136e0303c4778da10fbb5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14524",
"triggerID" : "c372fe7110b207eb6b0136e0303c4778da10fbb5",
"triggerType" : "PUSH"
}, {
"hash" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14515)
* c372fe7110b207eb6b0136e0303c4778da10fbb5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14524)
* ae1bcf3c42da3945c843864cdeac7f8cb89ef088 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397400645
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
}, {
"hash" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461",
"triggerID" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* e95e38b33e134cbc74d4c08065fe0c2ae1418937 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440)
* 61e621587610d74d3d3820d68d1f18ddaa4538d8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1399155550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506",
"triggerID" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"triggerType" : "PUSH"
}, {
"hash" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14515",
"triggerID" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"triggerType" : "PUSH"
}, {
"hash" : "c372fe7110b207eb6b0136e0303c4778da10fbb5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c372fe7110b207eb6b0136e0303c4778da10fbb5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14515)
* c372fe7110b207eb6b0136e0303c4778da10fbb5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397760535
> --conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar
but this is a very broad usability change. we should have brought this up for highlighting earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397954859
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
}, {
"hash" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461",
"triggerID" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"triggerType" : "PUSH"
}, {
"hash" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14466",
"triggerID" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"triggerType" : "PUSH"
}, {
"hash" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14487",
"triggerID" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 45b15748e01531fb144e37f5b04b34b811ab1474 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14466)
* ae1bcf3c42da3945c843864cdeac7f8cb89ef088 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14487)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1398007610
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396486493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 11cceab4fc10a46bb7d169079262b1e4c1ea558e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424)
* 27d54ea8f8f395f2044009d51ba8e969c74c2a4d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433)
* e95e38b33e134cbc74d4c08065fe0c2ae1418937 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1399013316
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506",
"triggerID" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"triggerType" : "PUSH"
}, {
"hash" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14515",
"triggerID" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8218fde3afec8a069de63eac7dda1abf45523f12 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506)
* ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14515)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1399006892
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506",
"triggerID" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"triggerType" : "PUSH"
}, {
"hash" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8218fde3afec8a069de63eac7dda1abf45523f12 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506)
* ffe3e3bd3ececa8b10fb6412ff411abe21d5ea27 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397688130
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
}, {
"hash" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461",
"triggerID" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 61e621587610d74d3d3820d68d1f18ddaa4538d8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397949920
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
}, {
"hash" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14461",
"triggerID" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"triggerType" : "PUSH"
}, {
"hash" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14466",
"triggerID" : "45b15748e01531fb144e37f5b04b34b811ab1474",
"triggerType" : "PUSH"
}, {
"hash" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae1bcf3c42da3945c843864cdeac7f8cb89ef088",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 45b15748e01531fb144e37f5b04b34b811ab1474 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14466)
* ae1bcf3c42da3945c843864cdeac7f8cb89ef088 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1398738792
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"triggerType" : "PUSH"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "1b075e25aa5811f36e83e12bfba11a08bc929bf1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491",
"triggerID" : "1398007610",
"triggerType" : "MANUAL"
}, {
"hash" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506",
"triggerID" : "8218fde3afec8a069de63eac7dda1abf45523f12",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b075e25aa5811f36e83e12bfba11a08bc929bf1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14495) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14491)
* 8218fde3afec8a069de63eac7dda1abf45523f12 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14506)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396338262
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b01684325958414250a21143756d98fdbcd5c034 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421)
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 11cceab4fc10a46bb7d169079262b1e4c1ea558e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396493142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* 27d54ea8f8f395f2044009d51ba8e969c74c2a4d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433)
* e95e38b33e134cbc74d4c08065fe0c2ae1418937 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #7702:
URL: https://github.com/apache/hudi/pull/7702#discussion_r1081601845
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/HoodieSparkKryoRegistrar.scala:
##########
@@ -42,22 +43,31 @@ import org.apache.spark.serializer.KryoRegistrator
* or renamed (w/o correspondingly updating such usages)</li>
* </ol>
*/
-class HoodieSparkKryoProvider extends HoodieCommonKryoProvider {
- override def registerClasses(): Array[Class[_]] = {
+class HoodieSparkKryoRegistrar extends HoodieCommonKryoRegistrar with KryoRegistrator {
+ override def registerClasses(kryo: Kryo): Unit = {
///////////////////////////////////////////////////////////////////////////
// NOTE: DO NOT REORDER REGISTRATIONS
///////////////////////////////////////////////////////////////////////////
- val classes = super[HoodieCommonKryoProvider].registerClasses()
- classes ++ Array(
- classOf[HoodieWriteConfig],
- classOf[HoodieSparkRecord],
- classOf[HoodieInternalRow]
- )
+ super[HoodieCommonKryoRegistrar].registerClasses(kryo)
+
+ kryo.register(classOf[HoodieWriteConfig])
+
+ kryo.register(classOf[HoodieSparkRecord])
+ kryo.register(classOf[HoodieInternalRow])
+
+ // NOTE: Hadoop's configuration is not a serializable object by itself, and hence
+ // we're relying on [[SerializableConfiguration]] wrapper to work it around
+ kryo.register(classOf[SerializableConfiguration], new JavaSerializer())
}
}
-object HoodieSparkKryoProvider {
+object HoodieSparkKryoRegistrar {
+
+ // NOTE: We're copying definition of the config introduced in Spark 3.0
+ // (to stay compatible w/ Spark 2.4)
+ private val KRYO_USER_REGISTRATORS = "spark.kryo.registrator"
+
def register(conf: SparkConf): SparkConf = {
- conf.registerKryoClasses(new HoodieSparkKryoProvider().registerClasses())
+ conf.set(KRYO_USER_REGISTRATORS, Seq(classOf[HoodieSparkKryoRegistrar].getName).mkString(","))
}
Review Comment:
We need to convert it to a string, so i kept it generic so that we can drop in one more class. Not strictly necessary though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7702:
URL: https://github.com/apache/hudi/pull/7702#issuecomment-1397389852
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b01684325958414250a21143756d98fdbcd5c034",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14421",
"triggerID" : "b01684325958414250a21143756d98fdbcd5c034",
"triggerType" : "PUSH"
}, {
"hash" : "384a9774018272e13b967817b0e48b1596a23dcc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "384a9774018272e13b967817b0e48b1596a23dcc",
"triggerType" : "PUSH"
}, {
"hash" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14424",
"triggerID" : "11cceab4fc10a46bb7d169079262b1e4c1ea558e",
"triggerType" : "PUSH"
}, {
"hash" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14433",
"triggerID" : "27d54ea8f8f395f2044009d51ba8e969c74c2a4d",
"triggerType" : "PUSH"
}, {
"hash" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440",
"triggerID" : "e95e38b33e134cbc74d4c08065fe0c2ae1418937",
"triggerType" : "PUSH"
}, {
"hash" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "61e621587610d74d3d3820d68d1f18ddaa4538d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN
* e95e38b33e134cbc74d4c08065fe0c2ae1418937 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14440)
* 61e621587610d74d3d3820d68d1f18ddaa4538d8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org