You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "minihippo (via GitHub)" <gi...@apache.org> on 2023/01/21 01:08:13 UTC
[GitHub] [hudi] minihippo opened a new pull request, #7722: [HUDI-5594] Add metaserver bundle validation
minihippo opened a new pull request, #7722:
URL: https://github.com/apache/hudi/pull/7722
### Change Logs
Add metaserver bundle validation
### Impact
check the metaserver bundle jar
### Risk level (write none, low medium or high below)
medium
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1399519966
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
}, {
"hash" : "0df611272a21db19d53a175e113abb3469237273",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542",
"triggerID" : "0df611272a21db19d53a175e113abb3469237273",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* 93d2446c2260a13c0247f5a86031e068e047cef2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522)
* 0df611272a21db19d53a175e113abb3469237273 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1399496803
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
}, {
"hash" : "0df611272a21db19d53a175e113abb3469237273",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0df611272a21db19d53a175e113abb3469237273",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* 93d2446c2260a13c0247f5a86031e068e047cef2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522)
* 0df611272a21db19d53a175e113abb3469237273 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403200683
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
}, {
"hash" : "0df611272a21db19d53a175e113abb3469237273",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542",
"triggerID" : "0df611272a21db19d53a175e113abb3469237273",
"triggerType" : "PUSH"
}, {
"hash" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"triggerType" : "PUSH"
}, {
"hash" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14620",
"triggerID" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"triggerType" : "PUSH"
}, {
"hash" : "9ac73126f172db765466530c28ccb2d0b0ce0f19",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14623",
"triggerID" : "9ac73126f172db765466530c28ccb2d0b0ce0f19",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* dd6c9d07b227a39313c2298ded7cc1a0e5936ed4 UNKNOWN
* 81b030b39bf437e85a313714121b9dba692b6f3f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14620)
* 9ac73126f172db765466530c28ccb2d0b0ce0f19 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14623)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1086260267
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -752,17 +749,23 @@ public Builder setLayoutVersion(Option<TimelineLayoutVersion> layoutVersion) {
return this;
}
- public Builder setProperties(Properties properties) {
- this.props = properties;
+ public Builder setMetaserverConfig(Properties props) {
+ this.metaserverConfig = new HoodieMetaserverConfig.Builder().fromProperties(props).build();
Review Comment:
is it because `props` only introduced for metaserver client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1085452892
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -110,7 +110,8 @@ class DefaultSource extends RelationProvider
}
log.info("Obtained hudi table path: " + tablePath)
- val metaClient = HoodieTableMetaClient.builder().setConf(fs.getConf).setBasePath(tablePath).build()
+ val metaClient = HoodieTableMetaClient.builder().setProperties(optParams.asJava)
+ .setConf(fs.getConf).setBasePath(tablePath).build()
Review Comment:
@minihippo this change is too broad. this should be fixed in `org.apache.hudi.DataSourceReadOptions` to target just for metaserver enabling config. `optParams` is already processed in the lines above into `parameters`, so we should follow it there.
##########
hudi-platform-service/hudi-metaserver/hudi-metaserver-server/src/main/java/org/apache/hudi/metaserver/store/RelationalDBBasedStorage.java:
##########
@@ -105,8 +106,7 @@ public Long getTableId(String db, String tb) throws MetaserverStorageException {
@Override
public String createNewTimestamp(long tableId) throws MetaserverStorageException {
- // todo: support SSS
- SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMddHHmmss");
+ SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMddHHmmss:SSS");
Review Comment:
can we reuse `org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator#MILLIS_INSTANT_TIME_FORMATTER` here? so to make formats aligned
##########
packaging/bundle-validation/validate.sh:
##########
@@ -185,6 +186,42 @@ test_kafka_connect_bundle() {
kill $ZOOKEEPER_PID $KAFKA_SERVER_PID $SCHEMA_REG_PID
}
+##
+# Function to test the hudi metaserver bundles.
+#
+# env vars (defined in container):
+# SPARK_HOME: path to the spark directory
+##
+test_hudi_metaserver_bundles () {
+ echo "::warning::validate.sh setting up hudi metaserver bundles validation"
+
+ echo "::warning::validate.sh Start hudi metaserver"
+ java -jar $JARS_DIR/metaserver.jar & local METASEVER=$!
+
+ echo "::warning::validate.sh Start hive server"
+ $DERBY_HOME/bin/startNetworkServer -h 0.0.0.0 &
+ local DERBY_PID=$!
+ $HIVE_HOME/bin/hiveserver2 --hiveconf hive.aux.jars.path=$JARS_DIR/hadoop-mr.jar &
+ local HIVE_PID=$!
Review Comment:
ok we can fix this later to decouple hive from this testcase's setup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1399567926
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
}, {
"hash" : "0df611272a21db19d53a175e113abb3469237273",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542",
"triggerID" : "0df611272a21db19d53a175e113abb3469237273",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* 0df611272a21db19d53a175e113abb3469237273 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403194321
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
}, {
"hash" : "0df611272a21db19d53a175e113abb3469237273",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542",
"triggerID" : "0df611272a21db19d53a175e113abb3469237273",
"triggerType" : "PUSH"
}, {
"hash" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"triggerType" : "PUSH"
}, {
"hash" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14620",
"triggerID" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"triggerType" : "PUSH"
}, {
"hash" : "9ac73126f172db765466530c28ccb2d0b0ce0f19",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ac73126f172db765466530c28ccb2d0b0ce0f19",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* dd6c9d07b227a39313c2298ded7cc1a0e5936ed4 UNKNOWN
* 81b030b39bf437e85a313714121b9dba692b6f3f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14620)
* 9ac73126f172db765466530c28ccb2d0b0ce0f19 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1086238927
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -752,17 +749,23 @@ public Builder setLayoutVersion(Option<TimelineLayoutVersion> layoutVersion) {
return this;
}
- public Builder setProperties(Properties properties) {
- this.props = properties;
+ public Builder setMetaserverConfig(Properties props) {
+ this.metaserverConfig = new HoodieMetaserverConfig.Builder().fromProperties(props).build();
Review Comment:
you just need a new method to take in metaserver config, why removing existing API `setProperties()` ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -752,17 +749,23 @@ public Builder setLayoutVersion(Option<TimelineLayoutVersion> layoutVersion) {
return this;
}
- public Builder setProperties(Properties properties) {
- this.props = properties;
+ public Builder setMetaserverConfig(Properties props) {
+ this.metaserverConfig = new HoodieMetaserverConfig.Builder().fromProperties(props).build();
Review Comment:
is it because `props` only introduced for metaserver client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] minihippo commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "minihippo (via GitHub)" <gi...@apache.org>.
minihippo commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1083456709
##########
hudi-platform-service/hudi-metaserver/hudi-metaserver-server/src/main/java/org/apache/hudi/metaserver/HoodieMetaserver.java:
##########
@@ -49,7 +49,7 @@ public class HoodieMetaserver {
private static TServer server;
private static Thread serverThread;
private static volatile MetaserverStorage metaserverStorage;
- private static HoodieMetaserverService metaserverService;
+ private static HoodieMetaserverGateway metaserverService;
Review Comment:
fix all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403833483
this commit passed. the last commit has no impact. we should be able to land
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1086266100
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -680,15 +680,12 @@ public void initializeBootstrapDirsIfNotExists() throws IOException {
private static HoodieTableMetaClient newMetaClient(Configuration conf, String basePath, boolean loadActiveTimelineOnLoad,
ConsistencyGuardConfig consistencyGuardConfig, Option<TimelineLayoutVersion> layoutVersion,
- String payloadClassName, String recordMergerStrategy, FileSystemRetryConfig fileSystemRetryConfig, Properties props) {
- HoodieMetaserverConfig metaserverConfig = null == props
- ? new HoodieMetaserverConfig.Builder().build()
- : new HoodieMetaserverConfig.Builder().fromProperties(props).build();
+ String payloadClassName, String recordMergerStrategy, FileSystemRetryConfig fileSystemRetryConfig, HoodieMetaserverConfig metaserverConfig) {
return metaserverConfig.isMetaserverEnabled()
? (HoodieTableMetaClient) ReflectionUtils.loadClass("org.apache.hudi.common.table.HoodieTableMetaserverClient",
- new Class<?>[] {Configuration.class, ConsistencyGuardConfig.class, String.class, FileSystemRetryConfig.class, String.class, String.class, HoodieMetaserverConfig.class},
- conf, consistencyGuardConfig, recordMergerStrategy, fileSystemRetryConfig,
- props.getProperty(HoodieTableConfig.DATABASE_NAME.key()), props.getProperty(HoodieTableConfig.NAME.key()), metaserverConfig)
+ new Class<?>[] {Configuration.class, String.class, ConsistencyGuardConfig.class, String.class, FileSystemRetryConfig.class,
+ String.class, String.class, HoodieMetaserverConfig.class}, conf, basePath, consistencyGuardConfig, recordMergerStrategy, fileSystemRetryConfig,
+ metaserverConfig.getDatabaseName(), metaserverConfig.getTableName(), metaserverConfig)
Review Comment:
ok this is a separate issue with `HoodieTableMetaserverClient`, we can ignore for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] minihippo commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "minihippo (via GitHub)" <gi...@apache.org>.
minihippo commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1087382863
##########
packaging/bundle-validation/validate.sh:
##########
@@ -185,6 +186,42 @@ test_kafka_connect_bundle() {
kill $ZOOKEEPER_PID $KAFKA_SERVER_PID $SCHEMA_REG_PID
}
+##
+# Function to test the hudi metaserver bundles.
+#
+# env vars (defined in container):
+# SPARK_HOME: path to the spark directory
+##
+test_hudi_metaserver_bundles () {
+ echo "::warning::validate.sh setting up hudi metaserver bundles validation"
+
+ echo "::warning::validate.sh Start hudi metaserver"
+ java -jar $JARS_DIR/metaserver.jar & local METASEVER=$!
+
+ echo "::warning::validate.sh Start hive server"
+ $DERBY_HOME/bin/startNetworkServer -h 0.0.0.0 &
+ local DERBY_PID=$!
+ $HIVE_HOME/bin/hiveserver2 --hiveconf hive.aux.jars.path=$JARS_DIR/hadoop-mr.jar &
+ local HIVE_PID=$!
Review Comment:
https://issues.apache.org/jira/browse/HUDI-5621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] minihippo commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "minihippo (via GitHub)" <gi...@apache.org>.
minihippo commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1083456709
##########
hudi-platform-service/hudi-metaserver/hudi-metaserver-server/src/main/java/org/apache/hudi/metaserver/HoodieMetaserver.java:
##########
@@ -49,7 +49,7 @@ public class HoodieMetaserver {
private static TServer server;
private static Thread serverThread;
private static volatile MetaserverStorage metaserverStorage;
- private static HoodieMetaserverService metaserverService;
+ private static HoodieMetaserverGateway metaserverService;
Review Comment:
fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1086228641
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java:
##########
@@ -44,7 +44,7 @@ public class HoodieInstantTimeGenerator {
public static final int MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH = MILLIS_INSTANT_TIMESTAMP_FORMAT.length();
// Formatter to generate Instant timestamps
// Unfortunately millisecond format is not parsable as is https://bugs.openjdk.java.net/browse/JDK-8031085. hence have to do appendValue()
- private static DateTimeFormatter MILLIS_INSTANT_TIME_FORMATTER = new DateTimeFormatterBuilder().appendPattern(SECS_INSTANT_TIMESTAMP_FORMAT)
+ public static DateTimeFormatter MILLIS_INSTANT_TIME_FORMATTER = new DateTimeFormatterBuilder().appendPattern(SECS_INSTANT_TIMESTAMP_FORMAT)
Review Comment:
this should be reverted
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -752,17 +749,23 @@ public Builder setLayoutVersion(Option<TimelineLayoutVersion> layoutVersion) {
return this;
}
- public Builder setProperties(Properties properties) {
- this.props = properties;
+ public Builder setMetaserverConfig(Properties props) {
+ this.metaserverConfig = new HoodieMetaserverConfig.Builder().fromProperties(props).build();
Review Comment:
you just need a new method to take in metaserver config, why removing existing API `setProperties()` ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -680,15 +680,12 @@ public void initializeBootstrapDirsIfNotExists() throws IOException {
private static HoodieTableMetaClient newMetaClient(Configuration conf, String basePath, boolean loadActiveTimelineOnLoad,
ConsistencyGuardConfig consistencyGuardConfig, Option<TimelineLayoutVersion> layoutVersion,
- String payloadClassName, String recordMergerStrategy, FileSystemRetryConfig fileSystemRetryConfig, Properties props) {
- HoodieMetaserverConfig metaserverConfig = null == props
- ? new HoodieMetaserverConfig.Builder().build()
- : new HoodieMetaserverConfig.Builder().fromProperties(props).build();
+ String payloadClassName, String recordMergerStrategy, FileSystemRetryConfig fileSystemRetryConfig, HoodieMetaserverConfig metaserverConfig) {
return metaserverConfig.isMetaserverEnabled()
? (HoodieTableMetaClient) ReflectionUtils.loadClass("org.apache.hudi.common.table.HoodieTableMetaserverClient",
- new Class<?>[] {Configuration.class, ConsistencyGuardConfig.class, String.class, FileSystemRetryConfig.class, String.class, String.class, HoodieMetaserverConfig.class},
- conf, consistencyGuardConfig, recordMergerStrategy, fileSystemRetryConfig,
- props.getProperty(HoodieTableConfig.DATABASE_NAME.key()), props.getProperty(HoodieTableConfig.NAME.key()), metaserverConfig)
+ new Class<?>[] {Configuration.class, String.class, ConsistencyGuardConfig.class, String.class, FileSystemRetryConfig.class,
+ String.class, String.class, HoodieMetaserverConfig.class}, conf, basePath, consistencyGuardConfig, recordMergerStrategy, fileSystemRetryConfig,
+ metaserverConfig.getDatabaseName(), metaserverConfig.getTableName(), metaserverConfig)
Review Comment:
you already passed `HoodieMetaserverConfig`, then no need to pass database name and table name separately. pls minimize number of args
##########
packaging/bundle-validation/validate.sh:
##########
@@ -185,6 +186,42 @@ test_kafka_connect_bundle() {
kill $ZOOKEEPER_PID $KAFKA_SERVER_PID $SCHEMA_REG_PID
}
+##
+# Function to test the hudi metaserver bundles.
+#
+# env vars (defined in container):
+# SPARK_HOME: path to the spark directory
+##
+test_hudi_metaserver_bundles () {
+ echo "::warning::validate.sh setting up hudi metaserver bundles validation"
+
+ echo "::warning::validate.sh Start hudi metaserver"
+ java -jar $JARS_DIR/metaserver.jar & local METASEVER=$!
+
+ echo "::warning::validate.sh Start hive server"
+ $DERBY_HOME/bin/startNetworkServer -h 0.0.0.0 &
+ local DERBY_PID=$!
+ $HIVE_HOME/bin/hiveserver2 --hiveconf hive.aux.jars.path=$JARS_DIR/hadoop-mr.jar &
+ local HIVE_PID=$!
Review Comment:
@minihippo pls file jira so we don't forget
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1404016203
last commit passed after rerun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403079994
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
}, {
"hash" : "0df611272a21db19d53a175e113abb3469237273",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542",
"triggerID" : "0df611272a21db19d53a175e113abb3469237273",
"triggerType" : "PUSH"
}, {
"hash" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"triggerType" : "PUSH"
}, {
"hash" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* 0df611272a21db19d53a175e113abb3469237273 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542)
* dd6c9d07b227a39313c2298ded7cc1a0e5936ed4 UNKNOWN
* 81b030b39bf437e85a313714121b9dba692b6f3f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1083244062
##########
hudi-platform-service/hudi-metaserver/hudi-metaserver-server/src/main/java/org/apache/hudi/metaserver/service/HoodieMetaserverProxyHandler.java:
##########
@@ -34,9 +34,9 @@
public class HoodieMetaserverProxyHandler implements InvocationHandler {
private static final Logger LOG = LogManager.getLogger(HoodieMetaserverProxyHandler.class);
- private final HoodieMetaserverService metaserverService;
+ private final HoodieMetaserverGateway metaserverService;
Review Comment:
/nit var name also
##########
packaging/bundle-validation/validate.sh:
##########
@@ -185,6 +186,42 @@ test_kafka_connect_bundle() {
kill $ZOOKEEPER_PID $KAFKA_SERVER_PID $SCHEMA_REG_PID
}
+##
+# Function to test the hudi metaserver bundles.
+#
+# env vars (defined in container):
+# SPARK_HOME: path to the spark directory
+##
+test_hudi_metaserver_bundles () {
+ echo "::warning::validate.sh setting up hudi metaserver bundles validation"
+
+ echo "::warning::validate.sh Start hudi metaserver"
+ java -jar $JARS_DIR/metaserver.jar & local METASEVER=$!
+
+ echo "::warning::validate.sh Start hive server"
+ $DERBY_HOME/bin/startNetworkServer -h 0.0.0.0 &
+ local DERBY_PID=$!
+ $HIVE_HOME/bin/hiveserver2 --hiveconf hive.aux.jars.path=$JARS_DIR/hadoop-mr.jar &
+ local HIVE_PID=$!
Review Comment:
we don't need hive server here as we don't need to run hiveql to validate
##########
hudi-platform-service/hudi-metaserver/hudi-metaserver-server/src/main/java/org/apache/hudi/metaserver/HoodieMetaserver.java:
##########
@@ -49,7 +49,7 @@ public class HoodieMetaserver {
private static TServer server;
private static Thread serverThread;
private static volatile MetaserverStorage metaserverStorage;
- private static HoodieMetaserverService metaserverService;
+ private static HoodieMetaserverGateway metaserverService;
Review Comment:
/nit var name should also update
##########
packaging/hudi-metaserver-server-bundle/pom.xml:
##########
@@ -6,20 +6,66 @@
<artifactId>hudi</artifactId>
<groupId>org.apache.hudi</groupId>
<version>0.13.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>hudi-metaserver-server-bundle</artifactId>
<properties>
<main.basedir>${project.parent.basedir}</main.basedir>
+ <h2.version>1.4.200</h2.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-metaserver-server</artifactId>
<version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.h2database</groupId>
+ <artifactId>h2</artifactId>
+ <version>${h2.version}</version>
+ <scope>compile</scope>
+ </dependency>
+ <!-- Logging -->
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-api</artifactId>
+ <version>${log4j2.version}</version>
Review Comment:
so the logging issue resolved? care to share the findings?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -110,7 +110,8 @@ class DefaultSource extends RelationProvider
}
log.info("Obtained hudi table path: " + tablePath)
- val metaClient = HoodieTableMetaClient.builder().setConf(fs.getConf).setBasePath(tablePath).build()
+ val metaClient = HoodieTableMetaClient.builder().setProperties(optParams.asJava)
+ .setConf(fs.getConf).setBasePath(tablePath).build()
Review Comment:
what issue are we fixing with this? this should be separated out as another PR. not mixing it with metaserver bundle testing setup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403186798
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
}, {
"hash" : "0df611272a21db19d53a175e113abb3469237273",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542",
"triggerID" : "0df611272a21db19d53a175e113abb3469237273",
"triggerType" : "PUSH"
}, {
"hash" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"triggerType" : "PUSH"
}, {
"hash" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14620",
"triggerID" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* dd6c9d07b227a39313c2298ded7cc1a0e5936ed4 UNKNOWN
* 81b030b39bf437e85a313714121b9dba692b6f3f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14620)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403034764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
}, {
"hash" : "0df611272a21db19d53a175e113abb3469237273",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542",
"triggerID" : "0df611272a21db19d53a175e113abb3469237273",
"triggerType" : "PUSH"
}, {
"hash" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* 0df611272a21db19d53a175e113abb3469237273 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542)
* dd6c9d07b227a39313c2298ded7cc1a0e5936ed4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] minihippo commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "minihippo (via GitHub)" <gi...@apache.org>.
minihippo commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1085474939
##########
packaging/bundle-validation/validate.sh:
##########
@@ -185,6 +186,42 @@ test_kafka_connect_bundle() {
kill $ZOOKEEPER_PID $KAFKA_SERVER_PID $SCHEMA_REG_PID
}
+##
+# Function to test the hudi metaserver bundles.
+#
+# env vars (defined in container):
+# SPARK_HOME: path to the spark directory
+##
+test_hudi_metaserver_bundles () {
+ echo "::warning::validate.sh setting up hudi metaserver bundles validation"
+
+ echo "::warning::validate.sh Start hudi metaserver"
+ java -jar $JARS_DIR/metaserver.jar & local METASEVER=$!
+
+ echo "::warning::validate.sh Start hive server"
+ $DERBY_HOME/bin/startNetworkServer -h 0.0.0.0 &
+ local DERBY_PID=$!
+ $HIVE_HOME/bin/hiveserver2 --hiveconf hive.aux.jars.path=$JARS_DIR/hadoop-mr.jar &
+ local HIVE_PID=$!
Review Comment:
+1. // todo: decouple hive-site.xml from spark-deploy's conf dir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] minihippo commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "minihippo (via GitHub)" <gi...@apache.org>.
minihippo commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1085475609
##########
hudi-platform-service/hudi-metaserver/hudi-metaserver-server/src/main/java/org/apache/hudi/metaserver/store/RelationalDBBasedStorage.java:
##########
@@ -105,8 +106,7 @@ public Long getTableId(String db, String tb) throws MetaserverStorageException {
@Override
public String createNewTimestamp(long tableId) throws MetaserverStorageException {
- // todo: support SSS
- SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMddHHmmss");
+ SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMddHHmmss:SSS");
Review Comment:
miss this, will fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1399155591
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* 93d2446c2260a13c0247f5a86031e068e047cef2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1399153802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* 93d2446c2260a13c0247f5a86031e068e047cef2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] minihippo commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "minihippo (via GitHub)" <gi...@apache.org>.
minihippo commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1085473683
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -110,7 +110,8 @@ class DefaultSource extends RelationProvider
}
log.info("Obtained hudi table path: " + tablePath)
- val metaClient = HoodieTableMetaClient.builder().setConf(fs.getConf).setBasePath(tablePath).build()
+ val metaClient = HoodieTableMetaClient.builder().setProperties(optParams.asJava)
+ .setConf(fs.getConf).setBasePath(tablePath).build()
Review Comment:
ok, I submit a pre-pr to fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #7722:
URL: https://github.com/apache/hudi/pull/7722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1399116610
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1399169088
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* 93d2446c2260a13c0247f5a86031e068e047cef2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] minihippo commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "minihippo (via GitHub)" <gi...@apache.org>.
minihippo commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1083457911
##########
packaging/bundle-validation/validate.sh:
##########
@@ -185,6 +186,42 @@ test_kafka_connect_bundle() {
kill $ZOOKEEPER_PID $KAFKA_SERVER_PID $SCHEMA_REG_PID
}
+##
+# Function to test the hudi metaserver bundles.
+#
+# env vars (defined in container):
+# SPARK_HOME: path to the spark directory
+##
+test_hudi_metaserver_bundles () {
+ echo "::warning::validate.sh setting up hudi metaserver bundles validation"
+
+ echo "::warning::validate.sh Start hudi metaserver"
+ java -jar $JARS_DIR/metaserver.jar & local METASEVER=$!
+
+ echo "::warning::validate.sh Start hive server"
+ $DERBY_HOME/bin/startNetworkServer -h 0.0.0.0 &
+ local DERBY_PID=$!
+ $HIVE_HOME/bin/hiveserver2 --hiveconf hive.aux.jars.path=$JARS_DIR/hadoop-mr.jar &
+ local HIVE_PID=$!
Review Comment:
cause the hive-site.xml is in, so if not start hive server, spark cannot start.
##########
packaging/bundle-validation/validate.sh:
##########
@@ -185,6 +186,42 @@ test_kafka_connect_bundle() {
kill $ZOOKEEPER_PID $KAFKA_SERVER_PID $SCHEMA_REG_PID
}
+##
+# Function to test the hudi metaserver bundles.
+#
+# env vars (defined in container):
+# SPARK_HOME: path to the spark directory
+##
+test_hudi_metaserver_bundles () {
+ echo "::warning::validate.sh setting up hudi metaserver bundles validation"
+
+ echo "::warning::validate.sh Start hudi metaserver"
+ java -jar $JARS_DIR/metaserver.jar & local METASEVER=$!
+
+ echo "::warning::validate.sh Start hive server"
+ $DERBY_HOME/bin/startNetworkServer -h 0.0.0.0 &
+ local DERBY_PID=$!
+ $HIVE_HOME/bin/hiveserver2 --hiveconf hive.aux.jars.path=$JARS_DIR/hadoop-mr.jar &
+ local HIVE_PID=$!
Review Comment:
cause the hive-site.xml is in, so if not start hive server, spark cannot start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] minihippo commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "minihippo (via GitHub)" <gi...@apache.org>.
minihippo commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1083458135
##########
packaging/hudi-metaserver-server-bundle/pom.xml:
##########
@@ -6,20 +6,66 @@
<artifactId>hudi</artifactId>
<groupId>org.apache.hudi</groupId>
<version>0.13.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>hudi-metaserver-server-bundle</artifactId>
<properties>
<main.basedir>${project.parent.basedir}</main.basedir>
+ <h2.version>1.4.200</h2.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-metaserver-server</artifactId>
<version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.h2database</groupId>
+ <artifactId>h2</artifactId>
+ <version>${h2.version}</version>
+ <scope>compile</scope>
+ </dependency>
+ <!-- Logging -->
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-api</artifactId>
+ <version>${log4j2.version}</version>
Review Comment:
Yes. Add these log related dependencies
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] minihippo commented on a diff in pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "minihippo (via GitHub)" <gi...@apache.org>.
minihippo commented on code in PR #7722:
URL: https://github.com/apache/hudi/pull/7722#discussion_r1083457659
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -110,7 +110,8 @@ class DefaultSource extends RelationProvider
}
log.info("Obtained hudi table path: " + tablePath)
- val metaClient = HoodieTableMetaClient.builder().setConf(fs.getConf).setBasePath(tablePath).build()
+ val metaClient = HoodieTableMetaClient.builder().setProperties(optParams.asJava)
+ .setConf(fs.getConf).setBasePath(tablePath).build()
Review Comment:
cause the metaserver enabled is not stored at hoodie.properties, it have to be configured expicity when reading. But the read options aren't transferred to the HoodieMetaClient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403375589
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
}, {
"hash" : "0df611272a21db19d53a175e113abb3469237273",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542",
"triggerID" : "0df611272a21db19d53a175e113abb3469237273",
"triggerType" : "PUSH"
}, {
"hash" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"triggerType" : "PUSH"
}, {
"hash" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14620",
"triggerID" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"triggerType" : "PUSH"
}, {
"hash" : "9ac73126f172db765466530c28ccb2d0b0ce0f19",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14623",
"triggerID" : "9ac73126f172db765466530c28ccb2d0b0ce0f19",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* dd6c9d07b227a39313c2298ded7cc1a0e5936ed4 UNKNOWN
* 9ac73126f172db765466530c28ccb2d0b0ce0f19 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14623)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7722: [HUDI-5594] Add metaserver bundle validation
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7722:
URL: https://github.com/apache/hudi/pull/7722#issuecomment-1403085610
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24edb48f16dfdf496883345be1f07a27aa301c84",
"triggerType" : "PUSH"
}, {
"hash" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14522",
"triggerID" : "93d2446c2260a13c0247f5a86031e068e047cef2",
"triggerType" : "PUSH"
}, {
"hash" : "0df611272a21db19d53a175e113abb3469237273",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542",
"triggerID" : "0df611272a21db19d53a175e113abb3469237273",
"triggerType" : "PUSH"
}, {
"hash" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd6c9d07b227a39313c2298ded7cc1a0e5936ed4",
"triggerType" : "PUSH"
}, {
"hash" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14620",
"triggerID" : "81b030b39bf437e85a313714121b9dba692b6f3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24edb48f16dfdf496883345be1f07a27aa301c84 UNKNOWN
* 0df611272a21db19d53a175e113abb3469237273 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14542)
* dd6c9d07b227a39313c2298ded7cc1a0e5936ed4 UNKNOWN
* 81b030b39bf437e85a313714121b9dba692b6f3f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14620)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org