You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/03/23 19:35:06 UTC
[PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
the-other-tim-brown opened a new pull request, #13018:
URL: https://github.com/apache/hudi/pull/13018
### Change Logs
- Updates the `getInt` and `getString` calls to return the default if not set to avoid NullPointerExceptions
### Impact
- Fixes runtime issue if the config is not set for `HIVE_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD`
### Risk level (write none, low medium or high below)
None
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13018:
URL: https://github.com/apache/hudi/pull/13018#issuecomment-2746383485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "336e6f8bff9f8f98004800c2c14ce79bbca3ebc0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "336e6f8bff9f8f98004800c2c14ce79bbca3ebc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 336e6f8bff9f8f98004800c2c14ce79bbca3ebc0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #13018:
URL: https://github.com/apache/hudi/pull/13018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
Posted by "sgomezvillamor (via GitHub)" <gi...@apache.org>.
sgomezvillamor commented on code in PR #13018:
URL: https://github.com/apache/hudi/pull/13018#discussion_r2022923412
##########
hudi-sync/hudi-datahub-sync/src/test/java/org/apache/hudi/sync/datahub/TestDataHubTableProperties.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.sync.datahub;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.hadoop.HoodieParquetInputFormat;
+import org.apache.hudi.sync.datahub.config.DataHubSyncConfig;
+
+import org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat;
+import org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_TYPE;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_VERSION;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+class TestDataHubTableProperties {
+ @Test
+ void testGetTableProperties() {
+ String path = "file:///tmp/path";
+ Map<String, String> expected = new HashMap<>();
+ expected.put(HUDI_TABLE_TYPE, "MERGE_ON_READ");
+ expected.put(HUDI_TABLE_VERSION, "SIX");
Review Comment:
As requested: https://github.com/apache/hudi/pull/13067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13018:
URL: https://github.com/apache/hudi/pull/13018#issuecomment-2746472290
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "336e6f8bff9f8f98004800c2c14ce79bbca3ebc0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4340",
"triggerID" : "336e6f8bff9f8f98004800c2c14ce79bbca3ebc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 336e6f8bff9f8f98004800c2c14ce79bbca3ebc0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4340)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13018:
URL: https://github.com/apache/hudi/pull/13018#discussion_r2009245744
##########
hudi-sync/hudi-datahub-sync/src/test/java/org/apache/hudi/sync/datahub/TestDataHubTableProperties.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.sync.datahub;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.hadoop.HoodieParquetInputFormat;
+import org.apache.hudi.sync.datahub.config.DataHubSyncConfig;
+
+import org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat;
+import org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_TYPE;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_VERSION;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+class TestDataHubTableProperties {
+ @Test
+ void testGetTableProperties() {
+ String path = "file:///tmp/path";
+ Map<String, String> expected = new HashMap<>();
+ expected.put(HUDI_TABLE_TYPE, "MERGE_ON_READ");
+ expected.put(HUDI_TABLE_VERSION, "SIX");
Review Comment:
@sgomezvillamor is the intention for this to be 6 or is SIX spelled out the expected value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
Posted by "sgomezvillamor (via GitHub)" <gi...@apache.org>.
sgomezvillamor commented on code in PR #13018:
URL: https://github.com/apache/hudi/pull/13018#discussion_r2012226165
##########
hudi-sync/hudi-datahub-sync/src/test/java/org/apache/hudi/sync/datahub/TestDataHubTableProperties.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.sync.datahub;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.hadoop.HoodieParquetInputFormat;
+import org.apache.hudi.sync.datahub.config.DataHubSyncConfig;
+
+import org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat;
+import org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_TYPE;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_VERSION;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+class TestDataHubTableProperties {
+ @Test
+ void testGetTableProperties() {
+ String path = "file:///tmp/path";
+ Map<String, String> expected = new HashMap<>();
+ expected.put(HUDI_TABLE_TYPE, "MERGE_ON_READ");
+ expected.put(HUDI_TABLE_VERSION, "SIX");
Review Comment:
What I mean is in any case this is responsibility of the datahub-sync controller.
About this being a number in the `hoodie.properties`, that matches the code here
https://github.com/apache/hudi/blob/9352576357da04080a15456a01d6993cd553dec2/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L703-L707
So, there is some misalignment in Hudi and apparently this is loaded from version code and reported as string enum representation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13018:
URL: https://github.com/apache/hudi/pull/13018#issuecomment-2746383975
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "336e6f8bff9f8f98004800c2c14ce79bbca3ebc0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4340",
"triggerID" : "336e6f8bff9f8f98004800c2c14ce79bbca3ebc0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 336e6f8bff9f8f98004800c2c14ce79bbca3ebc0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4340)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
Posted by "sgomezvillamor (via GitHub)" <gi...@apache.org>.
sgomezvillamor commented on code in PR #13018:
URL: https://github.com/apache/hudi/pull/13018#discussion_r2012138004
##########
hudi-sync/hudi-datahub-sync/src/test/java/org/apache/hudi/sync/datahub/TestDataHubTableProperties.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.sync.datahub;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.hadoop.HoodieParquetInputFormat;
+import org.apache.hudi.sync.datahub.config.DataHubSyncConfig;
+
+import org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat;
+import org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_TYPE;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_VERSION;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+class TestDataHubTableProperties {
+ @Test
+ void testGetTableProperties() {
+ String path = "file:///tmp/path";
+ Map<String, String> expected = new HashMap<>();
+ expected.put(HUDI_TABLE_TYPE, "MERGE_ON_READ");
+ expected.put(HUDI_TABLE_VERSION, "SIX");
Review Comment:
HUDI_TABLE_VERSION is set in https://github.com/apache/hudi/blob/9352576357da04080a15456a01d6993cd553dec2/hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/DataHubTableProperties.java#L71, which comes from here https://github.com/apache/hudi/blob/9352576357da04080a15456a01d6993cd553dec2/hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/DataHubTableProperties.java#L126, which comes from https://github.com/apache/hudi/blob/9352576357da04080a15456a01d6993cd553dec2/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L133-L135, which comes from https://github.com/apache/hudi/blob/9352576357da04080a15456a01d6993cd553dec2/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableVersion.java#L46
So the content is managed by Hudi itself and matches the toString serialization of the SIX enum.
Hope this helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13018:
URL: https://github.com/apache/hudi/pull/13018#discussion_r2012191636
##########
hudi-sync/hudi-datahub-sync/src/test/java/org/apache/hudi/sync/datahub/TestDataHubTableProperties.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.sync.datahub;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.hadoop.HoodieParquetInputFormat;
+import org.apache.hudi.sync.datahub.config.DataHubSyncConfig;
+
+import org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat;
+import org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_TYPE;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_VERSION;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+class TestDataHubTableProperties {
+ @Test
+ void testGetTableProperties() {
+ String path = "file:///tmp/path";
+ Map<String, String> expected = new HashMap<>();
+ expected.put(HUDI_TABLE_TYPE, "MERGE_ON_READ");
+ expected.put(HUDI_TABLE_VERSION, "SIX");
Review Comment:
Thanks, I am aware of all this already. Just wanted to make sure this is intentional since it is a number in the `hoodie.properties`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9211] Fix bug with config in DataHubSyncTool [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13018:
URL: https://github.com/apache/hudi/pull/13018#discussion_r2012787275
##########
hudi-sync/hudi-datahub-sync/src/test/java/org/apache/hudi/sync/datahub/TestDataHubTableProperties.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.sync.datahub;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.hadoop.HoodieParquetInputFormat;
+import org.apache.hudi.sync.datahub.config.DataHubSyncConfig;
+
+import org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat;
+import org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_BASE_PATH;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_TYPE;
+import static org.apache.hudi.sync.datahub.DataHubTableProperties.HUDI_TABLE_VERSION;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+class TestDataHubTableProperties {
+ @Test
+ void testGetTableProperties() {
+ String path = "file:///tmp/path";
+ Map<String, String> expected = new HashMap<>();
+ expected.put(HUDI_TABLE_TYPE, "MERGE_ON_READ");
+ expected.put(HUDI_TABLE_VERSION, "SIX");
Review Comment:
@sgomezvillamor regardless of the misalignment, could you enhance the docs to specify the expected properties synced to the datahub catalog, especially the table version so the user does not get confused (on top of https://github.com/apache/hudi/pull/12504)?
##########
hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/DataHubTableProperties.java:
##########
@@ -86,8 +86,8 @@ private static void addUserDefinedProperties(Map<String, String> properties, Dat
private static void addSparkRelatedProperties(Map<String, String> properties, DataHubSyncConfig config, HoodieTableMetadata tableMetadata) {
Map<String, String> sparkProperties = SparkDataSourceTableUtils.getSparkTableProperties(
config.getSplitStrings(META_SYNC_PARTITION_FIELDS),
- config.getString(META_SYNC_SPARK_VERSION),
- config.getInt(HIVE_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD),
+ config.getStringOrDefault(META_SYNC_SPARK_VERSION),
+ config.getIntOrDefault(HIVE_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD),
Review Comment:
I see ADB sync also uses `config.getString(META_SYNC_SPARK_VERSION)`. Should that be fixed separately?
```
Map<String, String> sparkTableProperties = SparkDataSourceTableUtils.getSparkTableProperties(config.getSplitStrings(META_SYNC_PARTITION_FIELDS),
config.getString(META_SYNC_SPARK_VERSION), config.getInt(ADB_SYNC_SCHEMA_STRING_LENGTH_THRESHOLD), schema);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org