You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "thomasg19930417 (via GitHub)" <gi...@apache.org> on 2023/06/09 01:23:19 UTC
[GitHub] [hudi] thomasg19930417 opened a new pull request, #8911: Compatible with hive 2.2.x to read hudi rt table
thomasg19930417 opened a new pull request, #8911:
URL: https://github.com/apache/hudi/pull/8911
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1588633923
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 33e446f39c09132f2d1d88f150bfc86965f191f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1590317570
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
}, {
"hash" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800",
"triggerID" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"triggerType" : "PUSH"
}, {
"hash" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17815",
"triggerID" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 025d39357622c7c7aedda759fa9e2a89c3a096c8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800)
* fd37969ecda4dc9b065750fbbfb0011d229d11a0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17815)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-5069778038
Closing this due to inactivity. Anyone that still requires compatibility can lookup this fix in the closed section of the PR.
We're planning to move to a Hive 4.0.2 moving forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1605807604
> It's kind of huge, not sure whether it deserves to be compatible with Hive 2.2
Should I turn off this PR first or temporarily
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227577203
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShimAbove23.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * When the hive version is greater than or equal to hive version 2.3
+ */
+public class HiveCompatibleShimAbove23 implements HiveCompatibleShim {
+
+ private static HiveCompatibleShimAbove23 INSTANCE = new HiveCompatibleShimAbove23();
+
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
+ try {
+ Class hiveInputFormatClass = this.getClass().getSuperclass();
+ Method pushProjectionsAndFilters = hiveInputFormatClass.getDeclaredMethod("pushProjectionsAndFilters", JobConf.class,
Review Comment:
Oh, sorry, I know your concern, can this be compiled with -Dhive.version=2.2.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1592302164
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
}, {
"hash" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800",
"triggerID" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"triggerType" : "PUSH"
}, {
"hash" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17815",
"triggerID" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"triggerType" : "PUSH"
}, {
"hash" : "a6e33c2a98128b7374b8dfd19324a7d1fb0575f1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17864",
"triggerID" : "a6e33c2a98128b7374b8dfd19324a7d1fb0575f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* fd37969ecda4dc9b065750fbbfb0011d229d11a0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17815)
* a6e33c2a98128b7374b8dfd19324a7d1fb0575f1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17864)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1586487905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ddc84cab970a6a43ea77a729213dc8c5200d845 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690)
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 052495cd90a328786080a236ece4ec97b813e8f9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227578284
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShimBelow23.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * When the hive version is less than hive version 2.3
+ */
+public class HiveCompatibleShimBelow23 implements HiveCompatibleShim {
+
+ private static HiveCompatibleShimBelow23 INSTANCE = new HiveCompatibleShimBelow23();
+
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
+ try {
+ Class hiveInputFormatClass = this.getClass().getSuperclass();
Review Comment:
This may not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1584337465
> I noticed that you added a lot of newlines, is it to fix checkStyle?
Yes, I will take a look at the correction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with hive 2.2.x to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1583838150
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ddc84cab970a6a43ea77a729213dc8c5200d845 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1226348446
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -562,8 +568,35 @@ public RecordReader getRecordReader(InputSplit split, JobConf job, Reporter repo
}
}
+ private void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
Review Comment:
@xicm hi ,In invokePushProjectAndFilters, it is not completely the difference between the two major versions of hive2.x and hive3.x. Will this part of the code be moved into hiveshims?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on pull request #8911: [Hudi-8882] Compatible with hive 2.2.x to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1583793737
Add issue link #8882
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1224074166
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveCompatibleUtils.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.function.Function;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.plan.PartitionDesc;
+
+public class HiveCompatibleUtils {
+
+ public static <K> Map<Map<Path, PartitionDesc>, Map<Path, PartitionDesc>> convertPartitionDesc(
+ Map<Map<K, PartitionDesc>, Map<K, PartitionDesc>> oldMap) {
+ Map<Map<Path, PartitionDesc>, Map<Path, PartitionDesc>> resMap = new LinkedHashMap<>();
+ Set<Entry<Map<K, PartitionDesc>, Map<K, PartitionDesc>>> entries = oldMap.entrySet();
+ for (Entry<Map<K, PartitionDesc>, Map<K, PartitionDesc>> entry : entries) {
+ Map<Path, PartitionDesc> newKeyMap = convertMapKeyToPath(entry.getKey());
+ Map<Path, PartitionDesc> newValueMap = convertMapKeyToPath(entry.getValue());
+ resMap.put(newKeyMap, newValueMap);
+ }
+ return resMap;
+ }
+
+ public static <K, V, T> Map<Path, V> convertMapKeyToPath(Map<K, V> pathToAliases) {
Review Comment:
redundant `T`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1589244111
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
}, {
"hash" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 80b1d58fbbdb619e66007f896fce83dc45f4e846 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795)
* 025d39357622c7c7aedda759fa9e2a89c3a096c8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1226162058
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -562,8 +568,35 @@ public RecordReader getRecordReader(InputSplit split, JobConf job, Reporter repo
}
}
+ private void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227399206
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -562,8 +568,35 @@ public RecordReader getRecordReader(InputSplit split, JobConf job, Reporter repo
}
}
+ private void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
Review Comment:
Is adding a shim to distinguish between 2.3 and 2.3+?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1586612044
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 052495cd90a328786080a236ece4ec97b813e8f9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1228975860
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShim.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.HiveInputFormat;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * Shim class to resolve Hive version compatibility.
+ */
+public interface HiveCompatibleShim {
Review Comment:
Can we merge this interface into HiveShim ?
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveCompatibleUtils.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.function.Function;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.plan.PartitionDesc;
+
+/**
+ * This class is used to assist compatibility with versions smaller than hive 2.3 (hive 2.x).
+ * There have been some API changes on hive (2.0-2.2) and hive 2.3, mainly the data structure
+ * returned in mapwork.getPathToAliases. In 2.3, the key returned was path, while in previous
+ * versions, String was returned. What was done here is to convert it uniformly to Path so
+ * that existing code can accommodate these changes
Review Comment:
All the document should end up with `.`.
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HudiHiveInputformatShimAbove23.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.HiveInputFormat;
+import org.apache.hadoop.mapred.JobConf;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * When the hive version is greater than or equal to hive version 2.3
+ */
+public class HudiHiveInputformatShimAbove23 implements HiveCompatibleShim {
+
+ public static final Logger LOG = LoggerFactory.getLogger(HudiHiveInputformatShimAbove23.class);
+
+ private static HudiHiveInputformatShimAbove23 INSTANCE = new HudiHiveInputformatShimAbove23();
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ private static Method PUSH_PROJECT_METHOD;
+
+ static {
+ try {
+ PUSH_PROJECT_METHOD = HiveInputFormat.class.getDeclaredMethod("pushProjectionsAndFilters", JobConf.class,
+ Class.class, Path.class);
Review Comment:
Do we need reflection for version 2.3+ ?, the compile version of Hudi Hive should be always greater than 2.3.
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -87,16 +91,19 @@
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
+import static org.apache.hudi.hadoop.utils.HiveCompatibleUtils.convertPartitionDesc;
+import static org.apache.hudi.hadoop.utils.HiveCompatibleUtils.convertMapKeyToPath;
+
/**
- * This is just a copy of the org.apache.hadoop.hive.ql.io.CombineHiveInputFormat from Hive 2.x Search for **MOD** to
- * see minor modifications to support custom inputformat in CombineHiveInputFormat. See
- * https://issues.apache.org/jira/browse/HIVE-9771
+ * This is just a copy of the org.apache.hadoop.hive.ql.io.CombineHiveInputFormat from Hive 2.x
+ * Search for **MOD** to see minor modifications to support custom inputformat in
+ * CombineHiveInputFormat. See https://issues.apache.org/jira/browse/HIVE-9771
* <p>
* <p>
- * CombineHiveInputFormat is a parameterized InputFormat which looks at the path name and determine the correct
- * InputFormat for that path name from mapredPlan.pathToPartitionInfo(). It can be used to read files with different
- * input format in the same map-reduce job.
- *
+ * CombineHiveInputFormat is a parameterized InputFormat which looks at the path name and determine
+ * the correct InputFormat for that path name from mapredPlan.pathToPartitionInfo(). It can be used
+ * to read files with different input format in the same map-reduce job.
Review Comment:
Revert all the unnecessary change to the document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1588482177
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 3c7b437851141a00abd75c470702f07aedb37de1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789)
* 33e446f39c09132f2d1d88f150bfc86965f191f8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1229102825
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShim.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.HiveInputFormat;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * Shim class to resolve Hive version compatibility.
+ */
+public interface HiveCompatibleShim {
Review Comment:
The differences are between 2.3 and <2.3, HiveShim is between hive2 and hive3, so we may not merge it into.
And I think the name `HudiHiveInputformatShim` is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1590312853
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
}, {
"hash" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800",
"triggerID" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"triggerType" : "PUSH"
}, {
"hash" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 025d39357622c7c7aedda759fa9e2a89c3a096c8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800)
* fd37969ecda4dc9b065750fbbfb0011d229d11a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1584293920
I noticed that you added a lot of newlines, is it to fix checkStyle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1224058977
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -390,24 +401,28 @@ private void processPaths(JobConf job, CombineFileInputFormatShim combine, List<
/**
* HiveFileFormatUtils.getPartitionDescFromPathRecursively is no longer available since Hive 3.
* This method is to make it compatible with both Hive 2 and Hive 3.
+ *
* @param pathToPartitionInfo
* @param dir
* @param cacheMap
* @return
* @throws IOException
*/
- private static PartitionDesc getPartitionFromPath(Map<Path, PartitionDesc> pathToPartitionInfo, Path dir,
+ private static PartitionDesc getPartitionFromPath(Map<Path, PartitionDesc> pathToPartitionInfo,
+ Path dir,
Map<Map<Path, PartitionDesc>, Map<Path, PartitionDesc>> cacheMap)
throws IOException {
Method method;
try {
- Class<?> hiveUtilsClass = Class.forName("org.apache.hadoop.hive.ql.io.HiveFileFormatUtils");
+ Class<?> hiveUtilsClass = Class.forName("org.apache.hudi.hadoop.utils.HudiHiveFileFormatUtils");
try {
// HiveFileFormatUtils.getPartitionDescFromPathRecursively method only available in Hive 2.x
- method = hiveUtilsClass.getMethod("getPartitionDescFromPathRecursively", Map.class, Path.class, Map.class);
+ method = hiveUtilsClass
+ .getMethod("getPartitionDescFromPathRecursively", Map.class, Path.class, Map.class);
} catch (NoSuchMethodException e) {
// HiveFileFormatUtils.getFromPathRecursively method only available in Hive 3.x
- method = hiveUtilsClass.getMethod("getFromPathRecursively", Map.class, Path.class, Map.class);
+ method = hiveUtilsClass
+ .getMethod("getFromPathRecursively", Map.class, Path.class, Map.class);
Review Comment:
`hiveUtilsClass` is `HudiHiveFileFormatUtils` now, is the reflection working?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous closed pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
URL: https://github.com/apache/hudi/pull/8911
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1224091960
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -390,24 +401,28 @@ private void processPaths(JobConf job, CombineFileInputFormatShim combine, List<
/**
* HiveFileFormatUtils.getPartitionDescFromPathRecursively is no longer available since Hive 3.
* This method is to make it compatible with both Hive 2 and Hive 3.
+ *
* @param pathToPartitionInfo
* @param dir
* @param cacheMap
* @return
* @throws IOException
*/
- private static PartitionDesc getPartitionFromPath(Map<Path, PartitionDesc> pathToPartitionInfo, Path dir,
+ private static PartitionDesc getPartitionFromPath(Map<Path, PartitionDesc> pathToPartitionInfo,
+ Path dir,
Map<Map<Path, PartitionDesc>, Map<Path, PartitionDesc>> cacheMap)
throws IOException {
Method method;
try {
- Class<?> hiveUtilsClass = Class.forName("org.apache.hadoop.hive.ql.io.HiveFileFormatUtils");
+ Class<?> hiveUtilsClass = Class.forName("org.apache.hudi.hadoop.utils.HudiHiveFileFormatUtils");
try {
// HiveFileFormatUtils.getPartitionDescFromPathRecursively method only available in Hive 2.x
- method = hiveUtilsClass.getMethod("getPartitionDescFromPathRecursively", Map.class, Path.class, Map.class);
+ method = hiveUtilsClass
+ .getMethod("getPartitionDescFromPathRecursively", Map.class, Path.class, Map.class);
} catch (NoSuchMethodException e) {
// HiveFileFormatUtils.getFromPathRecursively method only available in Hive 3.x
- method = hiveUtilsClass.getMethod("getFromPathRecursively", Map.class, Path.class, Map.class);
+ method = hiveUtilsClass
+ .getMethod("getFromPathRecursively", Map.class, Path.class, Map.class);
Review Comment:
Need to test if this works with hive3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1224098760
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -549,21 +567,48 @@ public RecordReader getRecordReader(InputSplit split, JobConf job, Reporter repo
throw new IOException("cannot find class " + inputFormatClassName);
}
- pushProjectionsAndFilters(job, inputFormatClass, hsplit.getPath(0));
-
+ invokePushProjectAndFilters(job, inputFormatClass, hsplit.getPath(0));
if (inputFormatClass.getName().equals(getParquetRealtimeInputFormatClassName())) {
HoodieCombineFileInputFormatShim shims = createInputFormatShim();
IOContextMap.get(job).setInputPath(((CombineHiveInputSplit) split).getPath(0));
return shims.getRecordReader(job, ((CombineHiveInputSplit) split).getInputSplitShim(),
reporter, CombineHiveRecordReader.class);
} else {
- return ShimLoader.getHadoopShims().getCombineFileInputFormat().getRecordReader(job, (CombineFileSplit) split,
- reporter, CombineHiveRecordReader.class);
+ return ShimLoader.getHadoopShims().getCombineFileInputFormat()
+ .getRecordReader(job, (CombineFileSplit) split,
+ reporter, CombineHiveRecordReader.class);
+ }
+ }
+
+ private void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
Review Comment:
@danny0405 Is it will be better if we add a shim class to do `pushProjectionsAndFilters`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227427696
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -562,8 +568,35 @@ public RecordReader getRecordReader(InputSplit split, JobConf job, Reporter repo
}
}
+ private void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
Review Comment:
yes, <2.3 and 2.3+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1586462614
@xicm Hi, could you please help me review the code again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1584148702
Will ask for @xicm for the review first, and let's see if we need add some methods in the shim class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1586970055
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 052495cd90a328786080a236ece4ec97b813e8f9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750)
* c18451c3843a534946fab76b62ff726c99df9c77 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759)
* d919f87b331ef3273c9910fa77c668c58fb2edc3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1230329630
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HudiHiveInputformatShimAbove23.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.HiveInputFormat;
+import org.apache.hadoop.mapred.JobConf;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * When the hive version is greater than or equal to hive version 2.3
+ */
+public class HudiHiveInputformatShimAbove23 implements HiveCompatibleShim {
+
+ public static final Logger LOG = LoggerFactory.getLogger(HudiHiveInputformatShimAbove23.class);
+
+ private static HudiHiveInputformatShimAbove23 INSTANCE = new HudiHiveInputformatShimAbove23();
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ private static Method PUSH_PROJECT_METHOD;
+
+ static {
+ try {
+ PUSH_PROJECT_METHOD = HiveInputFormat.class.getDeclaredMethod("pushProjectionsAndFilters", JobConf.class,
+ Class.class, Path.class);
Review Comment:
@danny0405 This method pushProjectionsAndFilters is protected. This method also has some other contexts, which can only be reflected or inherited in the shim. If inherited, the context may be lost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1605831612
We can keep it open for someone that needs the compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1586481187
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ddc84cab970a6a43ea77a729213dc8c5200d845 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690)
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1226365144
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -562,8 +568,35 @@ public RecordReader getRecordReader(InputSplit split, JobConf job, Reporter repo
}
}
+ private void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
Review Comment:
We can add a new shim class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1588432073
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* d919f87b331ef3273c9910fa77c668c58fb2edc3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763)
* 3c7b437851141a00abd75c470702f07aedb37de1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227456677
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/Hive2Shim.java:
##########
@@ -53,4 +59,17 @@ public int getDays(Object dateWritable) {
public long getMills(Object timestamp) {
return ((Timestamp) timestamp).getTime();
}
+
+ @Override
+ public PartitionDesc getPartitionDesc(Map<Path, PartitionDesc> pathToPartitionInfo, Path dir,
+ Map<Map<Path, PartitionDesc>, Map<Path, PartitionDesc>> cacheMap) throws IOException {
+
+ try {
+ Class<?> hiveUtilsClass = Class.forName("org.apache.hudi.hadoop.utils.HudiHiveFileFormatUtils");
Review Comment:
Do we need reflection here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1589257368
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
}, {
"hash" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800",
"triggerID" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 80b1d58fbbdb619e66007f896fce83dc45f4e846 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795)
* 025d39357622c7c7aedda759fa9e2a89c3a096c8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1586492109
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ddc84cab970a6a43ea77a729213dc8c5200d845 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690)
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 052495cd90a328786080a236ece4ec97b813e8f9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1588476894
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* d919f87b331ef3273c9910fa77c668c58fb2edc3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763)
* 3c7b437851141a00abd75c470702f07aedb37de1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789)
* 33e446f39c09132f2d1d88f150bfc86965f191f8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1590453892
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
}, {
"hash" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800",
"triggerID" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"triggerType" : "PUSH"
}, {
"hash" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17815",
"triggerID" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* fd37969ecda4dc9b065750fbbfb0011d229d11a0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17815)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227550989
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShimBelow23.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * When the hive version is less than hive version 2.3
+ */
+public class HiveCompatibleShimBelow23 implements HiveCompatibleShim {
+
+ private static HiveCompatibleShimBelow23 INSTANCE = new HiveCompatibleShimBelow23();
+
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
+ try {
+ Class hiveInputFormatClass = this.getClass().getSuperclass();
+ Method pushProjectionsAndFilters = hiveInputFormatClass.getDeclaredMethod("pushProjectionsAndFilters", JobConf.class,
Review Comment:
inputFormatClass. getDeclaredMethod()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1588922443
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 80b1d58fbbdb619e66007f896fce83dc45f4e846 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227574416
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShimAbove23.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * When the hive version is greater than or equal to hive version 2.3
+ */
+public class HiveCompatibleShimAbove23 implements HiveCompatibleShim {
+
+ private static HiveCompatibleShimAbove23 INSTANCE = new HiveCompatibleShimAbove23();
+
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
+ try {
+ Class hiveInputFormatClass = this.getClass().getSuperclass();
+ Method pushProjectionsAndFilters = hiveInputFormatClass.getDeclaredMethod("pushProjectionsAndFilters", JobConf.class,
Review Comment:
@xicm It has been modified to directly inherit HiveINputFormat when it is greater than version 2.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227568344
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShimAbove23.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * When the hive version is greater than or equal to hive version 2.3
+ */
+public class HiveCompatibleShimAbove23 implements HiveCompatibleShim {
+
+ private static HiveCompatibleShimAbove23 INSTANCE = new HiveCompatibleShimAbove23();
+
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
+ try {
+ Class hiveInputFormatClass = this.getClass().getSuperclass();
+ Method pushProjectionsAndFilters = hiveInputFormatClass.getDeclaredMethod("pushProjectionsAndFilters", JobConf.class,
Review Comment:
Do you need to consider specifying the hive version when compiling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227598591
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShimBelow23.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * When the hive version is less than hive version 2.3
+ */
+public class HiveCompatibleShimBelow23 implements HiveCompatibleShim {
+
+ private static HiveCompatibleShimBelow23 INSTANCE = new HiveCompatibleShimBelow23();
+
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
+ try {
+ Class hiveInputFormatClass = this.getClass().getSuperclass();
Review Comment:
@xicm Sorry, I have re-modified here. In addition, it seems that it is not possible to directly inherit HiveInputFormat when the above is greater than hive2.3. There are some other contexts in the pushProjectionsAndFilters method. I think there may be some problems when calling directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1229102825
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShim.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.HiveInputFormat;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * Shim class to resolve Hive version compatibility.
+ */
+public interface HiveCompatibleShim {
Review Comment:
The differences are between 2.3 and <2.3, HiveShim is between hive2 and hive3, so we may not merge it into.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with hive 2.2.x to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1583993589
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ddc84cab970a6a43ea77a729213dc8c5200d845 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1224058647
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -64,39 +76,29 @@
import org.apache.hadoop.mapred.lib.CombineFileInputFormat;
import org.apache.hadoop.mapred.lib.CombineFileSplit;
import org.apache.hadoop.mapreduce.JobContext;
+import org.apache.hive.common.util.HiveVersionInfo;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.hadoop.HoodieParquetInputFormat;
+import org.apache.hudi.hadoop.HoodieParquetInputFormatBase;
+import org.apache.hudi.hadoop.realtime.HoodieCombineRealtimeRecordReader;
+import org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.hadoop.utils.HudiHiveFileFormatUtils;
+import org.apache.hudi.hadoop.utils.HudiStringInternUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.DataInput;
-import java.io.DataOutput;
-import java.io.IOException;
-import java.lang.reflect.Method;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Set;
-import java.util.concurrent.Callable;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
Review Comment:
Just add the package you added.
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -121,13 +123,15 @@ protected HoodieCombineFileInputFormatShim createInputFormatShim() {
return new HoodieCombineHiveInputFormat.HoodieCombineFileInputFormatShim<>();
}
+
/**
* Create Hive splits based on CombineFileSplit.
*/
- private InputSplit[] getCombineSplits(JobConf job, int numSplits, Map<Path, PartitionDesc> pathToPartitionInfo)
+ private InputSplit[] getCombineSplits(JobConf job, int numSplits,
+ Map<Path, PartitionDesc> pathToPartitionInfo)
Review Comment:
Is this to fix checkStyle？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1598004412
@danny0405 Can you help me review the code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1600245375
It's kind of huge, not sure whether it deserves to be compatible with Hive 2.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1226161654
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -401,13 +407,14 @@ private static PartitionDesc getPartitionFromPath(Map<Path, PartitionDesc> pathT
throws IOException {
Method method;
try {
- Class<?> hiveUtilsClass = Class.forName("org.apache.hadoop.hive.ql.io.HiveFileFormatUtils");
+ Class<?> hiveUtilsClass = Class.forName("org.apache.hudi.hadoop.utils.HudiHiveFileFormatUtils");
Review Comment:
Looks good to me, how about adding a shim class to handle differences between versions? like we do in `org.apache.hudi.hadoop.utils.shims`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1586982637
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* c18451c3843a534946fab76b62ff726c99df9c77 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759)
* d919f87b331ef3273c9910fa77c668c58fb2edc3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1587252281
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* d919f87b331ef3273c9910fa77c668c58fb2edc3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1590312009
@danny0405 @xicm Hi, I have merged the historical commits, thank you two for reviewing the code again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1224076980
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveCompatibleUtils.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.function.Function;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.plan.PartitionDesc;
+
+public class HiveCompatibleUtils {
+
+ public static <K> Map<Map<Path, PartitionDesc>, Map<Path, PartitionDesc>> convertPartitionDesc(
Review Comment:
I guess the difference between 2.3 and 2.2 is partition type is `String` or `Path`? Could you add some comment to the method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1224131443
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveCompatibleUtils.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.function.Function;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.plan.PartitionDesc;
+
+public class HiveCompatibleUtils {
+
+ public static <K> Map<Map<Path, PartitionDesc>, Map<Path, PartitionDesc>> convertPartitionDesc(
Review Comment:
I will add some explanations here. The main difference is between string and path, as well as pushProjectionsAndFilters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1224127602
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -390,24 +401,28 @@ private void processPaths(JobConf job, CombineFileInputFormatShim combine, List<
/**
* HiveFileFormatUtils.getPartitionDescFromPathRecursively is no longer available since Hive 3.
* This method is to make it compatible with both Hive 2 and Hive 3.
+ *
* @param pathToPartitionInfo
* @param dir
* @param cacheMap
* @return
* @throws IOException
*/
- private static PartitionDesc getPartitionFromPath(Map<Path, PartitionDesc> pathToPartitionInfo, Path dir,
+ private static PartitionDesc getPartitionFromPath(Map<Path, PartitionDesc> pathToPartitionInfo,
+ Path dir,
Map<Map<Path, PartitionDesc>, Map<Path, PartitionDesc>> cacheMap)
throws IOException {
Method method;
try {
- Class<?> hiveUtilsClass = Class.forName("org.apache.hadoop.hive.ql.io.HiveFileFormatUtils");
+ Class<?> hiveUtilsClass = Class.forName("org.apache.hudi.hadoop.utils.HudiHiveFileFormatUtils");
try {
// HiveFileFormatUtils.getPartitionDescFromPathRecursively method only available in Hive 2.x
- method = hiveUtilsClass.getMethod("getPartitionDescFromPathRecursively", Map.class, Path.class, Map.class);
+ method = hiveUtilsClass
+ .getMethod("getPartitionDescFromPathRecursively", Map.class, Path.class, Map.class);
} catch (NoSuchMethodException e) {
// HiveFileFormatUtils.getFromPathRecursively method only available in Hive 3.x
- method = hiveUtilsClass.getMethod("getFromPathRecursively", Map.class, Path.class, Map.class);
+ method = hiveUtilsClass
+ .getMethod("getFromPathRecursively", Map.class, Path.class, Map.class);
Review Comment:
I will modify the hiveUtilsClass in this exception back to hive3 later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227550752
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/Hive3Shim.java:
##########
@@ -122,4 +126,16 @@ public long getMills(Object timestamp) {
throw new HoodieException("can not create writable v2 class!", e);
}
}
+
+ @Override
+ public PartitionDesc getPartitionDesc(Map<Path, PartitionDesc> pathToPartitionInfo, Path dir,
+ Map<Map<Path, PartitionDesc>, Map<Path, PartitionDesc>> cacheMap) throws IOException {
+ try {
+ Class<?> hiveUtilsClass = Class.forName("org.apache.hadoop.hive.ql.io.HiveFileFormatUtils");
+ Method method = hiveUtilsClass.getMethod("getFromPathRecursively", Map.class, Path.class, Map.class);
Review Comment:
Sorry for the delay, do reflection in static block.
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShimAbove23.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * When the hive version is greater than or equal to hive version 2.3
+ */
+public class HiveCompatibleShimAbove23 implements HiveCompatibleShim {
+
+ private static HiveCompatibleShimAbove23 INSTANCE = new HiveCompatibleShimAbove23();
+
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
+ try {
+ Class hiveInputFormatClass = this.getClass().getSuperclass();
Review Comment:
is this right?
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShimAbove23.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * When the hive version is greater than or equal to hive version 2.3
+ */
+public class HiveCompatibleShimAbove23 implements HiveCompatibleShim {
+
+ private static HiveCompatibleShimAbove23 INSTANCE = new HiveCompatibleShimAbove23();
+
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
+ try {
+ Class hiveInputFormatClass = this.getClass().getSuperclass();
+ Method pushProjectionsAndFilters = hiveInputFormatClass.getDeclaredMethod("pushProjectionsAndFilters", JobConf.class,
Review Comment:
The default version of hive is 2.3, can we extends `HiveInputFormat` and call `pushProjectionsAndFilters ` directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1586680694
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 052495cd90a328786080a236ece4ec97b813e8f9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750)
* c18451c3843a534946fab76b62ff726c99df9c77 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1588423179
@xicm The code has been moved into the new shim, please review the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1588655200
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 33e446f39c09132f2d1d88f150bfc86965f191f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792)
* 80b1d58fbbdb619e66007f896fce83dc45f4e846 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1588645167
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 33e446f39c09132f2d1d88f150bfc86965f191f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792)
* 80b1d58fbbdb619e66007f896fce83dc45f4e846 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1592260047
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
}, {
"hash" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800",
"triggerID" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"triggerType" : "PUSH"
}, {
"hash" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17815",
"triggerID" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"triggerType" : "PUSH"
}, {
"hash" : "a6e33c2a98128b7374b8dfd19324a7d1fb0575f1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a6e33c2a98128b7374b8dfd19324a7d1fb0575f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* fd37969ecda4dc9b065750fbbfb0011d229d11a0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17815)
* a6e33c2a98128b7374b8dfd19324a7d1fb0575f1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1592527200
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
}, {
"hash" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800",
"triggerID" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"triggerType" : "PUSH"
}, {
"hash" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17815",
"triggerID" : "fd37969ecda4dc9b065750fbbfb0011d229d11a0",
"triggerType" : "PUSH"
}, {
"hash" : "a6e33c2a98128b7374b8dfd19324a7d1fb0575f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17864",
"triggerID" : "a6e33c2a98128b7374b8dfd19324a7d1fb0575f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* a6e33c2a98128b7374b8dfd19324a7d1fb0575f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17864)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1586673010
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 052495cd90a328786080a236ece4ec97b813e8f9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750)
* c18451c3843a534946fab76b62ff726c99df9c77 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1229324979
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShim.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.HiveInputFormat;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * Shim class to resolve Hive version compatibility.
+ */
+public interface HiveCompatibleShim {
Review Comment:
Yeah, but still a little verbose for these shim classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1588438736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* d919f87b331ef3273c9910fa77c668c58fb2edc3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763)
* 3c7b437851141a00abd75c470702f07aedb37de1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227471241
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/Hive2Shim.java:
##########
@@ -53,4 +59,17 @@ public int getDays(Object dateWritable) {
public long getMills(Object timestamp) {
return ((Timestamp) timestamp).getTime();
}
+
+ @Override
+ public PartitionDesc getPartitionDesc(Map<Path, PartitionDesc> pathToPartitionInfo, Path dir,
+ Map<Map<Path, PartitionDesc>, Map<Path, PartitionDesc>> cacheMap) throws IOException {
+
+ try {
+ Class<?> hiveUtilsClass = Class.forName("org.apache.hudi.hadoop.utils.HudiHiveFileFormatUtils");
Review Comment:
@xicm has been modified to call the method directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1227582220
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShimAbove23.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * When the hive version is greater than or equal to hive version 2.3
+ */
+public class HiveCompatibleShimAbove23 implements HiveCompatibleShim {
+
+ private static HiveCompatibleShimAbove23 INSTANCE = new HiveCompatibleShimAbove23();
+
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
+ try {
+ Class hiveInputFormatClass = this.getClass().getSuperclass();
+ Method pushProjectionsAndFilters = hiveInputFormatClass.getDeclaredMethod("pushProjectionsAndFilters", JobConf.class,
Review Comment:
I'll test this place
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1589564166
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
}, {
"hash" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbd84a73cdc4621ff516adcf9445781dc3b75f7",
"triggerType" : "PUSH"
}, {
"hash" : "052495cd90a328786080a236ece4ec97b813e8f9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17750",
"triggerID" : "052495cd90a328786080a236ece4ec97b813e8f9",
"triggerType" : "PUSH"
}, {
"hash" : "c18451c3843a534946fab76b62ff726c99df9c77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17759",
"triggerID" : "c18451c3843a534946fab76b62ff726c99df9c77",
"triggerType" : "PUSH"
}, {
"hash" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17763",
"triggerID" : "d919f87b331ef3273c9910fa77c668c58fb2edc3",
"triggerType" : "PUSH"
}, {
"hash" : "3c7b437851141a00abd75c470702f07aedb37de1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17789",
"triggerID" : "3c7b437851141a00abd75c470702f07aedb37de1",
"triggerType" : "PUSH"
}, {
"hash" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17792",
"triggerID" : "33e446f39c09132f2d1d88f150bfc86965f191f8",
"triggerType" : "PUSH"
}, {
"hash" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17795",
"triggerID" : "80b1d58fbbdb619e66007f896fce83dc45f4e846",
"triggerType" : "PUSH"
}, {
"hash" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800",
"triggerID" : "025d39357622c7c7aedda759fa9e2a89c3a096c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbd84a73cdc4621ff516adcf9445781dc3b75f7 UNKNOWN
* 025d39357622c7c7aedda759fa9e2a89c3a096c8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17800)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1229100105
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HudiHiveInputformatShimAbove23.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import java.lang.reflect.Method;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.HiveInputFormat;
+import org.apache.hadoop.mapred.JobConf;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * When the hive version is greater than or equal to hive version 2.3
+ */
+public class HudiHiveInputformatShimAbove23 implements HiveCompatibleShim {
+
+ public static final Logger LOG = LoggerFactory.getLogger(HudiHiveInputformatShimAbove23.class);
+
+ private static HudiHiveInputformatShimAbove23 INSTANCE = new HudiHiveInputformatShimAbove23();
+ public static HiveCompatibleShim getInstance() {
+ return INSTANCE;
+ }
+
+ private static Method PUSH_PROJECT_METHOD;
+
+ static {
+ try {
+ PUSH_PROJECT_METHOD = HiveInputFormat.class.getDeclaredMethod("pushProjectionsAndFilters", JobConf.class,
+ Class.class, Path.class);
Review Comment:
Do we need to support -Dhive.version=2.2.x? If not, we don't need reflection here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1224098760
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:
##########
@@ -549,21 +567,48 @@ public RecordReader getRecordReader(InputSplit split, JobConf job, Reporter repo
throw new IOException("cannot find class " + inputFormatClassName);
}
- pushProjectionsAndFilters(job, inputFormatClass, hsplit.getPath(0));
-
+ invokePushProjectAndFilters(job, inputFormatClass, hsplit.getPath(0));
if (inputFormatClass.getName().equals(getParquetRealtimeInputFormatClassName())) {
HoodieCombineFileInputFormatShim shims = createInputFormatShim();
IOContextMap.get(job).setInputPath(((CombineHiveInputSplit) split).getPath(0));
return shims.getRecordReader(job, ((CombineHiveInputSplit) split).getInputSplitShim(),
reporter, CombineHiveRecordReader.class);
} else {
- return ShimLoader.getHadoopShims().getCombineFileInputFormat().getRecordReader(job, (CombineFileSplit) split,
- reporter, CombineHiveRecordReader.class);
+ return ShimLoader.getHadoopShims().getCombineFileInputFormat()
+ .getRecordReader(job, (CombineFileSplit) split,
+ reporter, CombineHiveRecordReader.class);
+ }
+ }
+
+ private void invokePushProjectAndFilters(JobConf job, Class<?> inputFormatClass, Path splitPath)
+ throws IOException {
Review Comment:
@danny0405 Is it better to add a shim class to do `pushProjectionsAndFilters`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on a diff in pull request #8911: [Hudi-8882] Compatible with versions below hive 2.3 to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on code in PR #8911:
URL: https://github.com/apache/hudi/pull/8911#discussion_r1230317107
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/shims/HiveCompatibleShim.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.utils.shims;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.HiveInputFormat;
+import org.apache.hadoop.mapred.JobConf;
+
+/**
+ * Shim class to resolve Hive version compatibility.
+ */
+public interface HiveCompatibleShim {
Review Comment:
@danny0405 Does this place still need to modify the class name, or make some other changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] thomasg19930417 commented on pull request #8911: [Hudi-8882] Compatible with hive 2.2.x to read hudi rt table
Posted by "thomasg19930417 (via GitHub)" <gi...@apache.org>.
thomasg19930417 commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1583794413
@danny0405 Please help to review the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8911: [Hudi-8882] Compatible with hive 2.2.x to read hudi rt table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8911:
URL: https://github.com/apache/hudi/pull/8911#issuecomment-1583844595
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690",
"triggerID" : "1ddc84cab970a6a43ea77a729213dc8c5200d845",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1ddc84cab970a6a43ea77a729213dc8c5200d845 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17690)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org