You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2023/09/07 05:08:44 UTC
[GitHub] [hudi] Zouxxyy opened a new pull request, #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Zouxxyy opened a new pull request, #9554:
URL: https://github.com/apache/hudi/pull/9554
…Format
### Change Logs
Currently, After doing schema evalution using spark-sql, query using hive will fail
```sql
-- spark-sql
set hoodie.schema.on.read.enable=true;
create table hudi_mor_test_tbl (
id bigint,
name string,
ts int,
dt string,
hh string
) using hudi
tblproperties (
type = 'mor',
primaryKey = 'id',
preCombineField = 'ts'
)
partitioned by (dt, hh);
insert into hudi_mor_test_tbl values (1, 'a1', 1001, '2021-12-09', '10');
ALTER TABLE hudi_mor_test_tbl ALTER COLUMN ts TYPE bigint;
-- hive
select * from hudi_mor_test_tbl_rt;
Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.IntWritable
```
The root cause is that `FileInputFormat` does not implement `SelfDescribingInputFormatInterface`, see
```java
/**
* Marker interface to indicate a given input format is self-describing and
* can perform schema evolution itself.
*/
public interface SelfDescribingInputFormatInterface {
}
```
### Impact
After doing schema evalution using spark-sql, query using hive will success
### Risk level (write none, low medium or high below)
none
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1696020770
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523",
"triggerID" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b4d6290f84232bd5216be195bebb562eba6ce1ba Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514)
* 4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1707796519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523",
"triggerID" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"triggerType" : "PUSH"
}, {
"hash" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19665",
"triggerID" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8561ef30bbcfa744c25e36f09c216460b56076e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19665)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1695226959
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91529a3dc8cbe6ab5683aed634863f199d969d77 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1694930868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91529a3dc8cbe6ab5683aed634863f199d969d77 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1695075865
@Zouxxyy Can you elaborate a little more what the purpose of this change? Does it has risk of breaking the compatibility for low version Hive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1720370591
@yihua see https://github.com/apache/hudi/pull/7129 , it turns out that this question has already been raised
I think he had the same problem as me
<img width="849" alt="image" src="https://github.com/apache/hudi/assets/37108074/44af7e3b-53fc-4abc-b476-8679ce2c945b">
The hive dependency is not passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1695292811
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91529a3dc8cbe6ab5683aed634863f199d969d77 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509)
* b4d6290f84232bd5216be195bebb562eba6ce1ba UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput… [hudi]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1865380057
@bvaradar Sorry for delay, thanks for your help, It seems that the CI is not stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1732120345
@Zouxxyy have you figured out why integration tests failed in the GH actions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1712679632
You can take a look at the readme: https://github.com/apache/hudi/tree/master/hudi-integ-test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1865373757
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523",
"triggerID" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"triggerType" : "PUSH"
}, {
"hash" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19665",
"triggerID" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"triggerType" : "PUSH"
}, {
"hash" : "db65ca67c67d44d19cd7a56b734f846466feead6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21622",
"triggerID" : "db65ca67c67d44d19cd7a56b734f846466feead6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db65ca67c67d44d19cd7a56b734f846466feead6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21622)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1865212729
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523",
"triggerID" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"triggerType" : "PUSH"
}, {
"hash" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19665",
"triggerID" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"triggerType" : "PUSH"
}, {
"hash" : "db65ca67c67d44d19cd7a56b734f846466feead6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21622",
"triggerID" : "db65ca67c67d44d19cd7a56b734f846466feead6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8561ef30bbcfa744c25e36f09c216460b56076e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19665)
* db65ca67c67d44d19cd7a56b734f846466feead6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21622)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1698367243
Can someone help to understand why `hudi-spark-common` cannot automatically depend on `hive-exec` in `hudi-hadoop-mr` ?
```text
mvn dependency:tree -pl hudi-spark-datasource/hudi-spark-common -Dspark2
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-spark-common_2.12 ---
[INFO] org.apache.hudi:hudi-spark-common_2.12:jar:0.15.0-SNAPSHOT
[INFO] +- org.apache.hudi:hudi-hive-sync:jar:0.15.0-SNAPSHOT:compile
[INFO] | +- org.apache.hudi:hudi-hadoop-mr:jar:0.15.0-SNAPSHOT:compile
[INFO] | +- org.apache.hudi:hudi-sync-common:jar:0.15.0-SNAPSHOT:compile
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1720264438
> @xushiyan @bvaradar Can someone help to understand why `hudi-spark-common` cannot automatically depend on `hive-exec` in `hudi-hadoop-mr` ?
>
> ```
> mvn dependency:tree -pl hudi-spark-datasource/hudi-spark-common -Dspark2
>
> [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-spark-common_2.12 ---
> [INFO] org.apache.hudi:hudi-spark-common_2.12:jar:0.15.0-SNAPSHOT
> [INFO] +- org.apache.hudi:hudi-hive-sync:jar:0.15.0-SNAPSHOT:compile
> [INFO] | +- org.apache.hudi:hudi-hadoop-mr:jar:0.15.0-SNAPSHOT:compile
> [INFO] | +- org.apache.hudi:hudi-sync-common:jar:0.15.0-SNAPSHOT:compile
> ```
I think this dependency is already the case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1720245787
@Zouxxyy you need to run the exact same commands as shown in the logs in the docker environment to debug the failed integration. It looks like the HiveSyncTool spark job fails due to class not found. Likely the new class, `SelfDescribingInputFormatInterface`, is not included in the bundle.
Also wondering, how does `SelfDescribingInputFormatInterface` automatically fix the schema evolution (I don't see any API implemented)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #9554:
URL: https://github.com/apache/hudi/pull/9554#discussion_r1307090034
##########
hudi-spark-datasource/hudi-spark-common/pom.xml:
##########
@@ -228,6 +228,19 @@
<scope>test</scope>
</dependency>
+ <!-- Hive -->
Review Comment:
If this is not added, the compilation will fail in the spark2 profile, because in `CreateHoodieTableCommand`'s `classOf[HoodieParquetInputFormat]` need to load `SelfDescribingInputFormatInterface`:
```scala
val inputFormat = tableType match {
case DataSourceWriteOptions.COW_TABLE_TYPE_OPT_VAL =>
classOf[HoodieParquetInputFormat].getCanonicalName
case DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL =>
classOf[HoodieParquetRealtimeInputFormat].getCanonicalName
case _=> throw new IllegalArgumentException(s"UnKnow table type:$tableType")
}
```
Compile successfully with Spark3 profile, because in spark3, spark-hive_2.12 depends on hive-exec:
```text
[INFO] +- org.apache.spark:spark-hive_2.12:jar:3.4.1:provided
[INFO] | +- org.apache.hive:hive-exec:jar:core:2.3.1:provided
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #9554:
URL: https://github.com/apache/hudi/pull/9554#discussion_r1308223983
##########
hudi-spark-datasource/hudi-spark-common/pom.xml:
##########
@@ -228,6 +228,19 @@
<scope>test</scope>
</dependency>
+ <!-- Hive -->
+ <dependency>
+ <groupId>${hive.groupid}</groupId>
+ <artifactId>hive-exec</artifactId>
+ <classifier>${hive.exec.classifier}</classifier>
+ <exclusions>
+ <exclusion>
Review Comment:
here is the error, `SelfDescribingInputFormatInterface` is from `hive-exec`
```
error: Class org.apache.hadoop.hive.ql.io.SelfDescribingInputFormatInterface not found - continuing with a stub.
[ERROR] classOf[HoodieParquetInputFormat].getCanonicalName
[ERROR] ^
[WARNING] one warning found
[ERROR] one error found
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1695563105
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b4d6290f84232bd5216be195bebb562eba6ce1ba Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1695306273
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91529a3dc8cbe6ab5683aed634863f199d969d77 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509)
* b4d6290f84232bd5216be195bebb562eba6ce1ba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9554:
URL: https://github.com/apache/hudi/pull/9554#discussion_r1308198370
##########
hudi-spark-datasource/hudi-spark-common/pom.xml:
##########
@@ -228,6 +228,19 @@
<scope>test</scope>
</dependency>
+ <!-- Hive -->
+ <dependency>
+ <groupId>${hive.groupid}</groupId>
+ <artifactId>hive-exec</artifactId>
+ <classifier>${hive.exec.classifier}</classifier>
+ <exclusions>
+ <exclusion>
Review Comment:
I didn't see why the second why can work, both of the classes are from Hudi repo, why the `hive-exec` is a requisite then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1712404491
here is the error in integration-tests, don't know much about the env of integration testing, can anyone help~
```
2023-09-08T05:11:59.7764700Z Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.SelfDescribingInputFormatInterface
2023-09-08T05:11:59.7764906Z at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
2023-09-08T05:11:59.7765092Z at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
2023-09-08T05:11:59.7765284Z at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2023-09-08T05:11:59.7765373Z ... 58 more
2023-09-08T05:11:59.7765560Z 23/09/08 05:11:59 INFO util.ShutdownHookManager: Shutdown hook called
2023-09-08T05:11:59.7766126Z 23/09/08 05:11:59 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-b81218a3-32e6-4851-9b25-b15373acd05b
2023-09-08T05:11:59.7766507Z 23/09/08 05:11:59 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-9b58a267-201d-4404-baeb-49e617b23ad1
2023-09-08T05:11:59.7766647Z
2023-09-08T05:11:59.7766919Z Sep 08, 2023 5:11:59 AM org.glassfish.jersey.internal.Errors logErrors
2023-09-08T05:11:59.7767534Z WARNING: The following warnings have been detected: WARNING: Cannot create new registration for component type class com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider: Existing previous registration found for the type.
2023-09-08T05:11:59.7767548Z
2023-09-08T05:11:59.7768090Z [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 96.75 s <<< FAILURE! - in org.apache.hudi.integ.command.ITTestHoodieSyncCommand
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1695955459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b4d6290f84232bd5216be195bebb562eba6ce1ba Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514)
* 4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9554:
URL: https://github.com/apache/hudi/pull/9554#discussion_r1308101795
##########
hudi-spark-datasource/hudi-spark-common/pom.xml:
##########
@@ -228,6 +228,19 @@
<scope>test</scope>
</dependency>
+ <!-- Hive -->
+ <dependency>
+ <groupId>${hive.groupid}</groupId>
+ <artifactId>hive-exec</artifactId>
+ <classifier>${hive.exec.classifier}</classifier>
+ <exclusions>
+ <exclusion>
Review Comment:
Not sure we should depend on Hive in `hudi-spark-datasource`, `hudi-hadoop-mr` should be the dependency if we want to use the hive inputformat?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput… [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1858420270
@Zouxxyy : Can you rebase and resolve the merge conflict ? We can take a look at the test failure after that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1707559011
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523",
"triggerID" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"triggerType" : "PUSH"
}, {
"hash" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523)
* d8561ef30bbcfa744c25e36f09c216460b56076e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput… [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-6700220144
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput… [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1884186097
Rerunning failed jobs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #9554:
URL: https://github.com/apache/hudi/pull/9554#discussion_r1307090034
##########
hudi-spark-datasource/hudi-spark-common/pom.xml:
##########
@@ -228,6 +228,19 @@
<scope>test</scope>
</dependency>
+ <!-- Hive -->
Review Comment:
If this is not added, the compilation will fail in the spark2 profile with `class not found`, see https://github.com/apache/hudi/actions/runs/5994914337/job/16257181299
Because in `CreateHoodieTableCommand`'s `classOf[HoodieParquetInputFormat]` need to load `SelfDescribingInputFormatInterface`:
```scala
val inputFormat = tableType match {
case DataSourceWriteOptions.COW_TABLE_TYPE_OPT_VAL =>
classOf[HoodieParquetInputFormat].getCanonicalName
case DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL =>
classOf[HoodieParquetRealtimeInputFormat].getCanonicalName
case _=> throw new IllegalArgumentException(s"UnKnow table type:$tableType")
}
```
Compile successfully with Spark3 profile, because in spark3, spark-hive_2.12 depends on hive-exec:
```text
[INFO] +- org.apache.spark:spark-hive_2.12:jar:3.4.1:provided
[INFO] | +- org.apache.hive:hive-exec:jar:core:2.3.1:provided
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1694936436
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91529a3dc8cbe6ab5683aed634863f199d969d77 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
URL: https://github.com/apache/hudi/pull/9554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy closed pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy closed pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
URL: https://github.com/apache/hudi/pull/9554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput… [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
URL: https://github.com/apache/hudi/pull/9554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1707565205
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523",
"triggerID" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"triggerType" : "PUSH"
}, {
"hash" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19665",
"triggerID" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523)
* d8561ef30bbcfa744c25e36f09c216460b56076e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19665)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1865204330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523",
"triggerID" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"triggerType" : "PUSH"
}, {
"hash" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19665",
"triggerID" : "d8561ef30bbcfa744c25e36f09c216460b56076e",
"triggerType" : "PUSH"
}, {
"hash" : "db65ca67c67d44d19cd7a56b734f846466feead6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db65ca67c67d44d19cd7a56b734f846466feead6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8561ef30bbcfa744c25e36f09c216460b56076e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19665)
* db65ca67c67d44d19cd7a56b734f846466feead6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput… [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1865156069
Fixed conflicts and rebased. Made minor changes to align with latest code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1696329368
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19509",
"triggerID" : "91529a3dc8cbe6ab5683aed634863f199d969d77",
"triggerType" : "PUSH"
}, {
"hash" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19514",
"triggerID" : "b4d6290f84232bd5216be195bebb562eba6ce1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523",
"triggerID" : "4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cc84c2ecfbe2b8553f0675ff60aa4abebd8f315 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19523)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #9554:
URL: https://github.com/apache/hudi/pull/9554#discussion_r1308145131
##########
hudi-spark-datasource/hudi-spark-common/pom.xml:
##########
@@ -228,6 +228,19 @@
<scope>test</scope>
</dependency>
+ <!-- Hive -->
+ <dependency>
+ <groupId>${hive.groupid}</groupId>
+ <artifactId>hive-exec</artifactId>
+ <classifier>${hive.exec.classifier}</classifier>
+ <exclusions>
+ <exclusion>
Review Comment:
`hudi-hadoop-mr` is already dependent, and currently `hudi-spark-datasource/hudi-spark` also depends on `hive-exec`
Of course, if we don't want to add `hive-exec` dependency in `hudi-spark-datasource/hudi-common`, we can change it like this
```scala
val inputFormat = tableType match {
case DataSourceWriteOptions. COW_TABLE_TYPE_OPT_VAL =>
"org.apache.hudi.hadoop.HoodieParquetInputFormat"
case DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL =>
"org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat"
case _=> throw new IllegalArgumentException(s"UnKnow table type:$tableType")
}
```
which way do you think is better~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #9554:
URL: https://github.com/apache/hudi/pull/9554#discussion_r1308223983
##########
hudi-spark-datasource/hudi-spark-common/pom.xml:
##########
@@ -228,6 +228,19 @@
<scope>test</scope>
</dependency>
+ <!-- Hive -->
+ <dependency>
+ <groupId>${hive.groupid}</groupId>
+ <artifactId>hive-exec</artifactId>
+ <classifier>${hive.exec.classifier}</classifier>
+ <exclusions>
+ <exclusion>
Review Comment:
here is the error:
```
error: Class org.apache.hadoop.hive.ql.io.SelfDescribingInputFormatInterface not found - continuing with a stub.
[ERROR] classOf[HoodieParquetInputFormat].getCanonicalName
[ERROR] ^
[WARNING] one warning found
[ERROR] one error found
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #9554: [HUDI-6760] Add SelfDescribingInputFormatInterface for hive FileInput…
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #9554:
URL: https://github.com/apache/hudi/pull/9554#issuecomment-1695249433
@danny0405
> Can you elaborate a little more what the purpose of this change?
See updated Change Logs.
> Does it has risk of breaking the compatibility for low version Hive?
This interface (`SelfDescribingInputFormatInterface `) has existed since hive 2.0, and there is no compatibility problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org