You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "akshat0395 (via GitHub)" <gi...@apache.org> on 2023/02/02 13:29:43 UTC
[GitHub] [hive] akshat0395 opened a new pull request, #4012: HIVE-26954: Upgrade avro to 1.11.1
akshat0395 opened a new pull request, #4012:
URL: https://github.com/apache/hive/pull/4012
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Upgrade Avro to 1.11.1
[HIVE-26954](https://issues.apache.org/jira/browse/HIVE-26954)
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Keeping file formats to latest version
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Build + Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1422395470
@ayushtkn I am not aware of the history but I agree with what you are saying. If there are no proofs that it breaks something then I am fine proceeding with the upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] Aggarwal-Raghav commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1414104231
@akshat0395, **_one question_**: Shouldn't we need to be in-sync with Hadoop (3.3.1 used in hive) on avro version even though we are already ahead. Because difference in file format version might cause problem. Because I think the hadoop jars will be loaded first in classpath which will already contain hadoop avro version (1.7.7)?
There is some discussion on same for this on PR 3878 (HIVE-26829).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1414317737
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4012)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4012&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4012&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4012&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4012&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4012&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] akshat0395 commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "akshat0395 (via GitHub)" <gi...@apache.org>.
akshat0395 commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1439897739
Thanks @ayushtkn, @zabetak for the input. I've done some further testing and analysis for the same.
- I've ran a Java API compatibility test between the current avro version i.e 1.8.2 used by Hive and the target version 1.11.1. The compatibility is ~ 95%, There are few new APIs added which are safe to use and doesnt have impact on hive and there are few APIs that are deprecated, The deprecated API are already replaced as part of [HIVE-24324 ](https://issues.apache.org/jira/browse/HIVE-24324), PR: [1621](https://github.com/apache/hive/pull/1621) as @ayushtkn also mentioned. In conclusion compatibility wise the upgrade seems safe
- Manual Tests:
- I've triggered build and ran test with the upgraded version in an cluster and it passed. This PR also passed the test
- I've create an avro table with different datatypes with a build that have avro 1.8.2
`CREATE TABLE avro_table
STORED AS AVRO
TBLPROPERTIES ('avro.schema.literal'='{
"type": "record",
"name": "my_record",
"fields": [
{"name": "tinyint_col", "type": "int"},
{"name": "smallint_col", "type": "int"},
{"name": "int_col", "type": "int"},
{"name": "bigint_col", "type": "long"},
{"name": "float_col", "type": "float"},
{"name": "double_col", "type": "double"},
{"name": "decimal_col", "type": {"type": "bytes", "logicalType": "decimal", "precision": 10, "scale": 2}},
{"name": "string_col", "type": "string"},
{"name": "varchar_col", "type": "string"},
{"name": "char_col", "type": "string"},
{"name": "binary_col", "type": "bytes"},
{"name": "boolean_col", "type": "boolean"},
{"name": "array_col", "type": {"type": "array", "items": "string"}},
{"name": "map_col", "type": {"type": "map", "values": "int"}},
{"name": "struct_col", "type": {"type": "record", "name": "my_struct", "fields": [{"name": "field1", "type": "string"}, {"name": "field2", "type": "int"}]}}
]
}');`
I've performed some basic Insert, Modify, and aggregation query on the current version, post that I've performed the same set of queries with the upgraded version and both seems to work without any issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1415492628
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4012)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4012&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4012&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4012&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4012&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4012&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4012&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1440487335
@akshat0395 Are there (enough) tests already in Hive reading and writing from Avro files/tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] akshat0395 commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "akshat0395 (via GitHub)" <gi...@apache.org>.
akshat0395 commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1420618093
++ @ayushtkn
Would be really helpful if you can share some insights here. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] akshat0395 commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "akshat0395 (via GitHub)" <gi...@apache.org>.
akshat0395 commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1441261057
> @akshat0395 Are there (enough) tests already in Hive reading and writing from Avro files/tables?
There are notable number of q test present in Hive specific to avro that covers various read/write scenarios. Following is the list of q test:
1. avro_add_column.q
2. avro_add_column2.q
3. avro_add_column3.q
4. avro_add_column_extschema.q
5. avro_alter_table_update_columns.q
6. avro_change_schema.q
7. avro_charvarchar.q
8. avro_comments.q
9. avro_compression_enabled.q
10. avro_compression_enabled_native.q
11. avro_date.q
12. avro_decimal.q
13. avro_decimal_native.q
14. avro_decimal_old.q
15. avro_deserialize_map_null.q
16. avro_evolved_schemas.q
17. avro_extschema_insert.q
18. avro_historical_timestamp.q
19. avro_hybrid_mixed_date.q
20. avro_hybrid_mixed_timestamp.q
21. avro_joins.q
22. avro_joins_native.q
23. avro_legacy_mixed_date.q
24. avro_legacy_mixed_timestamp.q
25. avro_native.q
26. avro_nullable_fields.q
27. avro_nullable_union.q
28. avro_partitioned.q
29. avro_partitioned_native.q
30. avro_proleptic_mixed_date.q
31. avro_proleptic_mixed_timestamp.q
32. avro_sanity_test.q
33. avro_schema_evolution_native.q
34. avro_schema_literal.q
35. avro_tableproperty_optimize.q
36. avro_timestamp.q
37. avro_timestamp2.q
38. avro_type_evolution.q
39. avro_write_legacy_timestamp.q
40. avro_write_new_timestamp.q
41. avrocountemptytbl.q
42. avrotblsjoin.q
I think this cover alot of ground, Please let me know your thoughts in case we need to add more specific cases @zabetak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] akshat0395 commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "akshat0395 (via GitHub)" <gi...@apache.org>.
akshat0395 commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1415105579
> @akshat0395, **_one question_**: Shouldn't we need to be in-sync with Hadoop (3.3.1 used in hive) on avro version even though we are already ahead. Because difference in file format version might cause problem. Because I think the hadoop jars will be loaded first in classpath which will already contain hadoop avro version (1.7.7)?
>
> There is some discussion on same for this on PR 3878 ([HIVE-26829](https://issues.apache.org/jira/browse/HIVE-26829)).
Thanks @Aggarwal-Raghav for sharing these details, I wasn't aware that there is a similar discussion going on.
You made some valid points, I think the solution to which could be extensive testing.
Right now I've run some similar tests that you have done in your upgrade, which seems to work fine to me, and after going to the discussion I feel writing some qtest and some unit test around avro APIs that are used in Hive might help confirming if the upgrade is safe or not, Till then this can be marked in progress.
Regarding compatibility with Hadoop, as we are already ahead of Hadoop and the changes seems to be backward compatible I dont think upgrade should be a problem there.
Also I do agree with @cnauroth regrading the upgrade to 1.11.1 rather than 1.11.0.
@Aggarwal-Raghav @cnauroth Please let me know what do you think about the proposed testing solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1421325965
I am not sure what blocks us from moving forward, we have already diverged from hadoop, Hadoop is at 1.7.7
https://github.com/apache/hadoop/blob/branch-3.3.1/hadoop-project/pom.xml#L65-L66
And we are at 1.8.2 already. So, if anything has to break due to hadoop it should have been broken by now, but we haven't got hold of anything till now, the upgrade was done on branch-3 and branch-2 as well([HIVE-19662](https://issues.apache.org/jira/browse/HIVE-19662)), so divergence b/w hadoop and hive avro version is already there.
[HIVE-24324](https://issues.apache.org/jira/browse/HIVE-24324) was done to pave way for future upgrades.
The tests are green. @akshat0395 can you deploy and try some tests on an actual cluster, if things work. If we don't find any conclusive evidence like. "This breaks due to Avro upgrade", I think we can move forward.
@zabetak / @abstractdog any thoughts around this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #4012:
URL: https://github.com/apache/hive/pull/4012#issuecomment-1440489996
@akshat0395 Do we have (should we add) in the repo tests that are reading from Avro files written with old writers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak closed pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak closed pull request #4012: HIVE-26954: Upgrade avro to 1.11.1
URL: https://github.com/apache/hive/pull/4012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org