You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2023/04/21 03:37:43 UTC
[GitHub] [spark] wangyum opened a new pull request, #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl jackson-mapper-asl from pre-built distribution
wangyum opened a new pull request, #40893:
URL: https://github.com/apache/spark/pull/40893
### What changes were proposed in this pull request?
- Remove `jackson-core-asl` from maven dependency.
- Change the scope of `jackson-mapper-asl` from compile to test.
- Replace all `Hive.get(conf)` with `Hive.getWithoutRegisterFns(conf)`.
### Why are the changes needed?
To fix CVE issue: https://github.com/apache/spark/security/dependabot/50.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1521073105
> OK, am I right that this does not make Spark any _less_ compatible with any version of Hive that is currently supported (>= 2.3.9)? If so then this is fine
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1521831422
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen closed pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
URL: https://github.com/apache/spark/pull/40893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1519004897
@sunchao can we expect a new release(focus on security) for Hive 2.3? Considering Spark master and all maintained branches use Hive 2.3.9, which was reported some CVEs e.g. thrift, guava, log4j, jackson.
Or, Spark should move forward to a new Hive version. (should take much effort and not sure of benefits other than getting rid of CVEs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1521012401
OK, am I right that this does not make Spark any _less_ compatible with any version of Hive that is currently supported (>= 2.3.9)? If so then this is fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1520573329
@pan3793 AFAIK the development efforts in Hive community are only in Hive 3.x/4.x at the moment, and the 2.x branch is barely maintained. I can try to start a conversation in the Hive community to have a new 2.3.10 release and see how it looks like.
From the long term perspective, it'd be better for Spark to move to Hive 3.x/4.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1517997626
Is this possible now that Hadoop 2 support is gone? just checking what the implications of this change are.
Are the Hive.get changes needed, or can we batch those changes with reverting the Hive <2.3.9 support? I also don't know what the implication of that is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bjornjorgensen commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1517500449
CC @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1518994829
@srowen
> Are the `Hive.get` changes needed
Yes, `Hive.get(conf)` triggers the Hive built-in JSON functions initialization, which requires the Jackson 1.x classes.
@sunchao I suppose Spark does not officially support building against Hive other than 2.3.9, for cases listed in SPARK-37446, it's the vendor's responsibility to port HIVE-21563 into their maintained Hive 2.3.8-[vender-custom-version]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1517218312
It drops support for building w/ pre Hive 2.3.9, then SPARK-37446 can be reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org