You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2023/04/21 03:37:43 UTC
[GitHub] [spark] wangyum opened a new pull request, #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl jackson-mapper-asl from pre-built distribution
wangyum opened a new pull request, #40893:
URL: https://github.com/apache/spark/pull/40893
### What changes were proposed in this pull request?
- Remove `jackson-core-asl` from maven dependency.
- Change the scope of `jackson-mapper-asl` from compile to test.
- Replace all `Hive.get(conf)` with `Hive.getWithoutRegisterFns(conf)`.
### Why are the changes needed?
To fix CVE issue: https://github.com/apache/spark/security/dependabot/50.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen closed pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
URL: https://github.com/apache/spark/pull/40893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1517997626
Is this possible now that Hadoop 2 support is gone? just checking what the implications of this change are.
Are the Hive.get changes needed, or can we batch those changes with reverting the Hive <2.3.9 support? I also don't know what the implication of that is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bjornjorgensen commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1517500449
CC @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1521073105
> OK, am I right that this does not make Spark any _less_ compatible with any version of Hive that is currently supported (>= 2.3.9)? If so then this is fine
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution [spark]
Posted by "Madhukar525722 (via GitHub)" <gi...@apache.org>.
Madhukar525722 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-2737367757
Hi @dongjoon-hyun
I tried replicating SPARK-44719, it failed with hive 2.3.9, as expected. Once i upgraded inbuilt hive 2.3.10 and removed jackson-core-asl and jackson-mapper-asl, the issue was not coming. Are there more identified similar scenarios of failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-2585918357
Hi, @Madhukar525722 . No, we can't because the existing users have Hive UDF jars which are built against old Hive 2.3.9 and older. Technically, we cannot enforce Apache Spark users to rebuilt every Hive UDF jars in their company. Actually, no user wants to rebuild all their Hive UDF jars against on newly arrived Hive 2.3.10.
> HI @pan3793 @dongjoon-hyun . Since, the spark 4.0.0 has been upgraded to hive 2.3.10 and hadoop 3.3.6. Cant we now implement this change back to remove jackson-mapper-asl and jackson-core-asl from pre-built distribution
However, Apache Spark community provides an alternative way in order to remove them from `jars` directory at Apache Spark 4.0.0 `jars` directory by isolating them into another directory `hive-jackson`.
- #45237
- #45201
Here are the difference between Apache Spark 4.0.0-preview2 and Apache Spark 3.5.4 .
```
$ ls spark-3.5.4-bin-hadoop3/jars/*asl*
spark-3.5.4-bin-hadoop3/jars/jackson-core-asl-1.9.13.jar spark-3.5.4-bin-hadoop3/jars/jackson-mapper-asl-1.9.13.jar
$ ls spark-4.0.0-preview2-bin-hadoop3/jars/*asl*
zsh: no matches found: spark-4.0.0-preview2-bin-hadoop3/jars/*asl*
```
Please try to use Apache Spark 4.0.0-preview2 and let us know your pain point, @Madhukar525722 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1521831422
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1519004897
@sunchao can we expect a new release(focus on security) for Hive 2.3? Considering Spark master and all maintained branches use Hive 2.3.9, which was reported some CVEs e.g. thrift, guava, log4j, jackson.
Or, Spark should move forward to a new Hive version. (should take much effort and not sure of benefits other than getting rid of CVEs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution [spark]
Posted by "Madhukar525722 (via GitHub)" <gi...@apache.org>.
Madhukar525722 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-2585621542
HI @pan3793 @dongjoon-hyun . Since, the spark 4.0.0 has been upgraded to hive 2.3.10 and hadoop 3.3.6. Cant we now implement this change back to remove jackson-mapper-asl and jackson-core-asl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-2740565445
@Madhukar525722 I think either upgrading Hive 2.3.10 or applying SPARK-51466 should allow Spark to get rid of `jackson-core-asl` and `jackson-mapper-asl`, see tech details at https://github.com/apache/spark/pull/46521 and https://github.com/apache/spark/pull/50232.
"SPARK-48231 Remove unused CodeHaus Jackson dependencies" is on the 4.1 roadmap. (I think it also could be done in 4.0, and we do remove them in our internal 4.0 distribution, but given 4.0.0 has reached RC3, might be too late)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1521012401
OK, am I right that this does not make Spark any _less_ compatible with any version of Hive that is currently supported (>= 2.3.9)? If so then this is fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1520573329
@pan3793 AFAIK the development efforts in Hive community are only in Hive 3.x/4.x at the moment, and the 2.x branch is barely maintained. I can try to start a conversation in the Hive community to have a new 2.3.10 release and see how it looks like.
From the long term perspective, it'd be better for Spark to move to Hive 3.x/4.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-2740693101
Yes, @pan3793's comment is correct with the current status. We need to revisit this at 4.1.0, @Madhukar525722 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1518994829
@srowen
> Are the `Hive.get` changes needed
Yes, `Hive.get(conf)` triggers the Hive built-in JSON functions initialization, which requires the Jackson 1.x classes.
@sunchao I suppose Spark does not officially support building against Hive other than 2.3.9, for cases listed in SPARK-37446, it's the vendor's responsibility to port HIVE-21563 into their maintained Hive 2.3.8-[vender-custom-version]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #40893: [SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #40893:
URL: https://github.com/apache/spark/pull/40893#issuecomment-1517218312
It drops support for building w/ pre Hive 2.3.9, then SPARK-37446 can be reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org