You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/09/07 09:09:36 UTC
[GitHub] [dolphinscheduler] sekikn opened a new pull request, #11832: [Improve] Upgrade Hadoop to 3.3.4
sekikn opened a new pull request, #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
## Purpose of the pull request
This PR enables to build DS against the latest version of Hadoop, 3.3.4.
## Brief change log
* Fix two classes under dolphinscheduler-api to use Apache HttpClient 4.x instead of the legacy Commons HttpClient, which was provided by the old version of Hadoop through transitive dependencies.
* Explicitly declare dependency on org.mortbay.jetty in dolphinscheduler-common/pom.xml, which is no longer provided by the recent version of Hadoop.
* Fix o.a.dolphinscheduler.server.master.processor.TaskExecuteStartProcessor not to use o.a.directory.api.util.Strings unnecessarily so that we can remove the dependency on org.apache.directory.api:api-util, which was provided by the old version of Hadoop through transitive dependencies.
* Upgrade the default version of Hadoop in dolphinscheduler-bom/pom.xml.
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is already covered by existing tests, such as LoginHandlerInterceptorTest for LoginHandlerInterceptor.
In addition, I made sure that both of the following command succeeded locally.
```
$ mvn clean install -Prelease -DskipTests -Djacoco.skip=true -Dspotless.check.skip=true -Dhadoop.version=2.7.3
$ mvn clean install -Prelease -DskipTests -Djacoco.skip=true -Dspotless.check.skip=true -Dhadoop.version=3.3.4
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on pull request #11832: [Improve] Upgrade Hadoop to 3.2.4
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1241883840
This is weird, the tests all passed and the packaged bin can run, let me take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 merged pull request #11832: [Improve] Upgrade Hadoop to 3.2.4
Posted by GitBox <gi...@apache.org>.
kezhenxu94 merged PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on pull request #11832: [Improve] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1239256178
> I'm not quite sure whether this have compatibility problems with lower versions since this version is too new.
I can imagine there might be compatibility issues, but there is critical CVEs in 2.x and we have to upgrade or remove Hadoop dependencies, we definitely don't want to receive CVE report over and over.
> The latest stable version of AWS EMR has only arrived Hadoop 3.2.1...
For the same major version 3.x (3.2.1 and 3.3.4), I think there should be OK, but newer version is preferred in my opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sekikn commented on pull request #11832: [Improve] Upgrade Hadoop to 3.2.4
Posted by GitBox <gi...@apache.org>.
sekikn commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1241323932
Currently, running `./mvnw dependency-check:aggregate` on the dev branch reports the following errors.
```
[ERROR] api-util-1.0.0-M20.jar: CVE-2018-1337(9.8)
[ERROR] avro-1.7.4.jar: CVE-2021-43045(7.5)
[ERROR] commons-compress-1.4.1.jar: CVE-2021-35517(7.5), CVE-2021-36090(7.5)
[ERROR] derby-10.10.2.0.jar: CVE-2015-1832(9.1)
[ERROR] gson-2.8.6.jar: CVE-2022-25647(7.5)
[ERROR] hadoop-common-2.7.7.jar: CVE-2022-25168(9.8)
[ERROR] hadoop-hdfs-2.7.7.jar: CVE-2020-9492(8.8), CVE-2018-11768(7.5)
[ERROR] hive-jdbc-2.3.3.jar: CVE-2021-34538(7.5), CVE-2021-4125(8.1), CVE-2018-11777(8.1), CVE-2020-13949(7.5)
[ERROR] hive-service-2.3.3.jar: CVE-2021-34538(7.5), CVE-2021-4125(8.1), CVE-2018-11777(8.1), CVE-2020-13949(7.5)
[ERROR] hive-storage-api-2.4.0.jar: CVE-2021-34538(7.5), CVE-2021-4125(8.1)
[ERROR] htrace-core-3.1.0-incubating.jar/META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml: CVE-2017-7525(9.8), CVE-2018-7489(9.8), CVE-2020-35491(8.1), CVE-2020-35490(8.1), CVE-2020-36518(7.5)
[ERROR] jackson-databind-2.13.0.jar: CVE-2020-36518(7.5)
[ERROR] jackson-mapper-asl-1.9.13.jar: CVE-2017-7525(9.8), CVE-2019-10172(7.5)
[ERROR] jetcd-core-0.5.11.jar: CVE-2020-15113(7.1)
[ERROR] jetty-client-9.4.31.v20200723.jar: CVE-2021-28165(7.5), CVE-2022-2048(7.5), CVE-2020-27216(7.0)
[ERROR] junit-platform-commons-1.6.2.jar: CVE-2022-31514(9.3)
[ERROR] junit-platform-commons-1.8.2.jar: CVE-2022-31514(9.3)
[ERROR] libfb303-0.9.3.jar: CVE-2016-5397(8.8), CVE-2018-1320(7.5), CVE-2019-0210(7.5), CVE-2020-13949(7.5), CVE-2019-0205(7.5)
[ERROR] libthrift-0.9.3.jar: CVE-2016-5397(8.8), CVE-2018-1320(7.5), CVE-2019-0210(7.5), CVE-2020-13949(7.5), CVE-2019-0205(7.5)
[ERROR] log4j-1.2-api-2.6.2.jar: CVE-2021-44832(6.6), CVE-2017-5645(9.8), CVE-2021-44228(10.0), CVE-2021-45046(9.0)
[ERROR] log4j-1.2.17.jar: CVE-2021-4104(7.5), CVE-2020-9493(9.8), CVE-2022-23307(8.8), CVE-2022-23305(9.8), CVE-2019-17571(9.8), CVE-2022-23302(8.8)
[ERROR] mybatis-plus-3.5.2.jar: CVE-2020-26945(8.1), CVE-2022-25517(9.8)
[ERROR] mybatis-plus-core-3.5.2.jar: CVE-2020-26945(8.1)
[ERROR] netty-3.6.2.Final.jar: CVE-2019-16869(7.5), CVE-2015-2156(7.5), CVE-2021-37136(7.5), CVE-2021-37137(7.5), CVE-2019-20445(9.1), CVE-2019-20444(9.1)
[ERROR] netty-transport-4.1.53.Final.jar: CVE-2021-37136(7.5), CVE-2021-37137(7.5)
[ERROR] netty-transport-4.1.63.Final.jar: CVE-2021-37136(7.5), CVE-2021-37137(7.5)
[ERROR] okhttp-3.12.12.jar: CVE-2021-0341(7.5)
[ERROR] okhttp-3.14.9.jar: CVE-2021-0341(7.5)
[ERROR] orc-core-1.3.3.jar: CVE-2018-8015(7.5)
[ERROR] pom.xml: CVE-2018-11804(7.5), CVE-2018-17190(9.8)
[ERROR] pom.xml: CVE-2018-11804(7.5), CVE-2018-17190(9.8)
[ERROR] snakeyaml-1.30.jar: CVE-2022-25857(7.5)
[ERROR] spring-cloud-starter-kubernetes-fabric8-config-2.1.3.jar: CVE-2020-5410(7.5)
[ERROR] spring-core-5.3.19.jar: CVE-2016-1000027(9.8)
[ERROR] spring-ldap-1.1.2.jar: CVE-2018-11040(7.5), CVE-2011-2730(7.5), CVE-2016-9878(7.5), CVE-2016-1000027(9.8), CVE-2018-1270(9.8), CVE-2022-22965(9.8)
[ERROR] spring-plugin-core-2.0.0.RELEASE.jar: CVE-2018-11040(7.5), CVE-2016-1000027(9.8), CVE-2022-22965(9.8)
[ERROR] spring-retry-1.3.1.jar: CVE-2018-11040(7.5), CVE-2016-1000027(9.8), CVE-2022-22965(9.8)
[ERROR] spring-retry-1.3.3.jar: CVE-2018-11040(7.5), CVE-2016-1000027(9.8), CVE-2022-22965(9.8)
[ERROR] spring-security-rsa-1.0.10.RELEASE.jar: CVE-2022-22978(9.8)
[ERROR] spring-tx-5.3.22.jar: CVE-2016-1000027(9.8)
[ERROR] websocket-client-9.4.31.v20200723.jar: CVE-2021-28165(7.5), CVE-2022-2048(7.5), CVE-2020-27216(7.0)
[ERROR] websocket-common-9.4.31.v20200723.jar: CVE-2021-28165(7.5), CVE-2022-2048(7.5), CVE-2020-27216(7.0)
[ERROR] xercesImpl-2.9.1.jar: CVE-2012-0881(7.5), CVE-2013-4002(7.1)
```
Compared with it, this PR removes the following vulnerabilities.
* api-util-1.0.0-M20.jar: CVE-2018-1337(9.8)
* commons-compress-1.4.1.jar: CVE-2021-35517(7.5), CVE-2021-36090(7.5)
* hadoop-common-2.7.7.jar: CVE-2022-25168(9.8)
* hadoop-hdfs-2.7.7.jar: CVE-2020-9492(8.8), CVE-2018-11768(7.5)
* netty-3.6.2.Final.jar: CVE-2019-16869(7.5), CVE-2015-2156(7.5), CVE-2021-37136(7.5), CVE-2021-37137(7.5), CVE-2019-20445(9.1), CVE-2019-20444(9.1)
* xercesImpl-2.9.1.jar: CVE-2012-0881(7.5), CVE-2013-4002(7.1)
On the other hand, this PR newly introduces the following vulnerabilities. But their scores are relatively lower than the removed ones and 3 out of 4 are duplicated with the ones already reported.
* jetty-server-9.4.43.v20210629.jar: CVE-2022-2048(7.5)
* jetty-util-9.4.43.v20210629.jar: CVE-2022-2048(7.5)
* nimbus-jose-jwt-9.8.1.jar/META-INF/maven/net.minidev/json-smart/pom.xml: CVE-2021-31684(7.5)
* okhttp-2.7.5.jar: CVE-2021-0341(7.5)
Given that, I think we can consider this PR improves our security, though [the OWASP check failed on CI](https://github.com/apache/dolphinscheduler/runs/8252203049?check_suite_focus=true).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on pull request #11832: [Improve] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1239143203
> Hi @sekikn , Why did you want to upgrade hadoop to the latest version?
I'm glad to see this as an addition to https://github.com/apache/dolphinscheduler/pull/11765 because there are multiple CVEs in Hadoop 2.7.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sekikn commented on pull request #11832: [Improve] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
sekikn commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1239412881
Thank you for the comments, @SbloodyS @kezhenxu94!
> I'm not quite sure whether this have compatibility problems with lower versions since this version is too new. The latest stable version of AWS EMR has only arrived Hadoop 3.2.1...
> For the same major version 3.x (3.2.1 and 3.3.4), I think there should be OK, but newer version is preferred in my opinion.
Taking both of your comments into consideration, I'd like to adopt Hadoop 3.2.4 rather than 3.3.4 here. That is the latest version of the 3.2.x line and was released just a few weeks before 3.3.4.
In addition to that, according to the CI results, we have to update known-dependencies.txt and upgrade the spark-hive version at least. I'll update this PR later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sekikn commented on pull request #11832: [Improve] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
sekikn commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1240830712
Thanks for the notification @kezhenxu94, just resolved conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #11832: [Improve] Upgrade Hadoop to 3.2.4
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1241000194
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=11832)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11832&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11832&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11832&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11832&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #11832: [Improve] Upgrade Hadoop to 3.2.4
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1240995591
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=11832)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11832&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11832&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11832&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11832&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11832&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #11832: [Improve] Upgrade Hadoop to 3.2.4
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1241871581
```
2022-09-09 11:23:40.346 +0000 INFO [org.apache.zookeeper.server.BlueThrottle] - Weighed connection throttling is disabled
2022-09-09 11:23:40.359 +0000 INFO [org.apache.zookeeper.server.util.RequestPathMetricsCollector] - zookeeper.pathStats.slotCapacity = 60
2022-09-09 11:23:40.359 +0000 INFO [org.apache.zookeeper.server.util.RequestPathMetricsCollector] - zookeeper.pathStats.slotDuration = 15
2022-09-09 11:23:40.359 +0000 INFO [org.apache.zookeeper.server.util.RequestPathMetricsCollector] - zookeeper.pathStats.maxDepth = 6
2022-09-09 11:23:40.359 +0000 INFO [org.apache.zookeeper.server.util.RequestPathMetricsCollector] - zookeeper.pathStats.initialDelay = 5
2022-09-09 11:23:40.359 +0000 INFO [org.apache.zookeeper.server.util.RequestPathMetricsCollector] - zookeeper.pathStats.delay = 5
2022-09-09 11:23:40.359 +0000 INFO [org.apache.zookeeper.server.util.RequestPathMetricsCollector] - zookeeper.pathStats.enabled = false
2022-09-09 11:23:40.365 +0000 INFO [org.apache.zookeeper.server.AuthenticationHelper] - zookeeper.enforce.auth.enabled = false
2022-09-09 11:23:40.365 +0000 INFO [org.apache.zookeeper.server.AuthenticationHelper] - zookeeper.enforce.auth.schemes = []
2022-09-09 11:23:40.365 +0000 INFO [org.apache.zookeeper.server.ZooKeeperServer] - tickTime set to 1000 ms
2022-09-09 11:23:40.365 +0000 INFO [org.apache.zookeeper.server.ZooKeeperServer] - minSessionTimeout set to 1000 ms
2022-09-09 11:23:40.365 +0000 INFO [org.apache.zookeeper.server.ZooKeeperServer] - maxSessionTimeout set to 20000 ms
2022-09-09 11:23:40.374 +0000 INFO [org.apache.zookeeper.server.ServerCnxnFactory] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2022-09-09 11:23:40.375 +0000 WARN [org.apache.zookeeper.server.ServerCnxnFactory] - maxCnxns is not configured, using default value 0.
2022-09-09 11:23:40.387 +0000 INFO [org.apache.zookeeper.server.NIOServerCnxnFactory] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers.
2022-09-09 11:23:40.395 +0000 INFO [org.apache.zookeeper.server.NIOServerCnxnFactory] - binding to port 0.0.0.0/0.0.0.0:43777
2022-09-09 11:23:40.431 +0000 INFO [org.apache.zookeeper.server.watch.WatchManagerFactory] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2022-09-09 11:23:40.432 +0000 INFO [org.apache.zookeeper.server.watch.WatchManagerFactory] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2022-09-09 11:23:40.432 +0000 INFO [org.apache.zookeeper.server.ZKDatabase] - zookeeper.snapshotSizeFactor = 0.33
2022-09-09 11:23:40.432 +0000 INFO [org.apache.zookeeper.server.ZKDatabase] - zookeeper.commitLogCount=500
2022-09-09 11:23:40.488 +0000 INFO [org.apache.zookeeper.server.persistence.SnapStream] - zookeeper.snapshot.compression.method = CHECKED
2022-09-09 11:23:40.488 +0000 INFO [org.apache.zookeeper.server.persistence.FileTxnSnapLog] - Snapshotting: 0x0 to /tmp/1662722620245-0/version-2/snapshot.0
2022-09-09 11:23:40.497 +0000 INFO [org.apache.zookeeper.server.ZKDatabase] - Snapshot loaded in 65 ms, highest zxid is 0x0, digest is 1371985504
2022-09-09 11:23:40.501 +0000 INFO [org.apache.zookeeper.server.persistence.FileTxnSnapLog] - Snapshotting: 0x0 to /tmp/1662722620245-0/version-2/snapshot.0
2022-09-09 11:23:40.501 +0000 INFO [org.apache.zookeeper.server.ZooKeeperServer] - Snapshot taken in 0 ms
2022-09-09 11:23:40.511 +0000 INFO [org.apache.zookeeper.server.PrepRequestProcessor] - PrepRequestProcessor (sid:0) started, reconfigEnabled=false
2022-09-09 11:23:40.512 +0000 INFO [org.apache.zookeeper.server.RequestThrottler] - zookeeper.request_throttler.shutdownTimeout = 10000 ms
2022-09-09 11:23:40.526 +0000 INFO [org.apache.zookeeper.server.ContainerManager] - Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0
2022-09-09 11:23:40.800 +0000 ERROR [org.apache.zookeeper.server.NIOServerCnxnFactory] - Thread Thread[main,5,main] died
java.lang.IllegalArgumentException: LoggerFactory is not a Logback LoggerContext but Logback is on the classpath. Either remove Logback or the competing implementation (class org.slf4j.impl.Reload4jLoggerFactory loaded from file:/home/csf/.m2/repository/org/slf4j/slf4j-reload4j/1.7.35/slf4j-reload4j-1.7.35.jar). If you are using WebLogic you will need to add 'org.slf4j' to prefer-application-packages in WEB-INF/weblogic.xml: org.slf4j.impl.Reload4jLoggerFactory
at org.springframework.util.Assert.instanceCheckFailed(Assert.java:702)
at org.springframework.util.Assert.isInstanceOf(Assert.java:621)
at org.springframework.boot.logging.logback.LogbackLoggingSystem.getLoggerContext(LogbackLoggingSystem.java:294)
at org.springframework.boot.logging.logback.LogbackLoggingSystem.beforeInitialize(LogbackLoggingSystem.java:118)
at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationStartingEvent(LoggingApplicationListener.java:238)
at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:220)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131)
at org.springframework.boot.context.event.EventPublishingRunListener.starting(EventPublishingRunListener.java:79)
at org.springframework.boot.SpringApplicationRunListeners.lambda$starting$0(SpringApplicationRunListeners.java:56)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120)
at org.springframework.boot.SpringApplicationRunListeners.starting(SpringApplicationRunListeners.java:56)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:299)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
at org.apache.dolphinscheduler.StandaloneServer.main(StandaloneServer.java:45)
```
I found that it can not start `standalone-server` in IDEA, throw above exception of `slf4j-reload4j`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on pull request #11832: [Improve] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1239188629
> @SbloodyS I'm mainly working on Apache Bigtop project, and I heard someone wants to add DolphinScheduler to Bigtop as a component. Bigtop uses Hadoop 3.x, but currently DS can't be built with it. So I'd like to help him by making DS compatible with Hadoop 3.x. If setting the default version to 3.3.4 is unwanted, it's OK with me to stay in 2.7.3, since I only want DS to be built with Hadoop 3.
I'm not quite sure whether this have compatibility problems with lower versions since this version is too new. The latest stable version of AWS has only arrived Hadoop 3.2.1...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on pull request #11832: [Improve] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1240554931
There are some conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on pull request #11832: [Improve] Upgrade Hadoop to 3.2.4
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1242607229
> I found that it can not start `standalone-server` in IDEA, throw above exception of `slf4j-reload4j`.
Sad...I also encounterd the same problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sekikn commented on pull request #11832: [Improve] Upgrade Hadoop to 3.3.4
Posted by GitBox <gi...@apache.org>.
sekikn commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1239139126
@SbloodyS I'm mainly working on Apache Bigtop project, and I heard someone wants to add DolphinScheduler to Bigtop as a component. Bigtop uses Hadoop 3.x, but currently DS can't be built with it. So I'd like to help him by making DS compatible with Hadoop 3.x.
If setting the default version to 3.3.4 is unwanted, it's OK with me to stay in 2.7.3, since I only want DS to be built with Hadoop 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on pull request #11832: [Improve] Upgrade Hadoop to 3.2.4
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1242593606
> This is weird, the tests all passed and the packaged bin can run, let me take a look
Fix it in https://github.com/apache/dolphinscheduler/pull/11881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on pull request #11832: [Improve] Upgrade Hadoop to 3.2.4
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #11832:
URL: https://github.com/apache/dolphinscheduler/pull/11832#issuecomment-1242618495
Fix it in https://github.com/apache/dolphinscheduler/pull/11881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org