You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "Mrhs121 (via GitHub)" <gi...@apache.org> on 2025/09/09 06:15:40 UTC
[I] [Bug] [Module Name] Since the plugin was changed to shade after 3.3.x, but the shade jar was not downloaded in the install script, it led to a NoClassDefFoundError [dolphinscheduler]
Mrhs121 opened a new issue, #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
1. download 3.3.1 released package
2. run install-pulgin.sh
3. run ./bin/dolphinscheduler-daemon.sh start master-server
then get the error
`Caused by: java.lang.NoClassDefFoundError: com/microsoft/sqlserver/jdbc/SQLServerException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 31 common frames omitted
Caused by: java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerException
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 36 common frames omitted`
### What you expected to happen
The root cause is that after version 3.3.x the plugin was switched to shade mode; however, the JAR downloaded by install-plugin.sh is not the shaded jar, so several dependencies are missing from the classpath. In the above error, for example, mssql-jdbc.jar is absent. When dolphinscheduler-datasource-azure-sql.jar is placed under plugins/datasource-plugins, the Azure datasource needs the SQLServerDataSource and SQLServerException classes, but because mssql-jdbc.jar is still not on the classpath, an exception is thrown.
### How to reproduce
For details, please refer to this discussion [https://github.com/apache/dolphinscheduler/discussions/17487](https://github.com/apache/dolphinscheduler/discussions/17487)
### Anything else
_No response_
### Version
dev
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273220556
> I can successfully start `api-server` and `master-server` and `worker-server` using the following `plugins_config`. And my installation package `apache-dolphinscheduler-3.3.1-bin.tar.gz ` has just been downloaded from official website. So I can not reproduce your issue. Please provide your reproducing steps in details.
>
> ```
> --alert-plugins--
> --end--
>
> --datasource-plugins--
> --end--
>
> --storage-plugins--
> dolphinscheduler-storage-hdfs
> --end--
>
> --task-plugins--
> dolphinscheduler-task-shell
> --end--
> ```
你是启动的worker吗，这个问题只有在启动worker的时候才会出现 ./bin/dolphinscheduler-daemon.sh start worker-server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 closed issue #17488: [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown
URL: https://github.com/apache/dolphinscheduler/issues/17488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273031121
> Hbase dependency is used by `dolphinscheduler-datasource-hive`. Maybe This may be an abnormal reference, and hive should not rely on this. You can remove it from `plugins_config` since hive datasource are useless.
This hbase-noop-htrace dependency replaced htrace-core with hbase-noop-htrace to resolve dependency conflicts. The original commit is here https://github.com/apache/dolphinscheduler/issues/12126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273007500
Hbase dependency is used by `dolphinscheduler-datasource-hive`. Maybe This may be an abnormal reference, and hive should not rely on this. You can remove it from `plugins_config` since hive datasource are useless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3272991706
> [@SbloodyS](https://github.com/SbloodyS) Could you please help confirm this htrace dependency issue when you have time? It seems like a regression bug. Thank you
Please provide your `plugins_config`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3272998792
> > [@SbloodyS](https://github.com/SbloodyS) Could you please help confirm this htrace dependency issue when you have time? It seems like a regression bug. Thank you
>
> Please provide your `plugins_config`.
The configuration file is the default configuration and no modifications have been made.
```
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# This mapping is used to resolve the Jar package name without version (or call artifactId)
#
# corresponding to the module in the user Config, helping Dolphinscheduler to load the correct Jar package.
# Don't modify the delimiter " -- ", just select the plugin you need
--alert-plugins--
dolphinscheduler-alert-aliyunVoice
dolphinscheduler-alert-dingtalk
dolphinscheduler-alert-email
dolphinscheduler-alert-feishu
dolphinscheduler-alert-http
dolphinscheduler-alert-pagerduty
dolphinscheduler-alert-prometheus
dolphinscheduler-alert-script
dolphinscheduler-alert-slack
dolphinscheduler-alert-telegram
dolphinscheduler-alert-webexteams
dolphinscheduler-alert-wechat
--end--
--datasource-plugins--
dolphinscheduler-datasource-aliyunserverlessspark
dolphinscheduler-datasource-athena
dolphinscheduler-datasource-azure-sql
dolphinscheduler-datasource-clickhouse
dolphinscheduler-datasource-dameng
dolphinscheduler-datasource-databend
dolphinscheduler-datasource-db2
dolphinscheduler-datasource-doris
dolphinscheduler-datasource-hana
dolphinscheduler-datasource-hive
dolphinscheduler-datasource-k8s
dolphinscheduler-datasource-kyuubi
dolphinscheduler-datasource-mysql
dolphinscheduler-datasource-oceanbase
dolphinscheduler-datasource-oracle
dolphinscheduler-datasource-postgresql
dolphinscheduler-datasource-presto
dolphinscheduler-datasource-redshift
dolphinscheduler-datasource-sagemaker
dolphinscheduler-datasource-snowflake
dolphinscheduler-datasource-spark
dolphinscheduler-datasource-sqlserver
dolphinscheduler-datasource-ssh
dolphinscheduler-datasource-starrocks
dolphinscheduler-datasource-trino
dolphinscheduler-datasource-vertica
dolphinscheduler-datasource-zeppelin
--end--
--storage-plugins--
dolphinscheduler-storage-abs
dolphinscheduler-storage-gcs
dolphinscheduler-storage-hdfs
dolphinscheduler-storage-obs
dolphinscheduler-storage-oss
dolphinscheduler-storage-s3
--end--
--task-plugins--
dolphinscheduler-task-aliyunserverlessspark
dolphinscheduler-task-chunjun
dolphinscheduler-task-datafactory
dolphinscheduler-task-datasync
dolphinscheduler-task-datax
dolphinscheduler-task-dinky
dolphinscheduler-task-dms
dolphinscheduler-task-dvc
dolphinscheduler-task-emr
dolphinscheduler-task-flink
dolphinscheduler-task-flink-stream
dolphinscheduler-task-hivecli
dolphinscheduler-task-http
dolphinscheduler-task-java
dolphinscheduler-task-jupyter
dolphinscheduler-task-k8s
dolphinscheduler-task-kubeflow
dolphinscheduler-task-linkis
dolphinscheduler-task-mlflow
dolphinscheduler-task-mr
dolphinscheduler-task-openmldb
dolphinscheduler-task-procedure
dolphinscheduler-task-python
dolphinscheduler-task-pytorch
dolphinscheduler-task-remoteshell
dolphinscheduler-task-sagemaker
dolphinscheduler-task-seatunnel
dolphinscheduler-task-shell
dolphinscheduler-task-spark
dolphinscheduler-task-sql
dolphinscheduler-task-sqoop
dolphinscheduler-task-zeppelin
--end--
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273034070
The plugin that minimizes running is `dolphinscheduler-storage-hdfs` and `dolphinscheduler-task-shell`. You can run it in a minimal way, and then install it one by one according to your actual needs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273481889
I've successfully reproduce this issue. You can try execute this shell script `cd worker-server/libs && n -s ../../libs/hbase-noop-htrace-4.1.1.jar hbase-noop-htrace-4.1.1.jar` then worker can be successfully started. I'll fix it ASAP. Thanks for your issue. @Mrhs121
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Gallardot (via GitHub)" <gi...@apache.org>.
Gallardot closed issue #17488: [Bug] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown
URL: https://github.com/apache/dolphinscheduler/issues/17488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273395985
It seems that other users have also encountered this problem ： https://github.com/apache/dolphinscheduler/issues/17370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273920540
> Will this fix be included in a release version?
This will be released in the version `3.3.2`.
> It's unacceptable for a bug that blocks the main process to be present in the latest stable version.
This bug will only be encountered by users when using HDFS, and it will not be the case if using other storage types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3269073065
Due to the license, we won't include any third-party jar packages that don't conform to apache license. At the same time, we will not use the shade package, which is reserved for the next big version and needs to be optimized, and will not be used in this version. Users need to manually download jdbc dependencies of related data sources.This operation is not indicated in the document, I will supplement the document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273092975
> The plugin that minimizes running is `dolphinscheduler-storage-hdfs` and `dolphinscheduler-task-shell`. You can run it in a minimal way, and then install it one by one according to your actual needs.
Through the above the stack can be found, **even in the minimizes of the running configuration, the worker also is unable to start**, because in the process of starting the worker need to initialize the `org.apache.hadoop.fs.FileSystem`, In the initialization `org.apache.hadoop.fs.FileSystem` process depend on the `org.apache.htrace.core.Tracer.Builder`, And `org.apache.htrace.core.Tracer.Builder` from `hbase-noop-htrace` jar.
So it can be seen that because there is no hbase-noop-htrace` jar in the classpath for starting the worker, the above error is caused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3272921942
> **The following another NoClassDefFoundError also needs attention. The root cause is the missing dependency "org.apache.hbase.thirdparty:hbase-noop-htrace" when starting the worker.**
>
> `java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3370) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:125) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3424) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3392) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:485) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:233) at org.apache.dolphinscheduler.plugin.storage.hdfs.HdfsStorageOperator.init(HdfsStorageOperator.java:136) at org.apache.dolphinscheduler.plugin.storage.hdfs.HdfsStorageOperator.<init>(HdfsStorageOperator.java:75) at org.apache.dolphinscheduler.plugin.storage.hdfs.LocalStorageOperator.<init>(LocalStorageOperator.java:26) `
>
> In version 3.2.2, this dependency comes from dolphinscheduler-task-all, but in 3.3.x it is set to provided, so it does not exist in the classpath for starting the worker
>
> `mvn dependency:tree -Dverbose -Dincludes='org.apache.hbase.thirdparty:hbase-noop-htrace'`
>
> ```
> [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ dolphinscheduler-worker ---
> [INFO] Verbose not supported since maven-dependency-plugin 3.0
> [INFO] org.apache.dolphinscheduler:dolphinscheduler-worker:jar:3.2.2
> [INFO] \- org.apache.dolphinscheduler:dolphinscheduler-task-all:jar:3.2.2:compile
> [INFO] \- org.apache.dolphinscheduler:dolphinscheduler-task-datax:jar:3.2.2:compile
> [INFO] \- org.apache.dolphinscheduler:dolphinscheduler-datasource-api:jar:3.2.2:compile
> [INFO] \- org.apache.hbase.thirdparty:hbase-noop-htrace:jar:4.1.1:compile
> [INFO]
> ```
>
> <img alt="Image" width="665" height="219" src="https://private-user-images.githubusercontent.com/15990723/487140029-a8e935ce-abed-4105-836f-3d30214fa870.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTc0Njk2OTUsIm5iZiI6MTc1NzQ2OTM5NSwicGF0aCI6Ii8xNTk5MDcyMy80ODcxNDAwMjktYThlOTM1Y2UtYWJlZC00MTA1LTgzNmYtM2QzMDIxNGZhODcwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwOTEwVDAxNTYzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRkZjk2M2QzMmMzMTU5MzU4YThlMmRhMTE4ZTg2YzgzNjAwNWMyMjc5Y2QwODNiYjQ4OWI0MjAwNmFiZjM0ZDAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.8HXtV17i30QTDJAEKFM1EsRBqS0LdrQderRnT7U1T5I">
> ```
> [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ dolphinscheduler-worker ---
> [INFO] Verbose not supported since maven-dependency-plugin 3.0
> [INFO] org.apache.dolphinscheduler:dolphinscheduler-worker:jar:dev-SNAPSHOT
> [INFO] \- org.apache.dolphinscheduler:dolphinscheduler-task-all:jar:dev-SNAPSHOT:provided
> [INFO] \- org.apache.dolphinscheduler:dolphinscheduler-task-aliyunserverlessspark:jar:dev-SNAPSHOT:provided
> [INFO] \- org.apache.dolphinscheduler:dolphinscheduler-datasource-all:jar:dev-SNAPSHOT:provided
> [INFO] \- org.apache.dolphinscheduler:dolphinscheduler-datasource-hive:jar:dev-SNAPSHOT:provided
> [INFO] \- org.apache.hbase.thirdparty:hbase-noop-htrace:jar:4.1.1:provided
> [INFO]
> ```
>
> <img alt="Image" width="629" height="114" src="https://private-user-images.githubusercontent.com/15990723/487140089-8c1ec93f-a7ed-4ca8-a3ab-0e75f74b7ee9.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTc0Njk2OTUsIm5iZiI6MTc1NzQ2OTM5NSwicGF0aCI6Ii8xNTk5MDcyMy80ODcxNDAwODktOGMxZWM5M2YtYTdlZC00Y2E4LWEzYWItMGU3NWY3NGI3ZWU5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwOTEwVDAxNTYzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTgwYWFiYjkwNzAxOGNmNmY1MWYwZWNmYmY0YzliYmJlNDEzMTBlN2M3NzE5ZGZkNWZlYjRhOTE2NGYxMmZkOTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.SfF0gBtNX-MYcaf2s-ul25x8OSZAOkYRUVs9JyD8gck">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273017543
The full stack is as follows, which can be seen the reference `org.apache.htrace.core.Tracer.Builder` comes from the call chain of `FileSystem.get(configuration)` in `HdfsStorageOperator`
```
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3370)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:125)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3424)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3392)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:485)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:233)
at org.apache.dolphinscheduler.plugin.storage.hdfs.HdfsStorageOperator.init(HdfsStorageOperator.java:135)
at org.apache.dolphinscheduler.plugin.storage.hdfs.HdfsStorageOperator.<init>(HdfsStorageOperator.java:75)
at org.apache.dolphinscheduler.plugin.storage.hdfs.LocalStorageOperator.<init>(LocalStorageOperator.java:26)
at org.apache.dolphinscheduler.plugin.storage.hdfs.LocalStorageOperatorFactory.createStorageOperate(LocalStorageOperatorFactory.java:36)
at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration.lambda$storageOperate$0(StorageConfiguration.java:43)
at java.util.Optional.map(Optional.java:215)
at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration.storageOperate(StorageConfiguration.java:38)
at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$ae6b1c9f.CGLIB$storageOperate$0(<generated>)
at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$ae6b1c9f$$FastClassBySpringCGLIB$$85175d46.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
at org.apache.dolphinscheduler.plugin.storage.api.StorageConfiguration$$EnhancerBySpringCGLIB$$ae6b1c9f.storageOperate(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
... 62 common frames omitted
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273195991
补充一点额外的信息，ds依赖的hadoop版本是3.2.4，3.2.4的Hadoop FileSystem 依赖的Tracer是org.apache.htrace.core.Tracer; https://github.com/apache/hadoop/blob/abe5358143720085498613d399be3bbf01e0f131/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L74C1-L74C38， 3.3.2 之后才改为org.apache.hadoop.tracing.Tracer的 https://github.com/apache/hadoop/blob/f91452b289aea1418f56d242c046b58d9f214a1d/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L86C8-L86C31
讲道理，Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder 这个问题应该是必现的啊，cp中确实找不到Tracer的jar啊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273145413
I can successfully start `api-server` and `master-server` and `worker-server` using the following `plugins_config`. And my installation package `apache-dolphinscheduler-3.3.1-bin.tar.gz ` has just been downloaded from official website. So I can not reproduce your issue. Please provide your reproducing steps in details.
```
--alert-plugins--
--end--
--datasource-plugins--
--end--
--storage-plugins--
dolphinscheduler-storage-hdfs
--end--
--task-plugins--
dolphinscheduler-task-shell
--end--
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "susanbushisan (via GitHub)" <gi...@apache.org>.
susanbushisan commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273910069
> > My suggestion is to add the dependence in the worker-server's pom.
>
> Yes. I've created PR [#17496](https://github.com/apache/dolphinscheduler/pull/17496) to fix it.
>
> > This issue can be closed now.
>
> We will close this issue until [#17496](https://github.com/apache/dolphinscheduler/pull/17496) is merged.
Will this fix be included in a release version? It's unacceptable for a bug that blocks the main process to be present in the latest stable version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3272936360
@SbloodyS Could you please help confirm this htrace dependency issue when you have time? It seems like a regression bug. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273510197
> I've successfully reproduce this issue. You can try execute this shell script `cd worker-server/libs && ln -s ../../libs/hbase-noop-htrace-4.1.1.jar hbase-noop-htrace-4.1.1.jar` then worker can be successfully started. I'll fix it ASAP. Thanks for your issue. [@Mrhs121](https://github.com/Mrhs121)
Yes, my current workaround solution is to copy hbase-noop-htrace-4.1.1.jar to worker-server/libs.
My suggestion is to add the dependence in the worker-server's pom. (This dependency exists in the pom of the master server)
```
<dependency>
<groupId>org.apache.hbase.thirdparty</groupId>
<artifactId>hbase-noop-htrace</artifactId>
</dependency>
```
By the way, I think the impact range of DSIP-38 might be a bit wide. It's not ruled out that there might be other issues that haven't been exposed yet.
This issue can be closed now. Thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273514744
> My suggestion is to add the dependence in the worker-server's pom.
Yes. I've created PR #17496 to fix it.
> This issue can be closed now.
We will close this issue until #17496 is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273005813
The reproduction steps are also very simple
1. download 3.3.x released package
2. run install-pulgin.sh
3. run ./bin/dolphinscheduler-daemon.sh start worker-server
Then you will see this error message `java.lang.ClassNotFoundException: org.apache.htrace.core.Tracer$Builder`
> > [@SbloodyS](https://github.com/SbloodyS) Could you please help confirm this htrace dependency issue when you have time? It seems like a regression bug. Thank you
>
> Please provide your `plugins_config`.
The reproduction steps are also very simple
1. download 3.3.x released package
2. run install-pulgin.sh
3. run ./bin/dolphinscheduler-daemon.sh start worker-server
Then you will see this error message `java.lang.ClassNotFoundException: org.apache.htrace.core.Tracer$Builder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "yaohuitc (via GitHub)" <gi...@apache.org>.
yaohuitc commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3455131051
> This will be released in the version `3.3.2`.
>
> >
The worker started through k8s helm chart failed to launch in version 3.3.1 due to missing plugin JAR files. Has this issue been resolved in subsequent versions? @SbloodyS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "yaohuitc (via GitHub)" <gi...@apache.org>.
yaohuitc commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3459317697
> > > This will be released in the version `3.3.2`.
> > > >
> >
> >
> > The worker started through k8s helm chart failed to launch in version 3.3.1 due to missing plugin JAR files. Has this issue been resolved in subsequent versions? [@SbloodyS](https://github.com/SbloodyS)
>
> You should [install plugins](https://dolphinscheduler.apache.org/en-us/docs/3.3.1/guide/installation/pseudo-cluster) manually if you want to use helm charts.
Does this mean that the docker images provided by open source cannot be used directly, and we need to build the worker's docker image ourselves? @SbloodyS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3455304010
> > This will be released in the version `3.3.2`.
> > >
>
> The worker started through k8s helm chart failed to launch in version 3.3.1 due to missing plugin JAR files. Has this issue been resolved in subsequent versions? [@SbloodyS](https://github.com/SbloodyS)
You should (install plugins)[https://dolphinscheduler.apache.org/en-us/docs/3.3.1/guide/installation/pseudo-cluster] manually if you want to use helm charts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3269272953
**The following another NoClassDefFoundError also needs attention. The root cause is the missing dependency "org.apache.hbase.thirdparty:hbase-noop-htrace" when starting the worker.**
`java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3370) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:125) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3424) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3392) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:485) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:233) at org.apache.dolphinscheduler.plugin.storage.hdfs.HdfsStorageOperator.init(HdfsStorageOperator.java:136) at org.apache.dolphinscheduler.plugin.storage.hdfs.HdfsStorageOperator.<init>(HdfsStorageOperator.java:75) at org.apache.dolphinscheduler.plugin.storage.hdfs.LocalStorageOperator.<init>(LocalStorageOperator.java:26) `
In version 3.2.2, this dependency comes from dolphinscheduler-task-all, but in 3.3.x it is set to provided, so it does not exist in the classpath for starting the worker
`mvn dependency:tree -Dverbose -Dincludes='org.apache.hbase.thirdparty:hbase-noop-htrace'`
```
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ dolphinscheduler-worker ---
[INFO] Verbose not supported since maven-dependency-plugin 3.0
[INFO] org.apache.dolphinscheduler:dolphinscheduler-worker:jar:3.2.2
[INFO] \- org.apache.dolphinscheduler:dolphinscheduler-task-all:jar:3.2.2:compile
[INFO] \- org.apache.dolphinscheduler:dolphinscheduler-task-datax:jar:3.2.2:compile
[INFO] \- org.apache.dolphinscheduler:dolphinscheduler-datasource-api:jar:3.2.2:compile
[INFO] \- org.apache.hbase.thirdparty:hbase-noop-htrace:jar:4.1.1:compile
[INFO]
```
<img width="665" height="219" alt="Image" src="https://github.com/user-attachments/assets/a8e935ce-abed-4105-836f-3d30214fa870" />
```
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ dolphinscheduler-worker ---
[INFO] Verbose not supported since maven-dependency-plugin 3.0
[INFO] org.apache.dolphinscheduler:dolphinscheduler-worker:jar:dev-SNAPSHOT
[INFO] \- org.apache.dolphinscheduler:dolphinscheduler-task-all:jar:dev-SNAPSHOT:provided
[INFO] \- org.apache.dolphinscheduler:dolphinscheduler-task-aliyunserverlessspark:jar:dev-SNAPSHOT:provided
[INFO] \- org.apache.dolphinscheduler:dolphinscheduler-datasource-all:jar:dev-SNAPSHOT:provided
[INFO] \- org.apache.dolphinscheduler:dolphinscheduler-datasource-hive:jar:dev-SNAPSHOT:provided
[INFO] \- org.apache.hbase.thirdparty:hbase-noop-htrace:jar:4.1.1:provided
[INFO]
```
<img width="629" height="114" alt="Image" src="https://github.com/user-attachments/assets/8c1ec93f-a7ed-4ca8-a3ab-0e75f74b7ee9" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273224631
> I can successfully start `api-server` and `master-server` and `worker-server` using the following `plugins_config`. And my installation package `apache-dolphinscheduler-3.3.1-bin.tar.gz ` has just been downloaded from official website. So I can not reproduce your issue. Please provide your reproducing steps in details.
>
> ```
> --alert-plugins--
> --end--
>
> --datasource-plugins--
> --end--
>
> --storage-plugins--
> dolphinscheduler-storage-hdfs
> --end--
>
> --task-plugins--
> dolphinscheduler-task-shell
> --end--
> ```
I've started the whole ds clusters excluding `alert-server`.
BTW, Please using english to describe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Doc] [Plugin] Since the plugin was changed to shade mode after 3.3.x and the shaded JAR was not downloaded by the install script, a NoClassDefFoundError was thrown [dolphinscheduler]
Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17488:
URL: https://github.com/apache/dolphinscheduler/issues/17488#issuecomment-3273238546
https://github.com/user-attachments/assets/eac251ca-1155-474a-8375-ea4deccaf29a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org