You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "gleonSun (via GitHub)" <gi...@apache.org> on 2025/01/08 06:08:57 UTC
[I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]
gleonSun opened a new issue, #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462
### Backend
CH (ClickHouse)
### Bug description
[Expected behavior]
Gluten can load the required Components and Backend, and function normally.
[actual behavior]
Gluten failed to load the required Components and Backend during initialization, resulting in abnormal execution.
### RootCause
The `ResourceUtil#getResources(java.util.regex.Pattern)` method loads META-INF from the `java.class.path`, but it might not be the META-INF from the project's corresponding resource directory. For example, when other projects reference Gluten, it retrieves the META-INF from their resource directories instead, which does not align with expectations.
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
java.lang.UnsupportedOperationException: empty.reduceLeft
at scala.collection.mutable.ArrayBuffer.scala$collection$IndexedSeqOptimized$$super$reduceLeft(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
at scala.collection.IndexedSeqOptimized.reduceLeft(IndexedSeqOptimized.scala:77) ~[scala-library-2.12.15.jar:?]
at scala.collection.IndexedSeqOptimized.reduceLeft$(IndexedSeqOptimized.scala:76) ~[scala-library-2.12.15.jar:?]
at scala.collection.mutable.ArrayBuffer.reduceLeft(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
at scala.collection.TraversableOnce.reduce(TraversableOnce.scala:254) ~[scala-library-2.12.15.jar:?]
at scala.collection.TraversableOnce.reduce$(TraversableOnce.scala:254) ~[scala-library-2.12.15.jar:?]
at scala.collection.AbstractTraversable.reduce(Traversable.scala:108) ~[scala-library-2.12.15.jar:?]
at org.apache.gluten.extension.columnar.transition.ConventionFunc$$anon$1.<init>(ConventionFunc.scala:78) ~[gluten.jar:?]
at org.apache.gluten.extension.columnar.transition.ConventionFunc$.newOverride(ConventionFunc.scala:76) ~[gluten.jar:?]
at org.apache.gluten.extension.columnar.transition.ConventionFunc$.create(ConventionFunc.scala:62) ~[gluten.jar:?]
at org.apache.gluten.extension.columnar.transition.InsertTransitions.<init>(Transitions.scala:27) ~[gluten.jar:?]
at org.apache.gluten.extension.columnar.transition.InsertTransitions$.create(Transitions.scala:71) ~[gluten.jar:?]
at org.apache.gluten.extension.columnar.transition.Transitions$.insert(Transitions.scala:89) ~[gluten.jar:?]
at org.apache.gluten.extension.GlutenColumnarRule.org$apache$gluten$extension$GlutenColumnarRule$$$anonfun$postColumnarTransitions$1(GlutenColumnarRule.scala:100) ~[gluten.jar:?]
at org.apache.gluten.extension.GlutenColumnarRule$$anonfun$postColumnarTransitions$2.apply(GlutenColumnarRule.scala:85) ~[gluten.jar:?]
at org.apache.gluten.extension.GlutenColumnarRule$$anonfun$postColumnarTransitions$2.apply(GlutenColumnarRule.scala:85) ~[gluten.jar:?]
at org.apache.gluten.extension.injector.InjectorControl$Disabler$DisablerOps$$anon$5$$anon$7.apply(InjectorControl.scala:149) ~[gluten.jar:?]
at org.apache.gluten.extension.injector.InjectorControl$Disabler$DisablerOps$$anon$5$$anon$7.apply(InjectorControl.scala:144) ~[gluten.jar:?]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]
Posted by "gleonSun (via GitHub)" <gi...@apache.org>.
gleonSun commented on issue #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462#issuecomment-2577322358
In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the `java.class.path` points to `xxx/kylin.jar` instead of `gluten.jar`. This prevents Gluten from locating the configuration files under `META-INF/gluten-components`, leading to an empty Components list being loaded. Consequently, when `ConventionFunc#newOverride` attempts to operate on the Components list, it triggers an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #8462: [CH] Gluten failed to load META-INF, causing abnormal execution
URL: https://github.com/apache/incubator-gluten/issues/8462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462#issuecomment-2577142671
@gleonSun Thanks for reporting!
Would you like to post the classpath, namely the value of `java.class.path` of your program? Which will make me better understand the issue. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #8462: [CH] Gluten failed to load META-INF, causing abnormal execution
URL: https://github.com/apache/incubator-gluten/issues/8462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462#issuecomment-2579174673
> > > In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the java.class.path points to xxx/kylin.jar instead of gluten.jar
> >
> >
> > Thanks for the detailed explanation. So gluten's compiled classes and shared libraries are included directedly into `xxx/kylin.jar` however the component files with `META-INF/gluten-components` are omitted in your case? Am I understanding correctly?
>
> Exactly
I see. Before https://github.com/apache/incubator-gluten/pull/8271 we use standard service file `META-INF/services/org.apache.gluten.backend.Backend` for backend's discovery. How did the previous way work in your case given that with current solution `META-INF/gluten-components` is excluded?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462#issuecomment-2579106956
> In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the java.class.path points to xxx/kylin.jar instead of gluten.jar
Thanks for the detailed explanation. So gluten's compiled classes and shared libraries are included directedly into `xxx/kylin.jar` in your case? Am I understanding correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]
Posted by "gleonSun (via GitHub)" <gi...@apache.org>.
gleonSun commented on issue #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462#issuecomment-2579164296
> > In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the java.class.path points to xxx/kylin.jar instead of gluten.jar
>
> Thanks for the detailed explanation. So gluten's compiled classes and shared libraries are included directedly into `xxx/kylin.jar` however the component files with `META-INF/gluten-components` are omitted in your case? Am I understanding correctly?
Exactly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org