You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "gleonSun (via GitHub)" <gi...@apache.org> on 2025/01/08 06:08:57 UTC

[I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]

gleonSun opened a new issue, #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462

   ### Backend
   
   CH (ClickHouse)
   
   ### Bug description
   
   [Expected behavior]
   Gluten can load the required Components and Backend, and function normally.
   
   [actual behavior]
   Gluten failed to load the required Components and Backend during initialization, resulting in abnormal execution.
   
   ### RootCause
   The `ResourceUtil#getResources(java.util.regex.Pattern)` method loads META-INF from the `java.class.path`, but it might not be the META-INF from the project's corresponding resource directory. For example, when other projects reference Gluten, it retrieves the META-INF from their resource directories instead, which does not align with expectations.
   
   ### Spark version
   
   None
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   ```bash
   java.lang.UnsupportedOperationException: empty.reduceLeft
           at scala.collection.mutable.ArrayBuffer.scala$collection$IndexedSeqOptimized$$super$reduceLeft(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
           at scala.collection.IndexedSeqOptimized.reduceLeft(IndexedSeqOptimized.scala:77) ~[scala-library-2.12.15.jar:?]
           at scala.collection.IndexedSeqOptimized.reduceLeft$(IndexedSeqOptimized.scala:76) ~[scala-library-2.12.15.jar:?]
           at scala.collection.mutable.ArrayBuffer.reduceLeft(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
           at scala.collection.TraversableOnce.reduce(TraversableOnce.scala:254) ~[scala-library-2.12.15.jar:?]
           at scala.collection.TraversableOnce.reduce$(TraversableOnce.scala:254) ~[scala-library-2.12.15.jar:?]
           at scala.collection.AbstractTraversable.reduce(Traversable.scala:108) ~[scala-library-2.12.15.jar:?]
           at org.apache.gluten.extension.columnar.transition.ConventionFunc$$anon$1.<init>(ConventionFunc.scala:78) ~[gluten.jar:?]
           at org.apache.gluten.extension.columnar.transition.ConventionFunc$.newOverride(ConventionFunc.scala:76) ~[gluten.jar:?]
           at org.apache.gluten.extension.columnar.transition.ConventionFunc$.create(ConventionFunc.scala:62) ~[gluten.jar:?]
           at org.apache.gluten.extension.columnar.transition.InsertTransitions.<init>(Transitions.scala:27) ~[gluten.jar:?]
           at org.apache.gluten.extension.columnar.transition.InsertTransitions$.create(Transitions.scala:71) ~[gluten.jar:?]
           at org.apache.gluten.extension.columnar.transition.Transitions$.insert(Transitions.scala:89) ~[gluten.jar:?]
           at org.apache.gluten.extension.GlutenColumnarRule.org$apache$gluten$extension$GlutenColumnarRule$$$anonfun$postColumnarTransitions$1(GlutenColumnarRule.scala:100) ~[gluten.jar:?]
           at org.apache.gluten.extension.GlutenColumnarRule$$anonfun$postColumnarTransitions$2.apply(GlutenColumnarRule.scala:85) ~[gluten.jar:?]
           at org.apache.gluten.extension.GlutenColumnarRule$$anonfun$postColumnarTransitions$2.apply(GlutenColumnarRule.scala:85) ~[gluten.jar:?]
           at org.apache.gluten.extension.injector.InjectorControl$Disabler$DisablerOps$$anon$5$$anon$7.apply(InjectorControl.scala:149) ~[gluten.jar:?]
           at org.apache.gluten.extension.injector.InjectorControl$Disabler$DisablerOps$$anon$5$$anon$7.apply(InjectorControl.scala:144) ~[gluten.jar:?]
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]

Posted by "gleonSun (via GitHub)" <gi...@apache.org>.
gleonSun commented on issue #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462#issuecomment-2577322358

   In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the `java.class.path` points to `xxx/kylin.jar` instead of `gluten.jar`. This prevents Gluten from locating the configuration files under `META-INF/gluten-components`, leading to an empty Components list being loaded. Consequently, when `ConventionFunc#newOverride` attempts to operate on the Components list, it triggers an exception.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #8462: [CH] Gluten failed to load META-INF, causing abnormal execution
URL: https://github.com/apache/incubator-gluten/issues/8462


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462#issuecomment-2577142671

   @gleonSun Thanks for reporting!
   
   Would you like to post the classpath, namely the value of `java.class.path` of your program? Which will make me better understand the issue. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #8462: [CH] Gluten failed to load META-INF, causing abnormal execution
URL: https://github.com/apache/incubator-gluten/issues/8462


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462#issuecomment-2579174673

   > > > In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the java.class.path points to xxx/kylin.jar instead of gluten.jar
   > > 
   > > 
   > > Thanks for the detailed explanation. So gluten's compiled classes and shared libraries are included directedly into `xxx/kylin.jar` however the component files with `META-INF/gluten-components` are omitted in your case? Am I understanding correctly?
   > 
   > Exactly
   
   I see. Before https://github.com/apache/incubator-gluten/pull/8271 we use standard service file `META-INF/services/org.apache.gluten.backend.Backend` for backend's discovery. How did the previous way work in your case given that with current solution `META-INF/gluten-components` is excluded?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462#issuecomment-2579106956

   > In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the java.class.path points to xxx/kylin.jar instead of gluten.jar
   
   Thanks for the detailed explanation. So gluten's compiled classes and shared libraries are included directedly into `xxx/kylin.jar` in your case? Am I understanding correctly?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] Gluten failed to load META-INF, causing abnormal execution [incubator-gluten]

Posted by "gleonSun (via GitHub)" <gi...@apache.org>.
gleonSun commented on issue #8462:
URL: https://github.com/apache/incubator-gluten/issues/8462#issuecomment-2579164296

   > > In my case, the Gluten project is wrapped by the Kylin service to use the Native Engine. As a result, the java.class.path points to xxx/kylin.jar instead of gluten.jar
   > 
   > Thanks for the detailed explanation. So gluten's compiled classes and shared libraries are included directedly into `xxx/kylin.jar` however the component files with `META-INF/gluten-components` are omitted in your case? Am I understanding correctly?
   
   Exactly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org