You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "hpplayer (via GitHub)" <gi...@apache.org> on 2025/03/18 02:42:07 UTC
[I] Cannot allocate memory in static TLS block [incubator-gluten]
hpplayer opened a new issue, #9036:
URL: https://github.com/apache/incubator-gluten/issues/9036
### Backend
VL (Velox)
### Bug description
Hi, I'm trying to execute Velox Spark jobs on arm64 instances (aws graviton), but running into the error below:
```
Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/spark-local-dir/gluten-xxxx/jni/.../linux/aarch64/libvelox.so: /tmp/spark-local-dir/gluten-xxx/jni/.../gluten-xxxx/linux/aarch64/libgluten.so: cannot allocate memory in static TLS block
at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2450)
at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2506)
at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2705)
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2635)
at java.base/java.lang.Runtime.load0(Runtime.java:768)
at java.base/java.lang.System.load(System.java:1854)
at org.apache.gluten.jni.JniLibLoader.loadFromPath0(JniLibLoader.java:96)
at org.apache.gluten.jni.JniLibLoader.loadWithLink(JniLibLoader.java:219)
at org.apache.gluten.jni.JniLibLoader.load(JniLibLoader.java:169)
at org.apache.gluten.backendsapi.velox.VeloxListenerApi.initialize(VeloxListenerApi.scala:157)
at org.apache.gluten.backendsapi.velox.VeloxListenerApi.onDriverStart(VeloxListenerApi.scala:93)
at org.apache.gluten.backendsapi.SubstraitBackend.onDriverStart(SubstraitBackend.scala:27)
at org.apache.gluten.backendsapi.SubstraitBackend.onDriverStart$(SubstraitBackend.scala:26)
at org.apache.gluten.backendsapi.velox.VeloxBackend.onDriverStart(VeloxBackend.scala:53)
at org.apache.gluten.GlutenDriverPlugin.init(GlutenPlugin.scala:71)
at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$driverPlugins$1(PluginContainer.scala:53)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at org.apache.spark.internal.plugin.DriverPluginContainer.<init>(PluginContainer.scala:46)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:210)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:193)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:576)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2772)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:950)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:944)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:54)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.<init>(SparkSQLCLIDriver.scala:333)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:160)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1005)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1093)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1102)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
### Spark version
Spark-3.2.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Cannot allocate memory in static TLS block [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #9036:
URL: https://github.com/apache/incubator-gluten/issues/9036#issuecomment-2731473732
@kevinw66 did you see the issue before? Looks the TLS issue is pretty common in arm world when loading a library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Cannot allocate memory in static TLS block [incubator-gluten]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on issue #9036:
URL: https://github.com/apache/incubator-gluten/issues/9036#issuecomment-2731859941
> [@kevinw66](https://github.com/kevinw66) did you see the issue before? Looks the TLS issue is pretty common in arm world when loading a library.
Not yet, I'll keep an eye on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Cannot allocate memory in static TLS block [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #9036:
URL: https://github.com/apache/incubator-gluten/issues/9036#issuecomment-2732005739
@hpplayer, are you using jemalloc? I once encountered a similar issue when using jemalloc which requires large thread-local storage for some initialization during loading libs dynamically. It was fixed by re-building jemalloc with TLS disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org