You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "Guoliang Sun (Jira)" <ji...@apache.org> on 2022/12/07 08:39:00 UTC
[jira] [Created] (KYLIN-5345) Setting the noexec attribute on the /tmp directory causes build tasks to fail
Guoliang Sun created KYLIN-5345:
-----------------------------------
Summary: Setting the noexec attribute on the /tmp directory causes build tasks to fail
Key: KYLIN-5345
URL: https://issues.apache.org/jira/browse/KYLIN-5345
Project: Kylin
Issue Type: Bug
Affects Versions: 5.0-alpha
Reporter: Guoliang Sun
Fix For: 5.0-alpha
Attachments: image-2022-12-07-16-32-47-365.png, image-2022-12-07-16-34-34-370.png
The /tmp directory sets the noexec attribute
!image-2022-12-07-16-32-47-365.png|width=386,height=162!
Execution task build fails in getting flat table statistics
!image-2022-12-07-16-34-34-370.png|width=419,height=262!
See log
{code:java}
2022-11-01T15:25:06,703 INFO [logger-thread-0] spark.SparkContext : Starting job: count at FlatTableAndDictBase.scala:384
2022-11-01T15:25:06,704 INFO [logger-thread-0] scheduler.DAGScheduler : submit job : 3, executionId is 1
2022-11-01T15:25:06,705 INFO [logger-thread-0] scheduler.DAGScheduler : Got job 3 (count at FlatTableAndDictBase.scala:384) with 1 output partitions
2022-11-01T15:25:06,705 INFO [logger-thread-0] scheduler.DAGScheduler : Final stage: ResultStage 4 (count at FlatTableAndDictBase.scala:384)
2022-11-01T15:25:06,705 INFO [logger-thread-0] scheduler.DAGScheduler : Parents of final stage: List(ShuffleMapStage 3)
2022-11-01T15:25:06,705 INFO [logger-thread-0] scheduler.DAGScheduler : Missing parents: List()
2022-11-01T15:25:06,706 INFO [logger-thread-0] scheduler.DAGScheduler : Submitting ResultStage 4 (MapPartitionsRDD[12] at count at FlatTableAndDictBase.scala:384), which has no missing parents
2022-11-01T15:25:06,707 INFO [logger-thread-0] scheduler.DAGScheduler : submit stage 4 with jobId: 3
2022-11-01T15:25:06,713 INFO [logger-thread-0] memory.MemoryStore : Block broadcast_5 stored as values in memory (estimated size 11.0 KiB, free 365.4 MiB)
2022-11-01T15:25:06,714 INFO [logger-thread-0] memory.MemoryStore : Block broadcast_5_piece0 stored as bytes in memory (estimated size 5.5 KiB, free 365.4 MiB)
2022-11-01T15:25:06,715 INFO [logger-thread-0] storage.BlockManagerInfo : Added broadcast_5_piece0 in memory on client-junhui.kyligence.com:45347 (size: 5.5 KiB, free: 366.2 MiB)
2022-11-01T15:25:06,715 INFO [logger-thread-0] spark.SparkContext : Created broadcast 5 from broadcast at DAGScheduler.scala:1430
2022-11-01T15:25:06,716 INFO [logger-thread-0] scheduler.DAGScheduler : Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[12] at count at FlatTableAndDictBase.scala:384) (first 15 tasks are for partitions Vector(0))
2022-11-01T15:25:06,716 INFO [logger-thread-0] cluster.YarnScheduler : Adding task set 4.0 with 1 tasks resource profile 0
2022-11-01T15:25:06,718 INFO [logger-thread-0] scheduler.TaskSetManager : Starting task 0.0 in stage 4.0 (TID 3) (TID 3 , sandbox.hortonworks.com, executor 1 , partition 0, NODE_LOCAL , 4464 bytes) taskResourceAssignments Map()
2022-11-01T15:25:08,118 INFO [logger-thread-0] storage.BlockManagerInfo : Added broadcast_5_piece0 in memory on sandbox.hortonworks.com:34590 (size: 5.5 KiB, free: 366.3 MiB)
2022-11-01T15:25:08,142 INFO [logger-thread-0] spark.MapOutputTrackerMasterEndpoint : Asked to send map output locations for shuffle 0 to 10.1.2.194:34547
2022-11-01T15:25:08,250 ERROR [logger-thread-0] spark.MapOutputTrackerMaster : null
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_171]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_171]
at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:87) ~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at org.apache.spark.MapOutputTracker$.serializeOutputStatuses(MapOutputTracker.scala:1367) ~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at org.apache.spark.ShuffleStatus.$anonfun$serializedMapStatus$2(MapOutputTracker.scala:326) ~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
at org.apache.spark.ShuffleStatus.withWriteLock(MapOutputTracker.scala:77) ~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at org.apache.spark.ShuffleStatus.serializedMapStatus(MapOutputTracker.scala:323) ~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at org.apache.spark.MapOutputTrackerMaster$MessageLoop.handleStatusMessage(MapOutputTracker.scala:699) ~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:716) ~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_171]
Caused by: java.lang.UnsatisfiedLinkError: /tmp/libzstd-jni-1.5.0-43051813466323778737.so: /tmp/libzstd-jni-1.5.0-43051813466323778737.so: failed to map segment from shared object: Operation not permitted
no zstd-jni-1.5.0-4 in java.library.path
Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni-1.5.0-4.so or load zstd-jni-1.5.0-4 from system libraries. Please try building from source the jar or providing libzstd-jni-1.5.0-4 in your system.
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) ~[?:1.8.0_171]
at java.lang.Runtime.loadLibrary0(Runtime.java:870) ~[?:1.8.0_171]
at java.lang.System.loadLibrary(System.java:1122) ~[?:1.8.0_171]
at com.github.luben.zstd.util.Native.load(Native.java:124) ~[zstd-jni-1.5.0-4.jar:1.5.0-4]
at com.github.luben.zstd.util.Native.load(Native.java:55) ~[zstd-jni-1.5.0-4.jar:1.5.0-4]
at com.github.luben.zstd.ZstdOutputStreamNoFinalizer.<clinit>(ZstdOutputStreamNoFinalizer.java:18) ~[zstd-jni-1.5.0-4.jar:1.5.0-4]
at com.github.luben.zstd.RecyclingBufferPool.<clinit>(RecyclingBufferPool.java:17) ~[zstd-jni-1.5.0-4.jar:1.5.0-4]
at org.apache.spark.io.ZStdCompressionCodec.<init>(CompressionCodec.scala:221) ~[spark-core_2.12-3.2.0-kylin-4.6.1.0-SNAPSHOT.jar:3.2.0-kylin-4.6.1.0-SNAPSHOT]
... 15 more {code}
Since this was found during the tests to verify the flame graph function, the flame graph code will be revert first and the function will be retuned later
--
This message was sent by Atlassian Jira
(v8.20.10#820010)