You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Weiqing Yang (Jira)" <ji...@apache.org> on 2019/11/04 19:00:00 UTC
[jira] [Commented] (SAMZA-2363) Rock DB failed to create hard link
on Azure file
[ https://issues.apache.org/jira/browse/SAMZA-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16966916#comment-16966916 ]
Weiqing Yang commented on SAMZA-2363:
-------------------------------------
/cc [~xinyu] [~pmaheshwari]
> Rock DB failed to create hard link on Azure file
> ------------------------------------------------
>
> Key: SAMZA-2363
> URL: https://issues.apache.org/jira/browse/SAMZA-2363
> Project: Samza
> Issue Type: Bug
> Components: kv, kv-store
> Reporter: Weiqing Yang
> Priority: Major
>
> I was tried to test the implementation ([https://github.com/apache/samza/pull/1197]) of Samza on K8s on AKS (using the latest Samza (master branch)), and got the error below. The job was trying to store logs and the state in Azure file.
> (P.S. Samza 1.1 does not hit this issue, and the job built on Samza 1.1 works fine on AKS.)
> {code:java}
> 2019-11-04 00:56:00.558 [main] SamzaContainer [INFO] Entering run loop.
> 2019-11-04 00:56:00.559 [main] ClusterBasedProcessorLifecycleListener [INFO] Container Started
> 2019-11-04 00:56:57.106 [kafka-producer-network-thread | kafka_producer-wikipedia_application-1] Metadata [INFO] Cluster ID: YNWIzBLFSa2Lg5wehzo7ZA
> 2019-11-04 00:57:03.920 [main] KafkaSystemAdmin [INFO] Fetching SSP metadata for: [SystemStreamPartition [kafka, wikipedia-stats-changelog, 7]]
> 2019-11-04 00:57:03.923 [main] KafkaSystemAdmin [INFO] Fetching SSP metadata for: [SystemStreamPartition [kafka, wikipedia-application-1-window-statsWindow, 7]]
> 2019-11-04 00:57:04.222 [main] RunLoop [ERROR] Task Partition 7 commit failed
> org.rocksdb.RocksDBException: while link file to /tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7-1572829023927.tmp/000013.sst: /tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7/000013.sst: Operation not supported
> at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
> at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
> at org.apache.samza.storage.kv.RocksDbKeyValueStore.checkpoint(RocksDbKeyValueStore.scala:245)
> at org.apache.samza.storage.kv.LoggedStore.checkpoint(LoggedStore.scala:125)
> at org.apache.samza.storage.kv.SerializedKeyValueStore.checkpoint(SerializedKeyValueStore.scala:170)
> at org.apache.samza.storage.kv.CachedStore.checkpoint(CachedStore.scala:297)
> at org.apache.samza.storage.kv.NullSafeKeyValueStore.checkpoint(NullSafeKeyValueStore.scala:103)
> at org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$checkpoint$1.apply(KeyValueStorageEngine.scala:215)
> at org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$checkpoint$1.apply(KeyValueStorageEngine.scala:212)
> at org.apache.samza.util.TimerUtil$class.updateTimer(TimerUtil.scala:37)
> at org.apache.samza.storage.kv.KeyValueStorageEngine.updateTimer(KeyValueStorageEngine.scala:37)
> at org.apache.samza.storage.kv.KeyValueStorageEngine.checkpoint(KeyValueStorageEngine.scala:212)
> at org.apache.samza.storage.TransactionalStateTaskStorageManager$$anonfun$2.apply(TransactionalStateTaskStorageManager.scala:67)
> at org.apache.samza.storage.TransactionalStateTaskStorageManager$$anonfun$2.apply(TransactionalStateTaskStorageManager.scala:66)
> at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
> at org.apache.samza.storage.TransactionalStateTaskStorageManager.checkpoint(TransactionalStateTaskStorageManager.scala:66)
> at org.apache.samza.container.TaskInstance.commit(TaskInstance.scala:267)
> at org.apache.samza.container.RunLoop$AsyncTaskWorker$5.run(RunLoop.java:547)
> at org.apache.samza.container.RunLoop$AsyncTaskWorker.commit(RunLoop.java:566)
> at org.apache.samza.container.RunLoop$AsyncTaskWorker.run(RunLoop.java:432)
> at org.apache.samza.container.RunLoop$AsyncTaskWorker.access$300(RunLoop.java:357)
> at org.apache.samza.container.RunLoop.runTasks(RunLoop.java:244)
> at org.apache.samza.container.RunLoop.run(RunLoop.java:176)
> at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:768)
> at org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java:139)
> at org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java:82)
> at org.apache.samza.runtime.LocalContainerRunner.main(LocalContainerRunner.java:80)
> 2019-11-04 00:57:04.223 [main] RunLoop [ERROR] Caught throwable and stopping run loop
> org.rocksdb.RocksDBException: while link file to /tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7-1572829023927.tmp/000013.sst: /tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7/000013.sst: Operation not supported
> at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)