You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tdas <gi...@git.apache.org> on 2016/03/11 05:28:59 UTC
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/11645
[WIP][SPARK-13809][SQL] State store for streaming aggregations
## What changes were proposed in this pull request?
In this PR, I am implementing a new abstraction for management of streaming state data - State Store. It is a key-value store for persisting running aggregates for aggregate operations in streaming dataframes. The motivation and design is discussed here.
https://docs.google.com/document/d/1-ncawFx8JS5Zyfq1HAEGBx56RDet9wfVp_hDM8ZL254/edit#
## How was this patch tested?
Unit tests (WIP)
## TODO
[ ] Unit tests
- [ ] StateStoreCoordinator
- [ ] StateStoreRDD fault-tolerance
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark state-store
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11645.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11645
----
commit c0194744378c2b1827aa6bb3a6109b3549710a44
Author: Tathagata Das <ta...@gmail.com>
Date: 2016-03-10T23:02:37Z
First draft of StateStore
commit 4f8dade98ab2720ffab78a6ace567b9705f1b488
Author: Tathagata Das <ta...@gmail.com>
Date: 2016-03-11T01:50:50Z
Updated tests
commit f417bde8bd65572e78125c7aa900221fa0bc4d7c
Author: Tathagata Das <ta...@gmail.com>
Date: 2016-03-11T04:24:39Z
Added basic unit test for StateStoreRDD
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917453
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEndpointRef, RpcEnv}
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.util.RpcUtils
+
+/** Trait representing all messages to [[StateStoreCoordinator]] */
+private sealed trait StateStoreCoordinatorMessage extends Serializable
+
+/** Classes representing messages */
+private case class ReportActiveInstance(storeId: StateStoreId, host: String, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class VerifyIfInstanceActive(storeId: StateStoreId, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class GetLocation(storeId: StateStoreId)
+ extends StateStoreCoordinatorMessage
+
+private case class DeactivateInstances(storeRootLocation: String)
+ extends StateStoreCoordinatorMessage
+
+private object StopCoordinator
--- End diff --
We should probably document what these do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197117270
**[Test build #53252 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53252/consoleFull)** for PR 11645 at commit [`e89f4d0`](https://github.com/apache/spark/commit/e89f4d0d1a45d5a4d8c59a433cf55ce5d19010da).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199959039
**[Test build #53804 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53804/consoleFull)** for PR 11645 at commit [`756762a`](https://github.com/apache/spark/commit/756762a044ce9a2735950ed27384151502ce020d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56918106
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,597 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.serializer.KryoSerializer
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ sparkConf: SparkConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ import HDFSBackedStateStoreProvider._
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ // private val tempDeltaFileStream = fs.create(tempDeltaFile, true)
+ private val allUpdates = new java.util.HashMap[UnsafeRow, StoreUpdate]()
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /** Update the value of a key using the value generated by the update function */
+ override def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = Option(mapToUpdate.get(key))
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ writeToDeltaFile(tempDeltaFileStream, ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: UnsafeRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keySet().iterator()
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ keyIter.remove()
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ writeToDeltaFile(tempDeltaFileStream, KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ finalizeDeltaFile(tempDeltaFileStream)
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ logInfo(s"Committed version $newVersion for $this")
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ logInfo("Canceled ")
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[(UnsafeRow, UnsafeRow)] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.values().asScala.toIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ val time = System.nanoTime()
+ newMap.putAll(loadMap(version))
+ }
+ val store = new HDFSBackedStateStore(version, newMap)
+ logInfo(s"Retrieved version $version of $this for update")
+ store
+ }
+
+ /** Do maintenance backing data files, including creating snapshots and cleaning up old files */
+ override def doMaintenance(): Unit = {
+ try {
+ doSnapshot()
+ cleanup()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up $this")
+ }
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ /* Internal classes and methods */
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir = new Path(id.rootLocation, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(hadoopConf)
+ private val serializer = new KryoSerializer(sparkConf)
+ private val minBatchesToRetain = sparkConf.getInt(
+ MIN_BATCHES_TO_RETAIN_CONF, MIN_BATCHES_TO_RETAIN_DEFAULT)
+ private val maxDeltaChainForSnapshots = sparkConf.getInt(
+ MAX_DELTA_CHAIN_FOR_SNAPSHOTS_CONF, MAX_DELTA_CHAIN_FOR_SNAPSHOTS_DEFAULT)
+
+ initialize()
+
+ private case class StoreFile(version: Long, path: Path, isSnapshot: Boolean)
+
+ /** Commit a set of updates to the store with the given new version */
+ private def commitUpdates(newVersion: Long, map: MapType, tempDeltaFile: Path): Path = {
+ synchronized {
+ val finalDeltaFile = deltaFile(newVersion)
+ fs.rename(tempDeltaFile, finalDeltaFile)
+ loadedMaps.put(newVersion, map)
+ finalDeltaFile
+ }
+ }
+
+ /**
+ * Get iterator of all the data of the latest version of the store.
+ * Note that this will look up the files to determined the latest known version.
+ */
+ private[state] def latestIterator(): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ val versionsInFiles = fetchFiles().map(_.version).toSet
+ val versionsLoaded = loadedMaps.keySet
+ val allKnownVersions = versionsInFiles ++ versionsLoaded
+ if (allKnownVersions.nonEmpty) {
+ loadMap(allKnownVersions.max).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ } else Iterator.empty
+ }
+
+ /** Get iterator of a specific version of the store */
+ private[state] def iterator(version: Long): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ loadMap(version).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ }
+
+ /** Initialize the store provider */
+ private def initialize(): Unit = {
+ if (!fs.exists(baseDir)) {
+ fs.mkdirs(baseDir)
+ } else {
+ if (!fs.isDirectory(baseDir)) {
+ throw new IllegalStateException(
+ s"Cannot use ${id.rootLocation} for storing state data for $this as" +
+ s"$baseDir already exists and is not a directory")
+ }
+ }
+ }
+
+ /** Load the required version of the map data from the backing files */
+ private def loadMap(version: Long): MapType = {
+ if (version <= 0) return new MapType
+ synchronized { loadedMaps.get(version) }.getOrElse {
+ val mapFromFile = readSnapshotFile(version).getOrElse {
+ val prevMap = loadMap(version - 1)
+ val newMap = new MapType(prevMap)
+ newMap.putAll(prevMap)
+ updateFromDeltaFile(version, newMap)
+ newMap
+ }
+ loadedMaps.put(version, mapFromFile)
+ mapFromFile
+ }
+ }
+
+ private def writeToDeltaFile(output: DataOutputStream, update: StoreUpdate): Unit = {
+
+ def writeUpdate(key: UnsafeRow, value: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ val valueBytes = value.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+
+ def writeRemove(key: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(-1)
+ }
+
+ update match {
+ case ValueAdded(key, value) =>
+ writeUpdate(key, value)
+ case ValueUpdated(key, value) =>
+ writeUpdate(key, value)
+ case KeyRemoved(key) =>
+ writeRemove(key)
+ }
+ }
+
+ private def finalizeDeltaFile(output: DataOutputStream): Unit = {
+ output.writeInt(-1) // Write this magic number to signify end of file
+ output.close()
+ }
+
+ private def updateFromDeltaFile(version: Long, map: MapType): Unit = {
+ val fileToRead = deltaFile(version)
+ if (!fs.exists(fileToRead)) {
+ throw new IllegalStateException(
+ s"Error reading delta file $fileToRead of $this: $fileToRead does not exist")
+ }
+ var input: DataInputStream = null
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while(!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading delta file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ map.remove(keyRow)
+ } else {
+ val valueRowBuffer = new Array[Byte](valueSize)
+ ByteStreams.readFully(input, valueRowBuffer, 0, valueSize)
+ val valueRow = new UnsafeRow(valueSchema.fields.length)
+ valueRow.pointTo(valueRowBuffer, valueSize)
+ map.put(keyRow, valueRow)
+ }
+ }
+ }
+ } finally {
+ if (input != null) input.close()
+ }
+ logInfo(s"Read delta file for version $version of $this from $fileToRead")
+ }
+
+ private def writeSnapshotFile(version: Long, map: MapType): Unit = {
+ val fileToWrite = snapshotFile(version)
+ val ser = serializer.newInstance()
+ var output: DataOutputStream = null
+ Utils.tryWithSafeFinally {
+ output = compressStream(fs.create(fileToWrite, false))
+ val iter = map.entrySet().iterator()
+ while(iter.hasNext) {
+ val entry = iter.next()
+ val keyBytes = entry.getKey.getBytes()
+ val valueBytes = entry.getValue.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+ output.writeInt(-1)
+ } {
+ if (output != null) output.close()
+ }
+ logInfo(s"Written snapshot file for version $version of $this at $fileToWrite")
+ }
+
+ private def readSnapshotFile(version: Long): Option[MapType] = {
+ val fileToRead = snapshotFile(version)
+ if (!fs.exists(fileToRead)) return None
+
+ val deser = serializer.newInstance()
+ val map = new MapType()
+ var input: DataInputStream = null
+
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while (!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading snapshot file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ throw new Exception(
+ s"Error reading snapshot file $fileToRead of $this: value size cannot be $valueSize")
+ } else {
+ val valueRowBuffer = new Array[Byte](valueSize)
+ ByteStreams.readFully(input, valueRowBuffer, 0, valueSize)
+ val valueRow = new UnsafeRow(valueSchema.fields.length)
+ valueRow.pointTo(valueRowBuffer, valueSize)
+ map.put(keyRow, valueRow)
+ }
+ }
+ }
+ logInfo(s"Read snapshot file for version $version of $this from $fileToRead")
+ Some(map)
+ } finally {
+ if (input != null) input.close()
+ }
+ }
+
+
+ /** Perform a snapshot of the store to allow delta files to be consolidated */
+ private def doSnapshot(): Unit = {
+ try {
+ val files = fetchFiles()
+ if (files.nonEmpty) {
+ val lastVersion = files.last.version
+ val deltaFilesForLastVersion =
+ filesForVersion(files, lastVersion).filter(_.isSnapshot == false)
+ synchronized { loadedMaps.get(lastVersion) } match {
+ case Some(map) =>
+ if (deltaFilesForLastVersion.size > maxDeltaChainForSnapshots) {
+ writeSnapshotFile(lastVersion, map)
+ }
+ case None =>
+ // The last map is not loaded, probably some other instance is incharge
+ }
+
+ }
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error doing snapshots for $this", e)
+ }
+ }
+
+ /**
+ * Clean up old snapshots and delta files that are not needed any more. It ensures that last
+ * few versions of the store can be recovered from the files, so re-executed RDD operations
+ * can re-apply updates on the past versions of the store.
+ */
+ private[state] def cleanup(): Unit = {
+ try {
+ val files = fetchFiles()
+ if (files.nonEmpty) {
+ val earliestVersionToRetain = files.last.version - minBatchesToRetain
+ if (earliestVersionToRetain > 0) {
+ val earliestFileToRetain = filesForVersion(files, earliestVersionToRetain).head
+ synchronized {
+ val mapsToRemove = loadedMaps.keys.filter(_ < earliestVersionToRetain).toSeq
+ mapsToRemove.foreach(loadedMaps.remove)
+ }
+ files.filter(_.version < earliestFileToRetain.version).foreach { f =>
+ fs.delete(f.path, true)
+ }
+ logInfo(s"Deleted files older than ${earliestFileToRetain.version} for $this")
+ }
+ }
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error cleaning up files for $this", e)
+ }
+ }
+
+ /** Files needed to recover the given version of the store */
+ private def filesForVersion(allFiles: Seq[StoreFile], version: Long): Seq[StoreFile] = {
+ require(version >= 0)
+ require(allFiles.exists(_.version == version))
+
+ val latestSnapshotFileBeforeVersion = allFiles
+ .filter(_.isSnapshot == true)
+ .takeWhile(_.version <= version)
+ .lastOption
+ val deltaBatchFiles = latestSnapshotFileBeforeVersion match {
+ case Some(snapshotFile) =>
+ val deltaBatchIds = (snapshotFile.version + 1) to version
+
+ val deltaFiles = allFiles.filter { file =>
+ file.version > snapshotFile.version && file.version <= version
+ }
+ verify(
+ deltaFiles.size == version - snapshotFile.version,
+ s"Unexpected list of delta files for version $version for $this: $deltaFiles"
+ )
+ deltaFiles
+
+ case None =>
+ allFiles.takeWhile(_.version <= version)
+ }
+ latestSnapshotFileBeforeVersion.toSeq ++ deltaBatchFiles
+ }
+
+ /** Fetch all the files that back the store */
+ private def fetchFiles(): Seq[StoreFile] = {
+ val files: Seq[FileStatus] = try {
+ fs.listStatus(baseDir)
+ } catch {
+ case _: java.io.FileNotFoundException =>
+ Seq.empty
+ }
+ val versionToFiles = new mutable.HashMap[Long, StoreFile]
+ files.foreach { status =>
+ val path = status.getPath
+ val nameParts = path.getName.split("\\.")
+ if (nameParts.size == 2) {
+ val version = nameParts(0).toLong
+ nameParts(1).toLowerCase match {
+ case "delta" =>
+ // ignore the file otherwise, snapshot file already exists for that batch id
+ if (!versionToFiles.contains(version)) {
+ versionToFiles.put(version, StoreFile(version, path, isSnapshot = false))
+ }
+ case "snapshot" =>
+ versionToFiles.put(version, StoreFile(version, path, isSnapshot = true))
+ case _ =>
+ logWarning(s"Could not identify file $path for $this")
+ }
+ }
+ }
+ val storeFiles = versionToFiles.values.toSeq.sortBy(_.version)
+ logDebug(s"Current set of files for $this: $storeFiles")
+ storeFiles
+ }
+
+ private def compressStream(outputStream: DataOutputStream): DataOutputStream = {
+ val compressed = new LZ4CompressionCodec(sparkConf).compressedOutputStream(outputStream)
+ new DataOutputStream(compressed)
+ }
+
+ private def decompressStream(inputStream: DataInputStream): DataInputStream = {
+ val compressed = new LZ4CompressionCodec(sparkConf).compressedInputStream(inputStream)
+ new DataInputStream(compressed)
+ }
+
+ private def deltaFile(version: Long): Path = {
+ new Path(baseDir, s"$version.delta")
+ }
+
+ private def snapshotFile(version: Long): Path = {
+ new Path(baseDir, s"$version.snapshot")
+ }
+
+ private def verify(condition: => Boolean, msg: String): Unit = {
+ if (!condition) {
+ throw new IllegalStateException(msg)
+ }
+ }
+}
+
+private[state] object HDFSBackedStateStoreProvider {
+ val MAX_DELTA_CHAIN_FOR_SNAPSHOTS_CONF = "spark.sql.streaming.stateStore.maxDeltaChain"
+ val MAX_DELTA_CHAIN_FOR_SNAPSHOTS_DEFAULT = 10
+
+ val MIN_BATCHES_TO_RETAIN_CONF = "spark.sql.streaming.stateStore.minBatchesToRetain"
+ val MIN_BATCHES_TO_RETAIN_DEFAULT = 2
--- End diff --
move to SQLConf
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57095337
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,585 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ storeConf: StateStoreConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ private val allUpdates = new java.util.HashMap[UnsafeRow, StoreUpdate]()
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ override def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = Option(mapToUpdate.get(key))
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ writeToDeltaFile(tempDeltaFileStream, ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: UnsafeRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keySet().iterator()
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ keyIter.remove()
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ writeToDeltaFile(tempDeltaFileStream, KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store, and return the new version. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ finalizeDeltaFile(tempDeltaFileStream)
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ logInfo(s"Committed version $newVersion for $this")
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ logInfo("Canceled ")
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[(UnsafeRow, UnsafeRow)] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.values().asScala.toIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ val time = System.nanoTime()
+ newMap.putAll(loadMap(version))
+ }
+ val store = new HDFSBackedStateStore(version, newMap)
+ logInfo(s"Retrieved version $version of $this for update")
+ store
+ }
+
+ /** Do maintenance backing data files, including creating snapshots and cleaning up old files */
+ override def doMaintenance(): Unit = {
+ try {
+ doSnapshot()
+ cleanup()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up $this")
+ }
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ /* Internal classes and methods */
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir =
+ new Path(id.checkpointLocation, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(hadoopConf)
+ private val sparkConf = Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
+
+ initialize()
+
+ private case class StoreFile(version: Long, path: Path, isSnapshot: Boolean)
+
+ /** Commit a set of updates to the store with the given new version */
+ private def commitUpdates(newVersion: Long, map: MapType, tempDeltaFile: Path): Path = {
+ synchronized {
+ val finalDeltaFile = deltaFile(newVersion)
+ fs.rename(tempDeltaFile, finalDeltaFile)
+ loadedMaps.put(newVersion, map)
+ finalDeltaFile
+ }
+ }
+
+ /**
+ * Get iterator of all the data of the latest version of the store.
+ * Note that this will look up the files to determined the latest known version.
+ */
+ private[state] def latestIterator(): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ val versionsInFiles = fetchFiles().map(_.version).toSet
+ val versionsLoaded = loadedMaps.keySet
+ val allKnownVersions = versionsInFiles ++ versionsLoaded
+ if (allKnownVersions.nonEmpty) {
+ loadMap(allKnownVersions.max).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ } else Iterator.empty
+ }
+
+ /** Get iterator of a specific version of the store */
+ private[state] def iterator(version: Long): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ loadMap(version).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ }
+
+ /** Initialize the store provider */
+ private def initialize(): Unit = {
+ if (!fs.exists(baseDir)) {
+ fs.mkdirs(baseDir)
+ } else {
+ if (!fs.isDirectory(baseDir)) {
+ throw new IllegalStateException(
+ s"Cannot use ${id.checkpointLocation} for storing state data for $this as" +
+ s"$baseDir already exists and is not a directory")
+ }
+ }
+ }
+
+ /** Load the required version of the map data from the backing files */
+ private def loadMap(version: Long): MapType = {
+ if (version <= 0) return new MapType
+ synchronized { loadedMaps.get(version) }.getOrElse {
+ val mapFromFile = readSnapshotFile(version).getOrElse {
+ val prevMap = loadMap(version - 1)
+ val newMap = new MapType(prevMap)
+ newMap.putAll(prevMap)
+ updateFromDeltaFile(version, newMap)
+ newMap
+ }
+ loadedMaps.put(version, mapFromFile)
+ mapFromFile
+ }
+ }
+
+ private def writeToDeltaFile(output: DataOutputStream, update: StoreUpdate): Unit = {
+
+ def writeUpdate(key: UnsafeRow, value: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ val valueBytes = value.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+
+ def writeRemove(key: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(-1)
+ }
+
+ update match {
+ case ValueAdded(key, value) =>
+ writeUpdate(key, value)
+ case ValueUpdated(key, value) =>
+ writeUpdate(key, value)
+ case KeyRemoved(key) =>
+ writeRemove(key)
+ }
+ }
+
+ private def finalizeDeltaFile(output: DataOutputStream): Unit = {
+ output.writeInt(-1) // Write this magic number to signify end of file
+ output.close()
+ }
+
+ private def updateFromDeltaFile(version: Long, map: MapType): Unit = {
+ val fileToRead = deltaFile(version)
+ if (!fs.exists(fileToRead)) {
+ throw new IllegalStateException(
+ s"Error reading delta file $fileToRead of $this: $fileToRead does not exist")
+ }
+ var input: DataInputStream = null
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while(!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading delta file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ map.remove(keyRow)
+ } else {
+ val valueRowBuffer = new Array[Byte](valueSize)
+ ByteStreams.readFully(input, valueRowBuffer, 0, valueSize)
+ val valueRow = new UnsafeRow(valueSchema.fields.length)
+ valueRow.pointTo(valueRowBuffer, valueSize)
+ map.put(keyRow, valueRow)
+ }
+ }
+ }
+ } finally {
+ if (input != null) input.close()
+ }
+ logInfo(s"Read delta file for version $version of $this from $fileToRead")
+ }
+
+ private def writeSnapshotFile(version: Long, map: MapType): Unit = {
+ val fileToWrite = snapshotFile(version)
+ var output: DataOutputStream = null
+ Utils.tryWithSafeFinally {
+ output = compressStream(fs.create(fileToWrite, false))
+ val iter = map.entrySet().iterator()
+ while(iter.hasNext) {
+ val entry = iter.next()
+ val keyBytes = entry.getKey.getBytes()
+ val valueBytes = entry.getValue.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+ output.writeInt(-1)
+ } {
+ if (output != null) output.close()
+ }
+ logInfo(s"Written snapshot file for version $version of $this at $fileToWrite")
+ }
+
+ private def readSnapshotFile(version: Long): Option[MapType] = {
+ val fileToRead = snapshotFile(version)
+ if (!fs.exists(fileToRead)) return None
+
+ val map = new MapType()
+ var input: DataInputStream = null
+
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while (!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
--- End diff --
nit: IOException
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57094820
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ storeConf: StateStoreConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ private val allUpdates = new java.util.HashMap[UnsafeRow, StoreUpdate]()
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ override def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = Option(mapToUpdate.get(key))
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ writeToDeltaFile(tempDeltaFileStream, ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: UnsafeRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keySet().iterator()
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ keyIter.remove()
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ writeToDeltaFile(tempDeltaFileStream, KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ finalizeDeltaFile(tempDeltaFileStream)
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ logInfo(s"Committed version $newVersion for $this")
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ logInfo("Canceled ")
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[(UnsafeRow, UnsafeRow)] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.values().asScala.toIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ val time = System.nanoTime()
+ newMap.putAll(loadMap(version))
+ }
+ val store = new HDFSBackedStateStore(version, newMap)
+ logInfo(s"Retrieved version $version of $this for update")
+ store
+ }
+
+ /** Do maintenance backing data files, including creating snapshots and cleaning up old files */
+ override def doMaintenance(): Unit = {
+ try {
+ doSnapshot()
+ cleanup()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up $this")
+ }
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ /* Internal classes and methods */
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir =
+ new Path(id.checkpointLocation, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(hadoopConf)
+ private val sparkConf = Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
+
+ initialize()
+
+ private case class StoreFile(version: Long, path: Path, isSnapshot: Boolean)
+
+ /** Commit a set of updates to the store with the given new version */
+ private def commitUpdates(newVersion: Long, map: MapType, tempDeltaFile: Path): Path = {
+ synchronized {
+ val finalDeltaFile = deltaFile(newVersion)
+ fs.rename(tempDeltaFile, finalDeltaFile)
+ loadedMaps.put(newVersion, map)
+ finalDeltaFile
+ }
+ }
+
+ /**
+ * Get iterator of all the data of the latest version of the store.
+ * Note that this will look up the files to determined the latest known version.
+ */
+ private[state] def latestIterator(): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ val versionsInFiles = fetchFiles().map(_.version).toSet
+ val versionsLoaded = loadedMaps.keySet
+ val allKnownVersions = versionsInFiles ++ versionsLoaded
+ if (allKnownVersions.nonEmpty) {
+ loadMap(allKnownVersions.max).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ } else Iterator.empty
+ }
+
+ /** Get iterator of a specific version of the store */
+ private[state] def iterator(version: Long): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ loadMap(version).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ }
+
+ /** Initialize the store provider */
+ private def initialize(): Unit = {
+ if (!fs.exists(baseDir)) {
+ fs.mkdirs(baseDir)
+ } else {
+ if (!fs.isDirectory(baseDir)) {
+ throw new IllegalStateException(
+ s"Cannot use ${id.checkpointLocation} for storing state data for $this as" +
+ s"$baseDir already exists and is not a directory")
+ }
+ }
+ }
+
+ /** Load the required version of the map data from the backing files */
+ private def loadMap(version: Long): MapType = {
+ if (version <= 0) return new MapType
+ synchronized { loadedMaps.get(version) }.getOrElse {
+ val mapFromFile = readSnapshotFile(version).getOrElse {
+ val prevMap = loadMap(version - 1)
+ val newMap = new MapType(prevMap)
+ newMap.putAll(prevMap)
+ updateFromDeltaFile(version, newMap)
+ newMap
+ }
+ loadedMaps.put(version, mapFromFile)
+ mapFromFile
+ }
+ }
+
+ private def writeToDeltaFile(output: DataOutputStream, update: StoreUpdate): Unit = {
+
+ def writeUpdate(key: UnsafeRow, value: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ val valueBytes = value.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+
+ def writeRemove(key: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(-1)
+ }
+
+ update match {
+ case ValueAdded(key, value) =>
+ writeUpdate(key, value)
+ case ValueUpdated(key, value) =>
+ writeUpdate(key, value)
+ case KeyRemoved(key) =>
+ writeRemove(key)
+ }
+ }
+
+ private def finalizeDeltaFile(output: DataOutputStream): Unit = {
+ output.writeInt(-1) // Write this magic number to signify end of file
+ output.close()
+ }
+
+ private def updateFromDeltaFile(version: Long, map: MapType): Unit = {
+ val fileToRead = deltaFile(version)
+ if (!fs.exists(fileToRead)) {
+ throw new IllegalStateException(
+ s"Error reading delta file $fileToRead of $this: $fileToRead does not exist")
+ }
+ var input: DataInputStream = null
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while(!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading delta file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ map.remove(keyRow)
+ } else {
+ val valueRowBuffer = new Array[Byte](valueSize)
+ ByteStreams.readFully(input, valueRowBuffer, 0, valueSize)
+ val valueRow = new UnsafeRow(valueSchema.fields.length)
+ valueRow.pointTo(valueRowBuffer, valueSize)
+ map.put(keyRow, valueRow)
+ }
+ }
+ }
+ } finally {
+ if (input != null) input.close()
+ }
+ logInfo(s"Read delta file for version $version of $this from $fileToRead")
+ }
+
+ private def writeSnapshotFile(version: Long, map: MapType): Unit = {
+ val fileToWrite = snapshotFile(version)
+ var output: DataOutputStream = null
+ Utils.tryWithSafeFinally {
+ output = compressStream(fs.create(fileToWrite, false))
+ val iter = map.entrySet().iterator()
+ while(iter.hasNext) {
+ val entry = iter.next()
+ val keyBytes = entry.getKey.getBytes()
+ val valueBytes = entry.getValue.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+ output.writeInt(-1)
+ } {
+ if (output != null) output.close()
+ }
+ logInfo(s"Written snapshot file for version $version of $this at $fileToWrite")
+ }
+
+ private def readSnapshotFile(version: Long): Option[MapType] = {
+ val fileToRead = snapshotFile(version)
+ if (!fs.exists(fileToRead)) return None
+
+ val map = new MapType()
+ var input: DataInputStream = null
+
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while (!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading snapshot file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ throw new Exception(
--- End diff --
Updated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198621844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53593/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57090752
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEndpointRef, RpcEnv}
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.util.RpcUtils
+
+/** Trait representing all messages to [[StateStoreCoordinator]] */
+private sealed trait StateStoreCoordinatorMessage extends Serializable
+
+/** Classes representing messages */
+private case class ReportActiveInstance(storeId: StateStoreId, host: String, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class VerifyIfInstanceActive(storeId: StateStoreId, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class GetLocation(storeId: StateStoreId)
+ extends StateStoreCoordinatorMessage
+
+private case class DeactivateInstances(storeRootLocation: String)
+ extends StateStoreCoordinatorMessage
+
+private object StopCoordinator
+ extends StateStoreCoordinatorMessage
+
+/** Helper object used to create reference to [[StateStoreCoordinator]]. */
+private[sql] object StateStoreCoordinatorRef extends Logging {
+
+ private val endpointName = "StateStoreCoordinator"
+
+ /**
+ * Create a reference to a [[StateStoreCoordinator]], This can be called from driver as well as
+ * executors.
+ */
+ def apply(env: SparkEnv): StateStoreCoordinatorRef = synchronized {
--- End diff --
We need two methods: one for driver, another for executor.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196245518
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56377425
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,465 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
--- End diff --
I think it would be good to add more `logInfo` to all the major transitions with timing in this API since I imagine we will want to know that stuff for debugging / performance tuning. In particular:
- starting / ending a commit
- loading (locally and when needing to recover)
- compacting
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56374715
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.{Logging, SparkEnv}
+import org.apache.spark.sql.catalyst.InternalRow
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: InternalRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[InternalRow]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background management */
+ def manage(): Unit = { }
+}
+
+sealed trait StoreUpdate
+case class ValueAdded(key: InternalRow, value: InternalRow) extends StoreUpdate
+case class ValueUpdated(key: InternalRow, value: InternalRow) extends StoreUpdate
+case class KeyRemoved(key: InternalRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrive stores
+ * by their unique ids.
+ */
+private[state] object StateStore extends Logging {
+
+ private val MANAGEMENT_TASK_INTERVAL_SECS = 60
+
+ private val loadedProviders = new mutable.HashMap[StateStoreId, StateStoreProvider]()
+ private val managementTimer = new Timer("StateStore Timer", true)
+ @volatile private var managementTask: TimerTask = null
+
+ /** Get or create a store associated with the id. */
+ def get(storeId: StateStoreId, directory: String, version: Long): StateStore = {
+ require(version >= 0)
+ val storeProvider = loadedProviders.synchronized {
+ startIfNeeded()
+ val provider = loadedProviders.getOrElseUpdate(
+ storeId, new HDFSBackedStateStoreProvider(storeId, directory))
+ reportActiveInstance(storeId)
+ provider
+ }
+ storeProvider.getStore(version)
+ }
+
+ def remove(storeId: StateStoreId): Unit = loadedProviders.synchronized {
+ loadedProviders.remove(storeId)
+ }
+
+ def isLoaded(storeId: StateStoreId): Boolean = loadedProviders.synchronized {
+ loadedProviders.contains(storeId)
+ }
+
+ /** Unload and stop all state store provider */
+ def stop(): Unit = loadedProviders.synchronized {
+ loadedProviders.clear()
+ if (managementTask != null) {
+ managementTask.cancel()
+ managementTask = null
+ logInfo("StateStore stopped")
+ }
+ }
+
+ private def startIfNeeded(): Unit = loadedProviders.synchronized {
+ if (managementTask == null) {
+ managementTask = new TimerTask {
+ override def run(): Unit = { manageAll() }
+ }
+ val periodMs = Option(SparkEnv.get).map(_.conf) match {
--- End diff --
When does SparkEnv return null?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917481
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
--- End diff --
Changed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199991258
**[Test build #53810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53810/consoleFull)** for PR 11645 at commit [`b147f59`](https://github.com/apache/spark/commit/b147f599362ca4a6851f2b2d2140330bcdc6de0c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200083439
**[Test build #53845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53845/consoleFull)** for PR 11645 at commit [`b147f59`](https://github.com/apache/spark/commit/b147f599362ca4a6851f2b2d2140330bcdc6de0c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200133903
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197157762
**[Test build #53268 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53268/consoleFull)** for PR 11645 at commit [`8123818`](https://github.com/apache/spark/commit/81238189f560f5df846b36e5cb5e50532f6f82a3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196733132
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53178/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199570676
**[Test build #53728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53728/consoleFull)** for PR 11645 at commit [`534ad48`](https://github.com/apache/spark/commit/534ad483ed021f0e707cc27ae8adadbd2b585d83).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197157889
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197120718
**[Test build #53255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53255/consoleFull)** for PR 11645 at commit [`b5e2421`](https://github.com/apache/spark/commit/b5e242132ae98342113f27da4824aaf06485e291).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195174683
**[Test build #52897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52897/consoleFull)** for PR 11645 at commit [`f417bde`](https://github.com/apache/spark/commit/f417bde8bd65572e78125c7aa900221fa0bc4d7c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199544745
**[Test build #53729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53729/consoleFull)** for PR 11645 at commit [`f2aa71b`](https://github.com/apache/spark/commit/f2aa71b94b06a7bef093af414a4575ea693d6d96).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56227266
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
--- End diff --
fix indent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196219633
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195181283
**[Test build #52898 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52898/consoleFull)** for PR 11645 at commit [`d8cee54`](https://github.com/apache/spark/commit/d8cee54ad183cf899031fb762bba032c6a9cffd5).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57032303
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[(UnsafeRow, UnsafeRow)]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background management */
+ def doMaintenance(): Unit = { }
+}
+
+sealed trait StoreUpdate
+case class ValueAdded(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class ValueUpdated(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class KeyRemoved(key: UnsafeRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrieve stores
+ * by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null),
+ * it also runs a periodic background tasks to do maintenance on the loaded stores. For each
+ * store, tt uses the [[StateStoreCoordinator]] to ensure whether the current loaded instance of
+ * the store is the active instance. Accordingly, it either keeps it loaded and performance
+ * maintenance, or unloads the store.
+ */
+private[state] object StateStore extends Logging {
+
+ val MAINTENANCE_INTERVAL_CONFIG = "spark.sql.streaming.stateStore.maintenanceInterval"
+ val MAINTENANCE_INTERVAL_DEFAULT_SECS = 60
--- End diff --
This cannot be defined in SQLConf as this is a executor-wide configuration. I am renaming this to
`spark.streaming.stateStore.maintenanceInterval`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199577399
**[Test build #53731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53731/consoleFull)** for PR 11645 at commit [`19a60a6`](https://github.com/apache/spark/commit/19a60a65b77b73cf69cde961c034d04ad17df436).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class StateStoreId(checkpointLocation: String, operatorId: Long, partitionId: Int)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196245464
**[Test build #53070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53070/consoleFull)** for PR 11645 at commit [`a0ba498`](https://github.com/apache/spark/commit/a0ba498185286a921ef303dbc087dad5d226fa41).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class ValueUpdated(key: InternalRow, value: InternalRow) extends StoreUpdate`
* ` case class KeyRemoved(key: InternalRow) extends StoreUpdate`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196609482
**[Test build #53147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53147/consoleFull)** for PR 11645 at commit [`34ae7ff`](https://github.com/apache/spark/commit/34ae7ffcb3544743e90a6441c78fd9002e839794).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56227289
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends StateStoreProvider with Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+
+ class HDFSBackedStateStore( val version: Long, mapToUpdate: MapType)
--- End diff --
add doc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196637005
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53147/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56227397
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends StateStoreProvider with Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+
+ class HDFSBackedStateStore( val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream =
+ serializer.newInstance().serializeStream(fs.create(tempDeltaFile, true))
+ private val allUpdates = new mutable.HashMap[InternalRow, StoreUpdate]
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /** Update the value of a key using the value generated by the update function */
+ override def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = mapToUpdate.get(key)
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+ allUpdates.get(key) match {
+ case Some(ValueAdded(_, _)) =>
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ tempDeltaFileStream.writeObject(ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: InternalRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keysIterator
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ mapToUpdate.remove(key)
+ allUpdates.get(key) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
--- End diff --
add more inline comments to explain the cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196704686
**[Test build #53178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53178/consoleFull)** for PR 11645 at commit [`13c29a2`](https://github.com/apache/spark/commit/13c29a2e8a3a6b2bed0bc14c62ed1be4033b3e55).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198621843
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200017608
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53804/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197032639
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57065449
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEndpointRef, RpcEnv}
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.util.RpcUtils
+
+/** Trait representing all messages to [[StateStoreCoordinator]] */
+private sealed trait StateStoreCoordinatorMessage extends Serializable
+
+/** Classes representing messages */
+private case class ReportActiveInstance(storeId: StateStoreId, host: String, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class VerifyIfInstanceActive(storeId: StateStoreId, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class GetLocation(storeId: StateStoreId)
+ extends StateStoreCoordinatorMessage
+
+private case class DeactivateInstances(storeRootLocation: String)
+ extends StateStoreCoordinatorMessage
+
+private object StopCoordinator
+ extends StateStoreCoordinatorMessage
+
+/** Helper object used to create reference to [[StateStoreCoordinator]]. */
+private[sql] object StateStoreCoordinatorRef extends Logging {
+
+ private val endpointName = "StateStoreCoordinator"
+
+ /**
+ * Create a reference to a [[StateStoreCoordinator]], This can be called from driver as well as
+ * executors.
+ */
+ def apply(env: SparkEnv): StateStoreCoordinatorRef = synchronized {
+ try {
+ val coordinator = new StateStoreCoordinator(env.rpcEnv)
+ val coordinatorRef = env.rpcEnv.setupEndpoint(endpointName, coordinator)
+ logInfo("Registered StateStoreCoordinator endpoint")
+ new StateStoreCoordinatorRef(coordinatorRef)
+ } catch {
+ case e: IllegalArgumentException =>
+ logDebug("Retrieving existing StateStoreCoordinator endpoint")
+ val rpcEndpointRef = RpcUtils.makeDriverRef(endpointName, env.conf, env.rpcEnv)
+ new StateStoreCoordinatorRef(rpcEndpointRef)
+ }
+ }
+}
+
+/**
+ * Reference to a [[StateStoreCoordinator]] that can be used to coordinator instances of
+ * [[StateStore]]s across all the executors, and get their locations for job scheduling.
+ */
+private[sql] class StateStoreCoordinatorRef private(rpcEndpointRef: RpcEndpointRef) {
+
+ private[state] def reportActiveInstance(
+ storeId: StateStoreId,
+ host: String,
+ executorId: String): Unit = {
+ rpcEndpointRef.send(ReportActiveInstance(storeId, host, executorId))
+ }
+
+ /** Verify whether the given executor has the active instance of a state store */
+ private[state] def verifyIfInstanceActive(storeId: StateStoreId, executorId: String): Boolean = {
+ rpcEndpointRef.askWithRetry[Boolean](VerifyIfInstanceActive(storeId, executorId))
+ }
+
+ /** Get the location of the state store */
+ private[state] def getLocation(storeId: StateStoreId): Option[String] = {
+ rpcEndpointRef.askWithRetry[Option[String]](GetLocation(storeId))
+ }
+
+ /** Deactivate instances related to a set of operator */
+ private[state] def deactivateInstances(storeRootLocation: String): Unit = {
+ rpcEndpointRef.askWithRetry[Boolean](DeactivateInstances(storeRootLocation))
+ }
+
+ private[state] def stop(): Unit = {
+ rpcEndpointRef.askWithRetry[Boolean](StopCoordinator)
+ }
+}
+
+
+/**
+ * Class for coordinating instances of [[StateStore]]s loaded in executors across the cluster,
+ * and get their locations for job scheduling.
+ */
+private class StateStoreCoordinator(override val rpcEnv: RpcEnv) extends RpcEndpoint {
+ private val instances = new mutable.HashMap[StateStoreId, ExecutorCacheTaskLocation]
+
+ override def receive: PartialFunction[Any, Unit] = {
+ case ReportActiveInstance(id, host, executorId) =>
+ instances.put(id, ExecutorCacheTaskLocation(host, executorId))
+ }
+
+ override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
+ case VerifyIfInstanceActive(id, execId) =>
+ val response = instances.get(id) match {
+ case Some(location) => location.executorId == execId
+ case None => false
+ }
+ context.reply(response)
+
+ case GetLocation(id) =>
+ context.reply(instances.get(id).map(_.toString))
+
+ case DeactivateInstances(loc) =>
+ val storeIdsToRemove =
+ instances.keys.filter(_.checkpointLocation == loc).toSeq
+ instances --= storeIdsToRemove
--- End diff --
nit: You can use one line code here: `instances.retain((storeId, _) => storeId.checkpointLocation != loc)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917090
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
--- End diff --
How is this different than just not committing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197117445
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56373801
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.{Logging, SparkEnv}
+import org.apache.spark.sql.catalyst.InternalRow
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit
--- End diff --
Using lambdas here might make this hard to codegen (we emit java code). Its okay for now though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56376116
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDDSuite.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+import java.nio.file.Files
+
+import scala.util.Random
+
+import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.rdd.RDD
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.util.Utils
+
+class StateStoreRDDSuite extends SparkFunSuite with BeforeAndAfter with BeforeAndAfterAll {
+
+ private val conf = new SparkConf().setMaster("local").setAppName(this.getClass.getCanonicalName)
+ private var tempDir = Files.createTempDirectory("StateStoreRDDSuite").toString
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ after {
+ StateStore.stop()
+ }
+
+ override def afterAll(): Unit = {
+ super.afterAll()
+ Utils.deleteRecursively(new File(tempDir))
+ }
+
+ test("versioning and immutability") {
+ quietly {
+ withSpark(new SparkContext(conf)) { sc =>
+ val path = Utils.createDirectory(tempDir, Random.nextString(10)).toString
+ val increment = (store: StateStore, iter: Iterator[String]) => {
+ iter.foreach { s =>
+ store.update(
+ wrapKey(s), oldRow => {
+ val oldValue = oldRow.map(unwrapValue).getOrElse(0)
+ wrapValue(oldValue + 1)
+ })
+ }
+ store.commit()
+ store.iterator().map(unwrapKeyValue)
+ }
+ val opId = 0
+ val rdd1 = makeRDD(sc, Seq("a", "b", "a"))
+ .withStateStores(increment, opId, storeVersion = 0, path)
+ assert(rdd1.collect().toSet === Set("a" -> 2, "b" -> 1))
+
+ // Generate next version of stores
+ val rdd2 = makeRDD(sc, Seq("a", "c"))
+ .withStateStores(increment, opId, storeVersion = 1, path)
+ assert(rdd2.collect().toSet === Set("a" -> 3, "b" -> 1, "c" -> 1))
+
+ // Make sure the previous RDD still has the same data.
+ assert(rdd1.collect().toSet === Set("a" -> 2, "b" -> 1))
+ }
+ }
+ }
+
+ test("recovering from files") {
+ quietly {
+ val opId = 0
+ val path = Utils.createDirectory(tempDir, Random.nextString(10)).toString
--- End diff --
createTempDir
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56376021
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala ---
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+
+import scala.collection.mutable
+import scala.util.Random
+
+import org.apache.hadoop.fs.Path
+import org.scalatest.{BeforeAndAfter, PrivateMethodTester}
+import org.scalatest.concurrent.Eventually._
+import org.scalatest.time.SpanSugar._
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+class StateStoreSuite extends SparkFunSuite with BeforeAndAfter with PrivateMethodTester {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ private val tempDir = Utils.createTempDir().toString
+
+ after {
+ StateStore.stop()
+ }
+
+ test("update, remove, commit, and all data iterator") {
+ val provider = newStoreProvider()
+
+ // Verify state before starting a new set of updates
+ assert(provider.latestIterator().isEmpty)
+
+ val store = provider.getStore(0)
+ assert(!store.hasCommitted)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+
+ // Verify state after updating
+ update(store, "a", 1)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+ assert(provider.latestIterator().isEmpty)
+
+ // Make updates, commit and then verify state
+ update(store, "b", 2)
+ update(store, "aa", 3)
+ remove(store, _.startsWith("a"))
+ assert(store.commit() === 1)
+
+ assert(store.hasCommitted)
+ assert(unwrapToSet(store.iterator()) === Set("b" -> 2))
+ assert(unwrapToSet(provider.latestIterator()) === Set("b" -> 2))
+ assert(fileExists(provider, version = 1, isSnapshot = false))
+ assert(getDataFromFiles(provider) === Set("b" -> 2))
+
+ // Trying to get newer versions should fail
+ intercept[Exception] {
+ provider.getStore(2)
+ }
+ intercept[Exception] {
+ getDataFromFiles(provider, 2)
+ }
+
+ // New updates to the reloaded store with new version, and does not change old version
+ val reloadedStore = new HDFSBackedStateStoreProvider(store.id, provider.directory).getStore(1)
+ update(reloadedStore, "c", 4)
+ assert(reloadedStore.commit() === 2)
+ assert(unwrapToSet(reloadedStore.iterator()) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider, version = 1) === Set("b" -> 2))
+ assert(getDataFromFiles(provider, version = 2) === Set("b" -> 2, "c" -> 4))
+ }
+
+ test("updates iterator with all combos of updates and removes") {
+ val provider = newStoreProvider()
+ var currentVersion: Int = 0
+ def withStore(body: StateStore => Unit): Unit = {
+ val store = provider.getStore(currentVersion)
+ body(store)
+ currentVersion += 1
+ }
+
+ // New data should be seen in updates as value added, even if they had multiple updates
+ withStore { store =>
+ update(store, "a", 1)
+ update(store, "aa", 1)
+ update(store, "aa", 2)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Added("a", 1), Added("aa", 2)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1, "aa" -> 2))
+ }
+
+ // Multiple updates to same key should be collapsed in the updates as a single value update
+ // Keys that have not been updated should not appear in the updates
+ withStore { store =>
+ update(store, "a", 4)
+ update(store, "a", 6)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 6)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Keys added, updated and finally removed before commit should not appear in updates
+ withStore { store =>
+ update(store, "b", 4) // Added, finally removed
+ update(store, "bb", 5) // Added, updated, finally removed
+ update(store, "bb", 6)
+ remove(store, _.startsWith("b"))
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set.empty)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Removed data should be seen in updates as a key removed
+ // Removed, but re-added data should be seen in updates as a value update
+ withStore { store =>
+ remove(store, _.startsWith("a"))
+ update(store, "a", 10)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 10), Removed("aa")))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 10))
+ }
+ }
+
+ test("cancel") {
+ val provider = newStoreProvider()
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ store.commit()
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ // cancelUpdates should not change the data in the files
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ store1.cancel()
+ assert(getDataFromFiles(provider) === Set("a" -> 1))
+ }
+
+ test("getStore with unexpected versions") {
+ val provider = newStoreProvider()
+
+ intercept[IllegalArgumentException] {
+ provider.getStore(-1)
+ }
+
+ // Prepare some data in the stoer
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ assert(store.commit() === 1)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ intercept[IllegalStateException] {
+ provider.getStore(2)
+ }
+
+ // Update store version with some data
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ assert(store1.commit() === 2)
+ assert(unwrapToSet(store1.iterator()) === Set("a" -> 1, "b" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "b" -> 1))
+
+ // Overwrite the version with other data
+ val store2 = provider.getStore(1)
+ update(store2, "c", 1)
+ assert(store2.commit() === 2)
+ assert(unwrapToSet(store2.iterator()) === Set("a" -> 1, "c" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "c" -> 1))
+ }
+
+ test("snapshotting") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ var currentVersion = 0
+ def updateVersionTo(targetVersion: Int): Unit = {
+ for (i <- currentVersion + 1 to targetVersion) {
+ val store = provider.getStore(currentVersion)
+ update(store, "a", i)
+ store.commit()
+ currentVersion += 1
+ }
+ require(currentVersion === targetVersion)
+ }
+
+ updateVersionTo(2)
+ require(getDataFromFiles(provider) === Set("a" -> 2))
+ provider.manage() // should not generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 2))
+
+ for (i <- 1 to currentVersion) {
+ assert(fileExists(provider, i, isSnapshot = false)) // all delta files present
+ assert(!fileExists(provider, i, isSnapshot = true)) // no snapshot files present
+ }
+
+ // After version 6, snapshotting should generate one snapshot file
+ updateVersionTo(6)
+ require(getDataFromFiles(provider) === Set("a" -> 6), "store not updated correctly")
+ provider.manage() // should generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 6), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 6))
+
+ val snapshotVersion = (0 to 6).find(version => fileExists(provider, version, isSnapshot = true))
+ assert(snapshotVersion.nonEmpty, "snapshot file not generated")
+
+ // After version 20, snapshotting should generate newer snapshot files
+ updateVersionTo(20)
+ require(getDataFromFiles(provider) === Set("a" -> 20), "store not updated correctly")
+ provider.manage() // do snapshot
+ assert(getDataFromFiles(provider) === Set("a" -> 20), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 20))
+
+ val latestSnapshotVersion = (0 to 20).filter(version =>
+ fileExists(provider, version, isSnapshot = true)).lastOption
+ assert(latestSnapshotVersion.nonEmpty, "no snapshot file found")
+ assert(latestSnapshotVersion.get > snapshotVersion.get, "newer snapshot not generated")
+
+ }
+
+ test("cleaning") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ for (i <- 1 to 20) {
+ val store = provider.getStore(i - 1)
+ update(store, "a", i)
+ store.commit()
+ provider.manage() // do cleanup
+ }
+ require(
+ unwrapToSet(provider.latestIterator()) === Set("a" -> 20),
+ "store not updated correctly")
+
+ assert(!fileExists(provider, version = 1, isSnapshot = false)) // first file should be deleted
+
+ // last couple of versions should be retrievable
+ assert(getDataFromFiles(provider, 20) === Set("a" -> 20))
+ assert(getDataFromFiles(provider, 19) === Set("a" -> 19))
+ }
+
+ test("StateStore.get") {
+ quietly {
+ val dir = Utils.createDirectory(tempDir, Random.nextString(5)).toString
--- End diff --
`createTempDir` will take care of random ness and cleanup after itself
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197113005
**[Test build #53250 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53250/consoleFull)** for PR 11645 at commit [`b8b4632`](https://github.com/apache/spark/commit/b8b4632b5ed9be181456da4f53156a0cbebebaf3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56227651
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.reflect.ClassTag
+import scala.util.control.NonFatal
+
+import org.apache.spark.{Logging, SparkEnv}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.util.RpcUtils
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ import StateStore._
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: InternalRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[InternalRow]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background management */
+ def manage(): Unit = { }
+}
+
+sealed trait StoreUpdate
+case class ValueAdded(key: InternalRow, value: InternalRow) extends StoreUpdate
+case class ValueUpdated(key: InternalRow, value: InternalRow) extends StoreUpdate
+case class KeyRemoved(key: InternalRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrive stores
+ * by their unique ids.
+ */
+private[state] object StateStore extends Logging {
+
+ val MANAGEMENT_TASK_INTERVAL_SECS = 60
--- End diff --
should this be configurable by some conf?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198596901
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197068486
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197032614
**[Test build #53225 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53225/consoleFull)** for PR 11645 at commit [`d5e2b10`](https://github.com/apache/spark/commit/d5e2b10ad11b22ea020dd3f3bb13a1a64aa33b50).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917929
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,597 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.serializer.KryoSerializer
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ sparkConf: SparkConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ import HDFSBackedStateStoreProvider._
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ // private val tempDeltaFileStream = fs.create(tempDeltaFile, true)
--- End diff --
remove?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200097875
**[Test build #53857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53857/consoleFull)** for PR 11645 at commit [`70cc7b1`](https://github.com/apache/spark/commit/70cc7b11c0d502d4a741ccdcab5e7aa006b5b311).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200121936
**[Test build #53845 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53845/consoleFull)** for PR 11645 at commit [`b147f59`](https://github.com/apache/spark/commit/b147f599362ca4a6851f2b2d2140330bcdc6de0c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197068481
**[Test build #53241 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53241/consoleFull)** for PR 11645 at commit [`f5660d2`](https://github.com/apache/spark/commit/f5660d23ce80f84e4d28235805d1c64060aad382).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57062898
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.Timer
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.ThreadUtils
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(checkpointLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store, and return the new version.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[(UnsafeRow, UnsafeRow)]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+/** Trait representing a provider of a specific version of a [[StateStore]]. */
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background maintenance */
+ def doMaintenance(): Unit = { }
+}
+
+/** Trait representing updates made to a [[StateStore]]. */
+sealed trait StoreUpdate
+
+case class ValueAdded(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+
+case class ValueUpdated(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+
+case class KeyRemoved(key: UnsafeRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrieve stores
+ * by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null),
+ * it also runs a periodic background tasks to do maintenance on the loaded stores. For each
+ * store, tt uses the [[StateStoreCoordinator]] to ensure whether the current loaded instance of
+ * the store is the active instance. Accordingly, it either keeps it loaded and performs
+ * maintenance, or unloads the store.
+ */
+private[state] object StateStore extends Logging {
+
+ val MAINTENANCE_INTERVAL_CONFIG = "spark.streaming.stateStore.maintenanceInterval"
+ val MAINTENANCE_INTERVAL_DEFAULT_SECS = 60
+
+ private val loadedProviders = new mutable.HashMap[StateStoreId, StateStoreProvider]()
+ private val maintenanceTimer = new Timer("StateStore Timer", true)
+ private val maintenanceTaskExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("state-store-maintenance-task")
+
+ @volatile private var maintenanceTask: ScheduledFuture[_] = null
+ @volatile private var _coordRef: StateStoreCoordinatorRef = null
+
+ /** Get or create a store associated with the id. */
+ def get(
+ storeId: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ version: Long,
+ storeConf: StateStoreConf,
+ hadoopConf: Configuration): StateStore = {
+ require(version >= 0)
+ val storeProvider = loadedProviders.synchronized {
+ startMaintenanceIfNeeded()
+ val provider = loadedProviders.getOrElseUpdate(
+ storeId,
+ new HDFSBackedStateStoreProvider(storeId, keySchema, valueSchema, storeConf, hadoopConf))
+ reportActiveStoreInstance(storeId)
+ provider
+ }
+ storeProvider.getStore(version)
+ }
+
+ /** Unload a state store provider */
+ def unload(storeId: StateStoreId): Unit = loadedProviders.synchronized {
+ loadedProviders.remove(storeId)
+ }
+
+ /** Whether a state store provider is loaded or not */
+ def isLoaded(storeId: StateStoreId): Boolean = loadedProviders.synchronized {
+ loadedProviders.contains(storeId)
+ }
+
+ /** Unload and stop all state store providers */
+ def stop(): Unit = loadedProviders.synchronized {
+ loadedProviders.clear()
+ _coordRef = null
+ if (maintenanceTask != null) {
+ maintenanceTask.cancel(false)
+ maintenanceTask = null
+ }
+ logInfo("StateStore stopped")
--- End diff --
You need to call `maintenanceTaskExecutor.shutdown` before this line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56920464
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEndpointRef, RpcEnv}
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.util.RpcUtils
+
+/** Trait representing all messages to [[StateStoreCoordinator]] */
+private sealed trait StateStoreCoordinatorMessage extends Serializable
+
+/** Classes representing messages */
+private case class ReportActiveInstance(storeId: StateStoreId, host: String, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class VerifyIfInstanceActive(storeId: StateStoreId, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class GetLocation(storeId: StateStoreId)
+ extends StateStoreCoordinatorMessage
+
+private case class DeactivateInstances(storeRootLocation: String)
+ extends StateStoreCoordinatorMessage
+
+private object StopCoordinator
+ extends StateStoreCoordinatorMessage
+
+/** Helper object used to create reference to [[StateStoreCoordinator]]. */
+private[sql] object StateStoreCoordinatorRef extends Logging {
+
+ private val endpointName = "StateStoreCoordinator"
+
+ /**
+ * Create a reference to a [[StateStoreCoordinator]], This can be called from driver as well as
+ * executors.
+ */
+ def apply(env: SparkEnv): StateStoreCoordinatorRef = synchronized {
+ try {
+ val coordinator = new StateStoreCoordinator(env.rpcEnv)
+ val coordinatorRef = env.rpcEnv.setupEndpoint(endpointName, coordinator)
+ logInfo("Registered StateStoreCoordinator endpoint")
+ new StateStoreCoordinatorRef(coordinatorRef)
+ } catch {
+ case e: IllegalArgumentException =>
--- End diff --
There isnt any API to do that at this point, definitely not atomically. @zsxwing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199631000
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56227488
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends StateStoreProvider with Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+
+ class HDFSBackedStateStore( val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream =
+ serializer.newInstance().serializeStream(fs.create(tempDeltaFile, true))
+ private val allUpdates = new mutable.HashMap[InternalRow, StoreUpdate]
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /** Update the value of a key using the value generated by the update function */
+ override def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = mapToUpdate.get(key)
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+ allUpdates.get(key) match {
+ case Some(ValueAdded(_, _)) =>
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ tempDeltaFileStream.writeObject(ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: InternalRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keysIterator
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ mapToUpdate.remove(key)
+ allUpdates.get(key) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ tempDeltaFileStream.writeObject(KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ tempDeltaFileStream.close()
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[InternalRow] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.valuesIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ newMap ++= loadMap(version)
+ }
+ new HDFSBackedStateStore(version, newMap)
+ }
+
+ override def manage(): Unit = {
+ try {
+ doSnapshot()
+ cleanup()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up $this")
+ }
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ /* Internal classes and methods */
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir = new Path(directory, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(new Configuration())
+ private val serializer = new KryoSerializer(new SparkConf)
+
+ initialize()
+
+ private case class StoreFile(version: Long, path: Path, isSnapshot: Boolean)
+
+ /** Commit a set of updates to the store with the given new version */
+ private def commitUpdates(newVersion: Long, map: MapType, tempDeltaFile: Path): Path = {
+ synchronized {
+ val finalDeltaFile = deltaFile(newVersion)
+ fs.rename(tempDeltaFile, finalDeltaFile)
+ loadedMaps.put(newVersion, map)
+ finalDeltaFile
+ }
+ }
+
+ /**
+ * Get iterator of all the data of the latest version of the store.
+ * Note that this will look up the files to determined the latest known version.
+ */
+ private[state] def latestIterator(): Iterator[InternalRow] = synchronized {
+ val versionsInFiles = fetchFiles().map(_.version).toSet
+ val versionsLoaded = loadedMaps.keySet
+ val allKnownVersions = versionsInFiles ++ versionsLoaded
+ if (allKnownVersions.nonEmpty) {
+ loadMap(allKnownVersions.max)
+ .iterator
+ .map { case (key, value) => new JoinedRow(key, value) }
+ } else Iterator.empty
+ }
+
+ /** Get iterator of a specific version of the store */
+ private[state] def iterator(version: Long): Iterator[InternalRow] = synchronized {
+ loadMap(version)
+ .iterator
+ .map { case (key, value) => new JoinedRow(key, value) }
+ }
+
+ /** Initialize the store provider */
+ private def initialize(): Unit = {
+ if (!fs.exists(baseDir)) {
+ fs.mkdirs(baseDir)
+ } else {
+ if (!fs.isDirectory(baseDir)) {
+ throw new IllegalStateException(
+ s"Cannot use $directory for storing state data as" +
+ s"$baseDir already exists and is not a directory")
+ }
+ }
+ }
+
+ /** Load the required version of the map data from the backing files */
+ private def loadMap(version: Long): MapType = {
+ if (version <= 0) return new MapType
+ synchronized { loadedMaps.get(version) }.getOrElse {
+ val mapFromFile = readSnapshotFile(version).getOrElse {
+ val prevMap = loadMap(version - 1)
+ val deltaUpdates = readDeltaFile(version)
+ val newMap = new MapType()
+ newMap ++= prevMap
+ newMap.sizeHint(prevMap.size)
+ while (deltaUpdates.hasNext) {
+ deltaUpdates.next match {
+ case ValueAdded(key, value) => newMap.put(key, value)
+ case ValueUpdated(key, value) => newMap.put(key, value)
+ case KeyRemoved(key) => newMap.remove(key)
+ }
+ }
+ newMap
+ }
+ loadedMaps.put(version, mapFromFile)
+ mapFromFile
+ }
+ }
+
+ private def readDeltaFile(version: Long): Iterator[StoreUpdate] = {
+ val fileToRead = deltaFile(version)
+ if (!fs.exists(fileToRead)) {
+ throw new IllegalStateException(
+ s"Cannot read delta file $fileToRead of $this: $fileToRead does not exist")
+ }
+ val deser = serializer.newInstance()
+ var deserStream: DeserializationStream = null
+ deserStream = deser.deserializeStream(fs.open(fileToRead))
+ val iter = deserStream.asIterator.asInstanceOf[Iterator[StoreUpdate]]
+ CompletionIterator[StoreUpdate, Iterator[StoreUpdate]](
+ iter, {
+ deserStream.close()
+ })
+ }
+
+ private def writeSnapshotFile(version: Long, map: MapType): Unit = {
+ val fileToWrite = snapshotFile(version)
+ val ser = serializer.newInstance()
+ var outputStream: SerializationStream = null
+ Utils.tryWithSafeFinally {
+ outputStream = ser.serializeStream(fs.create(fileToWrite, false))
+ outputStream.writeAll(map.iterator)
+ } {
+ if (outputStream != null) outputStream.close()
+ }
+ }
+
+ private def readSnapshotFile(version: Long): Option[MapType] = {
+ val fileToRead = snapshotFile(version)
+ if (!fs.exists(fileToRead)) return None
+
+ val deser = serializer.newInstance()
+ val map = new MapType()
+ var deserStream: DeserializationStream = null
+
+ try {
+ deserStream = deser.deserializeStream(fs.open(fileToRead))
+ val iter = deserStream.asIterator.asInstanceOf[Iterator[(InternalRow, InternalRow)]]
+ while (iter.hasNext) {
+ map += iter.next()
+ }
+ Some(map)
+ } finally {
+ if (deserStream != null) deserStream.close()
+ }
+ }
+
+
+ /** Perform a snapshot of the store to allow delta files to be consolidated */
+ private def doSnapshot(): Unit = {
+ try {
+ val files = fetchFiles()
+ if (files.nonEmpty) {
+ val lastVersion = files.last.version
+ val deltaFilesForLastVersion =
+ filesForVersion(files, lastVersion).filter(_.isSnapshot == false)
+ synchronized { loadedMaps.get(lastVersion) } match {
+ case Some(map) =>
+ if (deltaFilesForLastVersion.size > maxDeltaChainForSnapshots) {
+ writeSnapshotFile(lastVersion, map)
+ }
+ case None =>
+ // The last map is not loaded, probably some other instance is incharge
+ }
+
+ }
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error doing snapshots for $this", e)
+ }
+ }
+
+ /**
+ * Clean up old snapshots and delta files that are not needed any more. It ensures that last
+ * few versions of the store can be recovered from the files, so re-executed RDD operations
+ * can re-apply updates on the past versions of the store.
+ */
+ private[state] def cleanup(): Unit = {
+ try {
+ val files = fetchFiles()
+ if (files.nonEmpty) {
+ val earliestVersionToRetain = files.last.version - numBatchesToRetain
+ if (earliestVersionToRetain > 0) {
+ val earliestFileToRetain = filesForVersion(files, earliestVersionToRetain).head
+ synchronized {
+ loadedMaps.keys.filter(_ < earliestVersionToRetain).foreach(loadedMaps.remove)
+ }
+ files.filter(_.version < earliestFileToRetain.version).foreach { f =>
+ fs.delete(f.path, true)
+ }
+ }
+ }
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error cleaning up files for $this", e)
+ }
+ }
+
+ private def filesForVersion(allFiles: Seq[StoreFile], version: Long): Seq[StoreFile] = {
+ require(version >= 0)
+ require(allFiles.exists(_.version == version))
+
+ val latestSnapshotFileBeforeVersion = allFiles
+ .filter(_.isSnapshot == true)
+ .takeWhile(_.version <= version)
+ .lastOption
+ val deltaBatchFiles = latestSnapshotFileBeforeVersion match {
+ case Some(snapshotFile) =>
+ val deltaBatchIds = (snapshotFile.version + 1) to version
+
+ val deltaFiles = allFiles.filter { file =>
+ file.version > snapshotFile.version && file.version <= version
+ }
+ verify(
+ deltaFiles.size == version - snapshotFile.version,
+ s"Unexpected list of delta files for version $version: ${deltaFiles.mkString(",")}"
+ )
+ deltaFiles
+
+ case None =>
+ allFiles.takeWhile(_.version <= version)
+ }
+ latestSnapshotFileBeforeVersion.toSeq ++ deltaBatchFiles
+ }
+
+ private def fetchFiles(): Seq[StoreFile] = {
--- End diff --
add docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196636867
**[Test build #53147 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53147/consoleFull)** for PR 11645 at commit [`34ae7ff`](https://github.com/apache/spark/commit/34ae7ffcb3544743e90a6441c78fd9002e839794).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56227691
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEnv}
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.sql.execution.streaming.state.StateStoreCoordinator.StateStoreCoordinatorEndpoint
+
+private sealed trait StateStoreCoordinatorMessage extends Serializable
+private case class ReportActiveInstance(storeId: StateStoreId, host: String, executorId: String)
+ extends StateStoreCoordinatorMessage
+private case class VerifyIfInstanceActive(storeId: StateStoreId, executorId: String)
+ extends StateStoreCoordinatorMessage
+private object StopCoordinator extends StateStoreCoordinatorMessage
+
+
+class StateStoreCoordinator(rpcEnv: RpcEnv) {
+ private val coordinatorRef = rpcEnv.setupEndpoint(
+ "StateStoreCoordinator", new StateStoreCoordinatorEndpoint(rpcEnv, this))
+ private val instances = new mutable.HashMap[StateStoreId, ExecutorCacheTaskLocation]
+
+ def reportActiveInstance(storeId: StateStoreId, host: String, executorId: String): Boolean = {
+ instances.synchronized { instances.put(storeId, ExecutorCacheTaskLocation(host, executorId)) }
+ true
+ }
+
+ def verifyIfInstanceActive(storeId: StateStoreId, executorId: String): Boolean = {
+ instances.synchronized {
+ instances.get(storeId).forall(_.executorId == executorId)
+ }
+ }
+
+ def getLocation(storeId: StateStoreId): Option[String] = {
+ instances.synchronized { instances.get(storeId).map(_.toString) }
+ }
+
+ def makeInstancesInactive(operatorIds: Set[Long]): Unit = {
+ instances.synchronized {
+ val instancesToRemove =
+ instances.keys.filter(id => operatorIds.contains(id.operatorId)).toSeq
+ instances --= instancesToRemove
+ }
+ }
+}
+
+private[spark] object StateStoreCoordinator {
+
+ private[spark] class StateStoreCoordinatorEndpoint(
+ override val rpcEnv: RpcEnv, coordinator: StateStoreCoordinator)
+ extends RpcEndpoint with Logging {
+
+ override def receive: PartialFunction[Any, Unit] = {
+ case StopCoordinator =>
+ logInfo("StateStoreCoordinator stopped!")
+ stop()
+ }
+
+ override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
+ case ReportActiveInstance(id, host, executorId) =>
+ context.reply(coordinator.reportActiveInstance(id, host, executorId))
+ case VerifyIfInstanceActive(id, executor) =>
+ context.reply(coordinator.verifyIfInstanceActive(id, executor))
+ }
+ }
+}
+
+
--- End diff --
remove extra line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199547419
**[Test build #53729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53729/consoleFull)** for PR 11645 at commit [`f2aa71b`](https://github.com/apache/spark/commit/f2aa71b94b06a7bef093af414a4575ea693d6d96).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class StateStoreId(checkpointLocation: String, operatorId: Long, partitionId: Int)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198627157
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917331
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[(UnsafeRow, UnsafeRow)]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background management */
+ def doMaintenance(): Unit = { }
+}
+
+sealed trait StoreUpdate
+case class ValueAdded(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class ValueUpdated(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class KeyRemoved(key: UnsafeRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrieve stores
+ * by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null),
+ * it also runs a periodic background tasks to do maintenance on the loaded stores. For each
+ * store, tt uses the [[StateStoreCoordinator]] to ensure whether the current loaded instance of
+ * the store is the active instance. Accordingly, it either keeps it loaded and performance
+ * maintenance, or unloads the store.
+ */
+private[state] object StateStore extends Logging {
+
+ val MAINTENANCE_INTERVAL_CONFIG = "spark.sql.streaming.stateStore.maintenanceInterval"
+ val MAINTENANCE_INTERVAL_DEFAULT_SECS = 60
--- End diff --
These should be defined in `SQLConf`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196732147
**[Test build #53178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53178/consoleFull)** for PR 11645 at commit [`13c29a2`](https://github.com/apache/spark/commit/13c29a2e8a3a6b2bed0bc14c62ed1be4033b3e55).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ValueAdded(key: InternalRow, value: InternalRow) extends StoreUpdate`
* `case class ValueUpdated(key: InternalRow, value: InternalRow) extends StoreUpdate`
* `case class KeyRemoved(key: InternalRow) extends StoreUpdate`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196606910
**[Test build #53146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53146/consoleFull)** for PR 11645 at commit [`22d7e66`](https://github.com/apache/spark/commit/22d7e6639125a507467766c45393e36b3ee92f3f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by rishitesh <gi...@git.apache.org>.
Github user rishitesh commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56125425
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{Partition, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+/**
+ * Created by tdas on 3/9/16.
+ */
+class StateStoreRDD[INPUT: ClassTag, OUTPUT: ClassTag](
+ dataRDD: RDD[INPUT],
+ storeUpdateFunction: (StateStore, Iterator[INPUT]) => Iterator[OUTPUT],
+ operatorId: Long,
+ storeVersion: Long,
+ storeDirectory: String,
+ storeCoordinator: StateStoreCoordinator) extends RDD[OUTPUT](dataRDD) {
+
+ val nextVersion = storeVersion + 1
+
+ override protected def getPartitions: Array[Partition] = dataRDD.partitions
+ override def getPreferredLocations(partition: Partition): Seq[String] = {
+ Seq.empty
--- End diff --
Should not you be using preffered location here ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199599988
**[Test build #53740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53740/consoleFull)** for PR 11645 at commit [`5b7cf53`](https://github.com/apache/spark/commit/5b7cf538cf4b6485ddee102568e931d66f9c1a75).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199963041
**[Test build #53805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53805/consoleFull)** for PR 11645 at commit [`63fad92`](https://github.com/apache/spark/commit/63fad922f166c7fdaf21783f89d01e3dec13c0f6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917126
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[(UnsafeRow, UnsafeRow)]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background management */
+ def doMaintenance(): Unit = { }
+}
+
+sealed trait StoreUpdate
+case class ValueAdded(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class ValueUpdated(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class KeyRemoved(key: UnsafeRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrieve stores
+ * by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null),
+ * it also runs a periodic background tasks to do maintenance on the loaded stores. For each
+ * store, tt uses the [[StateStoreCoordinator]] to ensure whether the current loaded instance of
+ * the store is the active instance. Accordingly, it either keeps it loaded and performance
+ * maintenance, or unloads the store.
+ */
+private[state] object StateStore extends Logging {
+
+ val MAINTENANCE_INTERVAL_CONFIG = "spark.sql.streaming.stateStore.maintenanceInterval"
+ val MAINTENANCE_INTERVAL_DEFAULT_SECS = 60
+
+ private val loadedProviders = new mutable.HashMap[StateStoreId, StateStoreProvider]()
+ private val managementTimer = new Timer("StateStore Timer", true)
+
+ @volatile private var managementTask: TimerTask = null
+ @volatile private var _coordRef: StateStoreCoordinatorRef = null
+
+ /** Get or create a store associated with the id. */
+ def get(
+ storeId: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ version: Long,
+ hadoopConf: Configuration
+ ): StateStore = {
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917052
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
--- End diff --
`checkpointLocation` if we want to be consistent with the other PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56387482
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala ---
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+
+import scala.collection.mutable
+import scala.util.Random
+
+import org.apache.hadoop.fs.Path
+import org.scalatest.{BeforeAndAfter, PrivateMethodTester}
+import org.scalatest.concurrent.Eventually._
+import org.scalatest.time.SpanSugar._
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+class StateStoreSuite extends SparkFunSuite with BeforeAndAfter with PrivateMethodTester {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ private val tempDir = Utils.createTempDir().toString
+
+ after {
+ StateStore.stop()
+ }
+
+ test("update, remove, commit, and all data iterator") {
+ val provider = newStoreProvider()
+
+ // Verify state before starting a new set of updates
+ assert(provider.latestIterator().isEmpty)
+
+ val store = provider.getStore(0)
+ assert(!store.hasCommitted)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+
+ // Verify state after updating
+ update(store, "a", 1)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+ assert(provider.latestIterator().isEmpty)
+
+ // Make updates, commit and then verify state
+ update(store, "b", 2)
+ update(store, "aa", 3)
+ remove(store, _.startsWith("a"))
+ assert(store.commit() === 1)
+
+ assert(store.hasCommitted)
+ assert(unwrapToSet(store.iterator()) === Set("b" -> 2))
+ assert(unwrapToSet(provider.latestIterator()) === Set("b" -> 2))
+ assert(fileExists(provider, version = 1, isSnapshot = false))
+ assert(getDataFromFiles(provider) === Set("b" -> 2))
+
+ // Trying to get newer versions should fail
+ intercept[Exception] {
+ provider.getStore(2)
+ }
+ intercept[Exception] {
+ getDataFromFiles(provider, 2)
+ }
+
+ // New updates to the reloaded store with new version, and does not change old version
+ val reloadedStore = new HDFSBackedStateStoreProvider(store.id, provider.directory).getStore(1)
+ update(reloadedStore, "c", 4)
+ assert(reloadedStore.commit() === 2)
+ assert(unwrapToSet(reloadedStore.iterator()) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider, version = 1) === Set("b" -> 2))
+ assert(getDataFromFiles(provider, version = 2) === Set("b" -> 2, "c" -> 4))
+ }
+
+ test("updates iterator with all combos of updates and removes") {
+ val provider = newStoreProvider()
+ var currentVersion: Int = 0
+ def withStore(body: StateStore => Unit): Unit = {
+ val store = provider.getStore(currentVersion)
+ body(store)
+ currentVersion += 1
+ }
+
+ // New data should be seen in updates as value added, even if they had multiple updates
+ withStore { store =>
+ update(store, "a", 1)
+ update(store, "aa", 1)
+ update(store, "aa", 2)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Added("a", 1), Added("aa", 2)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1, "aa" -> 2))
+ }
+
+ // Multiple updates to same key should be collapsed in the updates as a single value update
+ // Keys that have not been updated should not appear in the updates
+ withStore { store =>
+ update(store, "a", 4)
+ update(store, "a", 6)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 6)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Keys added, updated and finally removed before commit should not appear in updates
+ withStore { store =>
+ update(store, "b", 4) // Added, finally removed
+ update(store, "bb", 5) // Added, updated, finally removed
+ update(store, "bb", 6)
+ remove(store, _.startsWith("b"))
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set.empty)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Removed data should be seen in updates as a key removed
+ // Removed, but re-added data should be seen in updates as a value update
+ withStore { store =>
+ remove(store, _.startsWith("a"))
+ update(store, "a", 10)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 10), Removed("aa")))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 10))
+ }
+ }
+
+ test("cancel") {
+ val provider = newStoreProvider()
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ store.commit()
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ // cancelUpdates should not change the data in the files
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ store1.cancel()
+ assert(getDataFromFiles(provider) === Set("a" -> 1))
+ }
+
+ test("getStore with unexpected versions") {
+ val provider = newStoreProvider()
+
+ intercept[IllegalArgumentException] {
+ provider.getStore(-1)
+ }
+
+ // Prepare some data in the stoer
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ assert(store.commit() === 1)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ intercept[IllegalStateException] {
+ provider.getStore(2)
+ }
+
+ // Update store version with some data
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ assert(store1.commit() === 2)
+ assert(unwrapToSet(store1.iterator()) === Set("a" -> 1, "b" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "b" -> 1))
+
+ // Overwrite the version with other data
+ val store2 = provider.getStore(1)
+ update(store2, "c", 1)
+ assert(store2.commit() === 2)
+ assert(unwrapToSet(store2.iterator()) === Set("a" -> 1, "c" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "c" -> 1))
+ }
+
+ test("snapshotting") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ var currentVersion = 0
+ def updateVersionTo(targetVersion: Int): Unit = {
+ for (i <- currentVersion + 1 to targetVersion) {
+ val store = provider.getStore(currentVersion)
+ update(store, "a", i)
+ store.commit()
+ currentVersion += 1
+ }
+ require(currentVersion === targetVersion)
+ }
+
+ updateVersionTo(2)
+ require(getDataFromFiles(provider) === Set("a" -> 2))
+ provider.manage() // should not generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 2))
+
+ for (i <- 1 to currentVersion) {
+ assert(fileExists(provider, i, isSnapshot = false)) // all delta files present
+ assert(!fileExists(provider, i, isSnapshot = true)) // no snapshot files present
+ }
+
+ // After version 6, snapshotting should generate one snapshot file
+ updateVersionTo(6)
+ require(getDataFromFiles(provider) === Set("a" -> 6), "store not updated correctly")
+ provider.manage() // should generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 6), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 6))
+
+ val snapshotVersion = (0 to 6).find(version => fileExists(provider, version, isSnapshot = true))
+ assert(snapshotVersion.nonEmpty, "snapshot file not generated")
+
+ // After version 20, snapshotting should generate newer snapshot files
+ updateVersionTo(20)
+ require(getDataFromFiles(provider) === Set("a" -> 20), "store not updated correctly")
+ provider.manage() // do snapshot
+ assert(getDataFromFiles(provider) === Set("a" -> 20), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 20))
+
+ val latestSnapshotVersion = (0 to 20).filter(version =>
+ fileExists(provider, version, isSnapshot = true)).lastOption
+ assert(latestSnapshotVersion.nonEmpty, "no snapshot file found")
+ assert(latestSnapshotVersion.get > snapshotVersion.get, "newer snapshot not generated")
+
+ }
+
+ test("cleaning") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ for (i <- 1 to 20) {
+ val store = provider.getStore(i - 1)
+ update(store, "a", i)
+ store.commit()
+ provider.manage() // do cleanup
+ }
+ require(
+ unwrapToSet(provider.latestIterator()) === Set("a" -> 20),
+ "store not updated correctly")
+
+ assert(!fileExists(provider, version = 1, isSnapshot = false)) // first file should be deleted
+
+ // last couple of versions should be retrievable
+ assert(getDataFromFiles(provider, 20) === Set("a" -> 20))
+ assert(getDataFromFiles(provider, 19) === Set("a" -> 19))
+ }
+
+ test("StateStore.get") {
+ quietly {
+ val dir = Utils.createDirectory(tempDir, Random.nextString(5)).toString
+ val storeId = StateStoreId(0, 0)
+
+ // Verify that trying to get incorrect versions throw errors
+ intercept[IllegalArgumentException] {
+ StateStore.get(storeId, dir, -1)
+ }
+ assert(!StateStore.isLoaded(storeId)) // version -1 should not attempt to load the store
+
+ intercept[IllegalStateException] {
+ StateStore.get(storeId, dir, 1)
+ }
+
+ // Increase version of the store
+ val store0 = StateStore.get(storeId, dir, 0)
+ assert(store0.version === 0)
+ update(store0, "a", 1)
+ store0.commit()
+
+ assert(StateStore.get(storeId, dir, 1).version == 1)
+ assert(StateStore.get(storeId, dir, 0).version == 0)
+
+ // Verify that you can remove the store and still reload and use it
+ StateStore.remove(storeId)
+ assert(!StateStore.isLoaded(storeId))
+
+ val store1 = StateStore.get(storeId, dir, 1)
+ assert(StateStore.isLoaded(storeId))
+ update(store1, "a", 2)
+ assert(store1.commit() === 2)
+ assert(unwrapToSet(store1.iterator()) === Set("a" -> 2))
+ }
+ }
+
+ test("background management") {
+ val conf = new SparkConf()
+ .setMaster("local")
+ .setAppName("test")
+ .set("spark.sql.streaming.stateStore.managementInterval", "10ms")
+ val opId = 0
+ val storeId = StateStoreId(opId, 0)
+ val dir = Utils.createDirectory(tempDir, Random.nextString(5)).toString
--- End diff --
I can use createTempDir here because i dont need to use it across SparkContexts. But the root tempDir is going to be deleted in afterAll() anyways.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56375262
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{Partition, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+/**
+ * An RDD that allows computations to be executed against [[StateStore]]s. It
+ * uses the [[StateStoreCoordinator]] to use the locations of loaded state stores as
+ * preferred locations.
+ */
+class StateStoreRDD[INPUT: ClassTag, OUTPUT: ClassTag](
--- End diff --
Elsewhere we use `T` and `U` for the type variables.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57094458
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDDSuite.scala ---
@@ -142,6 +142,38 @@ class StateStoreRDDSuite extends SparkFunSuite with BeforeAndAfter with BeforeAn
}
}
+ test("distributed test") {
+ quietly {
+ withSpark(new SparkContext(sparkConf.setMaster("local-cluster[2, 1, 1024]"))) { sc =>
--- End diff --
nit: should clone `sparkConf`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56373579
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.{Logging, SparkEnv}
+import org.apache.spark.sql.catalyst.InternalRow
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(operatorId: Long, partitionId: Int)
--- End diff --
This is unique within a given query? I.e. we have one state store per query and this identifies one set of k/v pairs within that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200017604
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197138848
**[Test build #53268 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53268/consoleFull)** for PR 11645 at commit [`8123818`](https://github.com/apache/spark/commit/81238189f560f5df846b36e5cb5e50532f6f82a3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196607441
**[Test build #53146 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53146/consoleFull)** for PR 11645 at commit [`22d7e66`](https://github.com/apache/spark/commit/22d7e6639125a507467766c45393e36b3ee92f3f).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200129722
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199979580
**[Test build #53794 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53794/consoleFull)** for PR 11645 at commit [`4752d73`](https://github.com/apache/spark/commit/4752d73af2b7257e0160d1eac81ee9b9279d3109).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199551897
**[Test build #53731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53731/consoleFull)** for PR 11645 at commit [`19a60a6`](https://github.com/apache/spark/commit/19a60a65b77b73cf69cde961c034d04ad17df436).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56919467
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(checkpointLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[(UnsafeRow, UnsafeRow)]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+/** Trait representing a provider of a specific version of a [[StateStore]]. */
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background maintenance */
+ def doMaintenance(): Unit = { }
+}
+
+/** Trait representing updates made to a [[StateStore]]. */
+sealed trait StoreUpdate
+
+case class ValueAdded(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+
+case class ValueUpdated(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+
+case class KeyRemoved(key: UnsafeRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrieve stores
+ * by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null),
+ * it also runs a periodic background tasks to do maintenance on the loaded stores. For each
+ * store, tt uses the [[StateStoreCoordinator]] to ensure whether the current loaded instance of
+ * the store is the active instance. Accordingly, it either keeps it loaded and performs
+ * maintenance, or unloads the store.
+ */
+private[state] object StateStore extends Logging {
+
+ val MAINTENANCE_INTERVAL_CONFIG = "spark.sql.streaming.stateStore.maintenanceInterval"
+ val MAINTENANCE_INTERVAL_DEFAULT_SECS = 60
+
+ private val loadedProviders = new mutable.HashMap[StateStoreId, StateStoreProvider]()
+ private val maintenanceTimer = new Timer("StateStore Timer", true)
+
+ @volatile private var maintenanceTask: TimerTask = null
+ @volatile private var _coordRef: StateStoreCoordinatorRef = null
+
+ /** Get or create a store associated with the id. */
+ def get(
+ storeId: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ version: Long,
+ hadoopConf: Configuration): StateStore = {
+ require(version >= 0)
+ val storeProvider = loadedProviders.synchronized {
+ startMaintenanceIfNeeded()
+ val sparkConf = Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
--- End diff --
This will run on the executor side, where isnt the conf retrieved from the SparkEnv anyways. Correct me if i am wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r55794534
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.reflect.ClassTag
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf, SparkEnv}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, RpcUtils, Utils}
+
+case class StateStoreId(operatorId: Long, partitionId: Int)
+
+private[state] object StateStore extends Logging {
+
+ sealed trait Update
+ case class ValueUpdated(key: InternalRow, value: InternalRow) extends Update
+ case class KeyRemoved(key: InternalRow) extends Update
+
+ private val loadedStores = new mutable.HashMap[StateStoreId, StateStore]()
+ private val managementTimer = new Timer("StateStore Timer", true)
+ @volatile private var managementTask: TimerTask = null
+
+ def get(storeId: StateStoreId, directory: String): StateStore = {
+ val store = loadedStores.synchronized {
+ startIfNeeded()
+ loadedStores.getOrElseUpdate(storeId, new StateStore(storeId, directory))
+ }
+ reportActiveInstance(storeId)
+ store
+ }
+
+ def clearAll(): Unit = loadedStores.synchronized {
+ loadedStores.clear()
+ if (managementTask != null) {
+ managementTask.cancel()
+ managementTask = null
+ }
+ }
+
+ private def remove(storeId: StateStoreId): Unit = {
+ loadedStores.remove(storeId)
+ }
+
+ private def reportActiveInstance(storeId: StateStoreId): Unit = {
+ val host = SparkEnv.get.blockManager.blockManagerId.host
+ val executorId = SparkEnv.get.blockManager.blockManagerId.executorId
+ askCoordinator[Boolean](ReportActiveInstance(storeId, host, executorId))
+ }
+
+ private def verifyIfInstanceActive(storeId: StateStoreId): Boolean = {
+ val executorId = SparkEnv.get.blockManager.blockManagerId.executorId
+ askCoordinator[Boolean](VerifyIfInstanceActive(storeId, executorId)).getOrElse(false)
+ }
+
+ private def askCoordinator[T: ClassTag](message: StateStoreCoordinatorMessage): Option[T] = {
+ try {
+ val env = SparkEnv.get
+ if (env != null) {
+ val coordinatorRef = RpcUtils.makeDriverRef("StateStoreCoordinator", env.conf, env.rpcEnv)
+ Some(coordinatorRef.askWithRetry[T](message))
+ } else {
+ None
+ }
+ } catch {
+ case NonFatal(e) =>
+ clearAll()
+ None
+ }
+ }
+
+ private def startIfNeeded(): Unit = loadedStores.synchronized {
+ if (managementTask == null) {
+ managementTask = new TimerTask {
+ override def run(): Unit = { manageFiles() }
+ }
+ managementTimer.schedule(managementTask, 10000, 10000)
+ }
+ }
+
+ private def manageFiles(): Unit = {
+ loadedStores.synchronized { loadedStores.values.toSeq }.foreach { store =>
+ try {
+ store.manageFiles()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up store ${store.id}")
+ }
+ }
+ }
+}
+
+private[sql] class StateStore(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir = new Path(directory, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(new Configuration())
+ private val serializer = new KryoSerializer(new SparkConf)
+
+ @volatile private var uncommittedDelta: UncommittedUpdates = null
+
+ initialize()
+
+ /**
+ * Prepare for updates to create a new `version` of the map. The store ensure that updates
+ * are made on the `version - 1` of the store data. If `version` already exists, it will
+ * be overwritten when the updates are committed.
+ */
+ private[state] def prepareForUpdates(version: Long): Unit = synchronized {
+ require(version >= 0)
+ if (uncommittedDelta != null) {
+ cancelUpdates()
+ }
+ val newMap = new MapType()
+ if (version > 0) {
+ val oldMap = loadMap(version - 1)
+ newMap ++= oldMap
+ }
+ uncommittedDelta = new UncommittedUpdates(version, newMap)
+ }
+
+ /** Update the value of a key using the `updateFunc` */
+ def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit = {
+ verify(uncommittedDelta != null, "Cannot update data before calling newVersion()")
+ uncommittedDelta.update(key, updateFunc)
+ }
+
+ /** Remove keys that satisfy the following condition */
+ def remove(condition: InternalRow => Boolean): Unit = {
+ verify(uncommittedDelta != null, "Cannot remove data before calling newVersion()")
+ uncommittedDelta.remove(condition)
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ def commitUpdates(): Unit = {
+ verify(uncommittedDelta != null, "Cannot commit data before calling newVersion()")
+ uncommittedDelta.commitAndWriteDeltaFile()
+ uncommittedDelta = null
+ }
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancelUpdates(): Unit = {
+ if (uncommittedDelta != null) {
+ uncommittedDelta.cancel()
+ uncommittedDelta = null
+ }
+ }
+
+ /**
+ * Get all the data of the latest version of the store.
+ * Note that this will look up the files to determined the latest known version.
+ */
+
+ def getAll(): Iterator[InternalRow] = synchronized {
+ verify(uncommittedDelta == null, "Cannot getAll() while there are uncommitted updates")
+ val versionsInFiles = fetchFiles().map(_.version).toSet
+ val versionsLoaded = loadedMaps.keySet
+ val allKnownVersions = versionsInFiles ++ versionsLoaded
+ if (allKnownVersions.nonEmpty) {
+ loadMap(allKnownVersions.max)
+ .iterator
+ .map { case (key, value) => new JoinedRow(key, value) }
+ } else Iterator.empty
+ }
+
+ private[state] def hasUncommittedUpdates: Boolean = {
+ uncommittedDelta != null
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ // Internal classes and methods
+
+ private case class StoreFile(version: Long, path: Path, isSnapshot: Boolean)
+
+ private class UncommittedUpdates(val version: Long, val map: MapType) {
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream =
+ serializer.newInstance().serializeStream(fs.create(tempDeltaFile, true))
+
--- End diff --
Missing `startUpdates()`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917001
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[(UnsafeRow, UnsafeRow)]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background management */
+ def doMaintenance(): Unit = { }
+}
+
+sealed trait StoreUpdate
+case class ValueAdded(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class ValueUpdated(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class KeyRemoved(key: UnsafeRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrieve stores
+ * by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null),
+ * it also runs a periodic background tasks to do maintenance on the loaded stores. For each
+ * store, tt uses the [[StateStoreCoordinator]] to ensure whether the current loaded instance of
+ * the store is the active instance. Accordingly, it either keeps it loaded and performance
--- End diff --
nit: "performs"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197068265
**[Test build #53241 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53241/consoleFull)** for PR 11645 at commit [`f5660d2`](https://github.com/apache/spark/commit/f5660d23ce80f84e4d28235805d1c64060aad382).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56386741
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.{Logging, SparkEnv}
+import org.apache.spark.sql.catalyst.InternalRow
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: InternalRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[InternalRow]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background management */
+ def manage(): Unit = { }
+}
+
+sealed trait StoreUpdate
+case class ValueAdded(key: InternalRow, value: InternalRow) extends StoreUpdate
+case class ValueUpdated(key: InternalRow, value: InternalRow) extends StoreUpdate
+case class KeyRemoved(key: InternalRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrive stores
+ * by their unique ids.
+ */
+private[state] object StateStore extends Logging {
+
+ private val MANAGEMENT_TASK_INTERVAL_SECS = 60
+
+ private val loadedProviders = new mutable.HashMap[StateStoreId, StateStoreProvider]()
+ private val managementTimer = new Timer("StateStore Timer", true)
+ @volatile private var managementTask: TimerTask = null
+
+ /** Get or create a store associated with the id. */
+ def get(storeId: StateStoreId, directory: String, version: Long): StateStore = {
+ require(version >= 0)
+ val storeProvider = loadedProviders.synchronized {
+ startIfNeeded()
+ val provider = loadedProviders.getOrElseUpdate(
+ storeId, new HDFSBackedStateStoreProvider(storeId, directory))
+ reportActiveInstance(storeId)
+ provider
+ }
+ storeProvider.getStore(version)
+ }
+
+ def remove(storeId: StateStoreId): Unit = loadedProviders.synchronized {
+ loadedProviders.remove(storeId)
+ }
+
+ def isLoaded(storeId: StateStoreId): Boolean = loadedProviders.synchronized {
+ loadedProviders.contains(storeId)
+ }
+
+ /** Unload and stop all state store provider */
+ def stop(): Unit = loadedProviders.synchronized {
+ loadedProviders.clear()
+ if (managementTask != null) {
+ managementTask.cancel()
+ managementTask = null
+ logInfo("StateStore stopped")
+ }
+ }
+
+ private def startIfNeeded(): Unit = loadedProviders.synchronized {
+ if (managementTask == null) {
+ managementTask = new TimerTask {
+ override def run(): Unit = { manageAll() }
+ }
+ val periodMs = Option(SparkEnv.get).map(_.conf) match {
--- End diff --
SparkEnv.get returns null when a SparkContext is not active. I am afraid that this StateStore object's timer will keep ticking even if there is not SparkContext active (in local mode), and this is to make sure that this background this does not fail unexpectedly.
Actually on second thought, the timer does not need to be started if SparkEnv.get returns nulls, it only needs to be lazily stopped when SparkEnv.get becomes null (which it already does). I will modify this logic.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199537769
**[Test build #53718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53718/consoleFull)** for PR 11645 at commit [`3824053`](https://github.com/apache/spark/commit/3824053324aae89956601db060bfb77b013a2303).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56390002
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{Partition, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+/**
+ * An RDD that allows computations to be executed against [[StateStore]]s. It
+ * uses the [[StateStoreCoordinator]] to use the locations of loaded state stores as
+ * preferred locations.
+ */
+class StateStoreRDD[INPUT: ClassTag, OUTPUT: ClassTag](
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57045404
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(checkpointLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[(UnsafeRow, UnsafeRow)]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+/** Trait representing a provider of a specific version of a [[StateStore]]. */
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background maintenance */
+ def doMaintenance(): Unit = { }
+}
+
+/** Trait representing updates made to a [[StateStore]]. */
+sealed trait StoreUpdate
+
+case class ValueAdded(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+
+case class ValueUpdated(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+
+case class KeyRemoved(key: UnsafeRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrieve stores
+ * by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null),
+ * it also runs a periodic background tasks to do maintenance on the loaded stores. For each
+ * store, tt uses the [[StateStoreCoordinator]] to ensure whether the current loaded instance of
+ * the store is the active instance. Accordingly, it either keeps it loaded and performs
+ * maintenance, or unloads the store.
+ */
+private[state] object StateStore extends Logging {
+
+ val MAINTENANCE_INTERVAL_CONFIG = "spark.streaming.stateStore.maintenanceInterval"
+ val MAINTENANCE_INTERVAL_DEFAULT_SECS = 60
+
+ private val loadedProviders = new mutable.HashMap[StateStoreId, StateStoreProvider]()
+ private val maintenanceTimer = new Timer("StateStore Timer", true)
--- End diff --
[It's better to use ScheduledExecutor instead of Timer](http://stackoverflow.com/questions/409932/java-timer-vs-executorservice/409993#409993). You can use `ThreadUtils.newDaemonSingleThreadScheduledExecutor` for ScheduledExecutor. And don't forget to shutdown it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199983888
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199964991
**[Test build #53805 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53805/consoleFull)** for PR 11645 at commit [`63fad92`](https://github.com/apache/spark/commit/63fad922f166c7fdaf21783f89d01e3dec13c0f6).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56386061
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.{Logging, SparkEnv}
+import org.apache.spark.sql.catalyst.InternalRow
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(operatorId: Long, partitionId: Int)
--- End diff --
This is unique to each partition in each incremental aggregation operator.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200051228
**[Test build #53791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53791/consoleFull)** for PR 11645 at commit [`2f29d9a`](https://github.com/apache/spark/commit/2f29d9aaabcc39fe82e7d11bff76c72edff565b9).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56898970
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDDSuite.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+import java.nio.file.Files
+
+import scala.util.Random
+
+import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.rdd.RDD
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.util.Utils
+
+class StateStoreRDDSuite extends SparkFunSuite with BeforeAndAfter with BeforeAndAfterAll {
+
+ private val conf = new SparkConf().setMaster("local").setAppName(this.getClass.getCanonicalName)
+ private var tempDir = Files.createTempDirectory("StateStoreRDDSuite").toString
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ after {
+ StateStore.stop()
+ }
+
+ override def afterAll(): Unit = {
+ super.afterAll()
+ Utils.deleteRecursively(new File(tempDir))
+ }
+
+ test("versioning and immutability") {
+ quietly {
+ withSpark(new SparkContext(conf)) { sc =>
+ val path = Utils.createDirectory(tempDir, Random.nextString(10)).toString
--- End diff --
The root `tempDir` is cleaned up after all tests. See comments below to understand the reason behind this design.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196607452
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53146/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200016705
**[Test build #53804 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53804/consoleFull)** for PR 11645 at commit [`756762a`](https://github.com/apache/spark/commit/756762a044ce9a2735950ed27384151502ce020d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196215657
**[Test build #53064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53064/consoleFull)** for PR 11645 at commit [`7adca70`](https://github.com/apache/spark/commit/7adca7004085a463fd1768708f01a7a01775cea8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195173686
@marmbrus @rxin @zsxwing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195180583
**[Test build #52898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52898/consoleFull)** for PR 11645 at commit [`d8cee54`](https://github.com/apache/spark/commit/d8cee54ad183cf899031fb762bba032c6a9cffd5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196219616
**[Test build #53064 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53064/consoleFull)** for PR 11645 at commit [`7adca70`](https://github.com/apache/spark/commit/7adca7004085a463fd1768708f01a7a01775cea8).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199640243
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199640246
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53743/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199578052
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200122344
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57082773
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ storeConf: StateStoreConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ private val allUpdates = new java.util.HashMap[UnsafeRow, StoreUpdate]()
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ override def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = Option(mapToUpdate.get(key))
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ writeToDeltaFile(tempDeltaFileStream, ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: UnsafeRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keySet().iterator()
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ keyIter.remove()
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ writeToDeltaFile(tempDeltaFileStream, KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ finalizeDeltaFile(tempDeltaFileStream)
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ logInfo(s"Committed version $newVersion for $this")
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ logInfo("Canceled ")
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[(UnsafeRow, UnsafeRow)] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.values().asScala.toIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ val time = System.nanoTime()
+ newMap.putAll(loadMap(version))
+ }
+ val store = new HDFSBackedStateStore(version, newMap)
+ logInfo(s"Retrieved version $version of $this for update")
+ store
+ }
+
+ /** Do maintenance backing data files, including creating snapshots and cleaning up old files */
+ override def doMaintenance(): Unit = {
+ try {
+ doSnapshot()
+ cleanup()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up $this")
+ }
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ /* Internal classes and methods */
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir =
+ new Path(id.checkpointLocation, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(hadoopConf)
+ private val sparkConf = Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
+
+ initialize()
+
+ private case class StoreFile(version: Long, path: Path, isSnapshot: Boolean)
+
+ /** Commit a set of updates to the store with the given new version */
+ private def commitUpdates(newVersion: Long, map: MapType, tempDeltaFile: Path): Path = {
+ synchronized {
+ val finalDeltaFile = deltaFile(newVersion)
+ fs.rename(tempDeltaFile, finalDeltaFile)
+ loadedMaps.put(newVersion, map)
+ finalDeltaFile
+ }
+ }
+
+ /**
+ * Get iterator of all the data of the latest version of the store.
+ * Note that this will look up the files to determined the latest known version.
+ */
+ private[state] def latestIterator(): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ val versionsInFiles = fetchFiles().map(_.version).toSet
+ val versionsLoaded = loadedMaps.keySet
+ val allKnownVersions = versionsInFiles ++ versionsLoaded
+ if (allKnownVersions.nonEmpty) {
+ loadMap(allKnownVersions.max).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ } else Iterator.empty
+ }
+
+ /** Get iterator of a specific version of the store */
+ private[state] def iterator(version: Long): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ loadMap(version).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ }
+
+ /** Initialize the store provider */
+ private def initialize(): Unit = {
+ if (!fs.exists(baseDir)) {
+ fs.mkdirs(baseDir)
+ } else {
+ if (!fs.isDirectory(baseDir)) {
+ throw new IllegalStateException(
+ s"Cannot use ${id.checkpointLocation} for storing state data for $this as" +
+ s"$baseDir already exists and is not a directory")
+ }
+ }
+ }
+
+ /** Load the required version of the map data from the backing files */
+ private def loadMap(version: Long): MapType = {
+ if (version <= 0) return new MapType
+ synchronized { loadedMaps.get(version) }.getOrElse {
+ val mapFromFile = readSnapshotFile(version).getOrElse {
+ val prevMap = loadMap(version - 1)
+ val newMap = new MapType(prevMap)
+ newMap.putAll(prevMap)
+ updateFromDeltaFile(version, newMap)
+ newMap
+ }
+ loadedMaps.put(version, mapFromFile)
+ mapFromFile
+ }
+ }
+
+ private def writeToDeltaFile(output: DataOutputStream, update: StoreUpdate): Unit = {
+
+ def writeUpdate(key: UnsafeRow, value: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ val valueBytes = value.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+
+ def writeRemove(key: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(-1)
+ }
+
+ update match {
+ case ValueAdded(key, value) =>
+ writeUpdate(key, value)
+ case ValueUpdated(key, value) =>
+ writeUpdate(key, value)
+ case KeyRemoved(key) =>
+ writeRemove(key)
+ }
+ }
+
+ private def finalizeDeltaFile(output: DataOutputStream): Unit = {
+ output.writeInt(-1) // Write this magic number to signify end of file
+ output.close()
+ }
+
+ private def updateFromDeltaFile(version: Long, map: MapType): Unit = {
+ val fileToRead = deltaFile(version)
+ if (!fs.exists(fileToRead)) {
+ throw new IllegalStateException(
+ s"Error reading delta file $fileToRead of $this: $fileToRead does not exist")
+ }
+ var input: DataInputStream = null
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while(!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading delta file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ map.remove(keyRow)
+ } else {
+ val valueRowBuffer = new Array[Byte](valueSize)
+ ByteStreams.readFully(input, valueRowBuffer, 0, valueSize)
+ val valueRow = new UnsafeRow(valueSchema.fields.length)
+ valueRow.pointTo(valueRowBuffer, valueSize)
+ map.put(keyRow, valueRow)
+ }
+ }
+ }
+ } finally {
+ if (input != null) input.close()
+ }
+ logInfo(s"Read delta file for version $version of $this from $fileToRead")
+ }
+
+ private def writeSnapshotFile(version: Long, map: MapType): Unit = {
+ val fileToWrite = snapshotFile(version)
+ var output: DataOutputStream = null
+ Utils.tryWithSafeFinally {
+ output = compressStream(fs.create(fileToWrite, false))
+ val iter = map.entrySet().iterator()
+ while(iter.hasNext) {
+ val entry = iter.next()
+ val keyBytes = entry.getKey.getBytes()
+ val valueBytes = entry.getValue.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+ output.writeInt(-1)
+ } {
+ if (output != null) output.close()
+ }
+ logInfo(s"Written snapshot file for version $version of $this at $fileToWrite")
+ }
+
+ private def readSnapshotFile(version: Long): Option[MapType] = {
+ val fileToRead = snapshotFile(version)
+ if (!fs.exists(fileToRead)) return None
+
+ val map = new MapType()
+ var input: DataInputStream = null
+
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while (!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading snapshot file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ throw new Exception(
+ s"Error reading snapshot file $fileToRead of $this: value size cannot be $valueSize")
+ } else {
+ val valueRowBuffer = new Array[Byte](valueSize)
+ ByteStreams.readFully(input, valueRowBuffer, 0, valueSize)
+ val valueRow = new UnsafeRow(valueSchema.fields.length)
+ valueRow.pointTo(valueRowBuffer, valueSize)
+ map.put(keyRow, valueRow)
+ }
+ }
+ }
+ logInfo(s"Read snapshot file for version $version of $this from $fileToRead")
+ Some(map)
+ } finally {
+ if (input != null) input.close()
+ }
+ }
+
+
+ /** Perform a snapshot of the store to allow delta files to be consolidated */
+ private def doSnapshot(): Unit = {
+ try {
+ val files = fetchFiles()
+ if (files.nonEmpty) {
+ val lastVersion = files.last.version
+ val deltaFilesForLastVersion =
+ filesForVersion(files, lastVersion).filter(_.isSnapshot == false)
+ synchronized { loadedMaps.get(lastVersion) } match {
+ case Some(map) =>
+ if (deltaFilesForLastVersion.size > storeConf.maxDeltasForSnapshot) {
+ writeSnapshotFile(lastVersion, map)
+ }
+ case None =>
+ // The last map is not loaded, probably some other instance is incharge
+ }
+
+ }
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error doing snapshots for $this", e)
+ }
+ }
+
+ /**
+ * Clean up old snapshots and delta files that are not needed any more. It ensures that last
+ * few versions of the store can be recovered from the files, so re-executed RDD operations
+ * can re-apply updates on the past versions of the store.
+ */
+ private[state] def cleanup(): Unit = {
+ try {
+ val files = fetchFiles()
+ if (files.nonEmpty) {
+ val earliestVersionToRetain = files.last.version - storeConf.minVersionsToRetain
+ if (earliestVersionToRetain > 0) {
+ val earliestFileToRetain = filesForVersion(files, earliestVersionToRetain).head
+ synchronized {
+ val mapsToRemove = loadedMaps.keys.filter(_ < earliestVersionToRetain).toSeq
+ mapsToRemove.foreach(loadedMaps.remove)
+ }
+ files.filter(_.version < earliestFileToRetain.version).foreach { f =>
+ fs.delete(f.path, true)
+ }
+ logInfo(s"Deleted files older than ${earliestFileToRetain.version} for $this")
+ }
+ }
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error cleaning up files for $this", e)
+ }
+ }
+
+ /** Files needed to recover the given version of the store */
+ private def filesForVersion(allFiles: Seq[StoreFile], version: Long): Seq[StoreFile] = {
+ require(version >= 0)
+ require(allFiles.exists(_.version == version))
+
+ val latestSnapshotFileBeforeVersion = allFiles
+ .filter(_.isSnapshot == true)
+ .takeWhile(_.version <= version)
+ .lastOption
+ val deltaBatchFiles = latestSnapshotFileBeforeVersion match {
+ case Some(snapshotFile) =>
+ val deltaBatchIds = (snapshotFile.version + 1) to version
+
+ val deltaFiles = allFiles.filter { file =>
+ file.version > snapshotFile.version && file.version <= version
+ }
+ verify(
+ deltaFiles.size == version - snapshotFile.version,
+ s"Unexpected list of delta files for version $version for $this: $deltaFiles"
+ )
+ deltaFiles
+
+ case None =>
+ allFiles.takeWhile(_.version <= version)
+ }
+ latestSnapshotFileBeforeVersion.toSeq ++ deltaBatchFiles
+ }
+
+ /** Fetch all the files that back the store */
+ private def fetchFiles(): Seq[StoreFile] = {
+ val files: Seq[FileStatus] = try {
+ fs.listStatus(baseDir)
+ } catch {
+ case _: java.io.FileNotFoundException =>
+ Seq.empty
+ }
+ val versionToFiles = new mutable.HashMap[Long, StoreFile]
+ files.foreach { status =>
+ val path = status.getPath
+ val nameParts = path.getName.split("\\.")
+ if (nameParts.size == 2) {
+ val version = nameParts(0).toLong
+ nameParts(1).toLowerCase match {
+ case "delta" =>
+ // ignore the file otherwise, snapshot file already exists for that batch id
+ if (!versionToFiles.contains(version)) {
+ versionToFiles.put(version, StoreFile(version, path, isSnapshot = false))
+ }
+ case "snapshot" =>
+ versionToFiles.put(version, StoreFile(version, path, isSnapshot = true))
+ case _ =>
+ logWarning(s"Could not identify file $path for $this")
+ }
+ }
+ }
+ val storeFiles = versionToFiles.values.toSeq.sortBy(_.version)
+ logDebug(s"Current set of files for $this: $storeFiles")
+ storeFiles
+ }
+
+ private def compressStream(outputStream: DataOutputStream): DataOutputStream = {
+ val compressed = new LZ4CompressionCodec(sparkConf).compressedOutputStream(outputStream)
+ new DataOutputStream(compressed)
+ }
+
+ private def decompressStream(inputStream: DataInputStream): DataInputStream = {
+ val compressed = new LZ4CompressionCodec(sparkConf).compressedInputStream(inputStream)
+ new DataInputStream(compressed)
+ }
+
+ private def deltaFile(version: Long): Path = {
+ new Path(baseDir, s"$version.delta")
+ }
+
+ private def snapshotFile(version: Long): Path = {
+ new Path(baseDir, s"$version.snapshot")
+ }
+
+ private def verify(condition: => Boolean, msg: String): Unit = {
+ if (!condition) {
+ throw new IllegalStateException(msg)
+ }
+ }
+}
+
+private[state] object HDFSBackedStateStoreProvider {
--- End diff --
nit: Remove this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r55794434
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.reflect.ClassTag
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf, SparkEnv}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, RpcUtils, Utils}
+
+case class StateStoreId(operatorId: Long, partitionId: Int)
+
+private[state] object StateStore extends Logging {
+
+ sealed trait Update
+ case class ValueUpdated(key: InternalRow, value: InternalRow) extends Update
+ case class KeyRemoved(key: InternalRow) extends Update
+
+ private val loadedStores = new mutable.HashMap[StateStoreId, StateStore]()
+ private val managementTimer = new Timer("StateStore Timer", true)
+ @volatile private var managementTask: TimerTask = null
+
+ def get(storeId: StateStoreId, directory: String): StateStore = {
+ val store = loadedStores.synchronized {
+ startIfNeeded()
+ loadedStores.getOrElseUpdate(storeId, new StateStore(storeId, directory))
+ }
+ reportActiveInstance(storeId)
+ store
+ }
+
+ def clearAll(): Unit = loadedStores.synchronized {
+ loadedStores.clear()
+ if (managementTask != null) {
+ managementTask.cancel()
+ managementTask = null
+ }
+ }
+
+ private def remove(storeId: StateStoreId): Unit = {
--- End diff --
Maybe we also need `synchronized` for remove()?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57066269
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEndpointRef, RpcEnv}
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.util.RpcUtils
+
+/** Trait representing all messages to [[StateStoreCoordinator]] */
+private sealed trait StateStoreCoordinatorMessage extends Serializable
+
+/** Classes representing messages */
+private case class ReportActiveInstance(storeId: StateStoreId, host: String, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class VerifyIfInstanceActive(storeId: StateStoreId, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class GetLocation(storeId: StateStoreId)
+ extends StateStoreCoordinatorMessage
+
+private case class DeactivateInstances(storeRootLocation: String)
+ extends StateStoreCoordinatorMessage
+
+private object StopCoordinator
+ extends StateStoreCoordinatorMessage
+
+/** Helper object used to create reference to [[StateStoreCoordinator]]. */
+private[sql] object StateStoreCoordinatorRef extends Logging {
+
+ private val endpointName = "StateStoreCoordinator"
+
+ /**
+ * Create a reference to a [[StateStoreCoordinator]], This can be called from driver as well as
+ * executors.
--- End diff --
nit: this can not be called from executors as creating `StateStoreCoordinator` will succeed even if there is one `StateStoreCoordinator` in driver.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197121312
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198627156
**[Test build #53596 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53596/consoleFull)** for PR 11645 at commit [`e6b1fb3`](https://github.com/apache/spark/commit/e6b1fb3f5ec4cdd711f0f8048933e745d8bf3d81).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200513821
I am merging this. Thanks @zsxwing and @marmbrus for the comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196607451
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199936199
**[Test build #53791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53791/consoleFull)** for PR 11645 at commit [`2f29d9a`](https://github.com/apache/spark/commit/2f29d9aaabcc39fe82e7d11bff76c72edff565b9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199443033
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53690/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196454272
**[Test build #53085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53085/consoleFull)** for PR 11645 at commit [`48afbe6`](https://github.com/apache/spark/commit/48afbe6b64ba07e28cc1e56d9aa3d850e03906e5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917605
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
--- End diff --
It will clean up any temporary files / uncommitted state. General garbage collection for avoiding resource leaks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56387539
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,465 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
--- End diff --
Will add.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56212422
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{Partition, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+/**
+ * Created by tdas on 3/9/16.
+ */
+class StateStoreRDD[INPUT: ClassTag, OUTPUT: ClassTag](
+ dataRDD: RDD[INPUT],
+ storeUpdateFunction: (StateStore, Iterator[INPUT]) => Iterator[OUTPUT],
+ operatorId: Long,
+ storeVersion: Long,
+ storeDirectory: String,
+ storeCoordinator: StateStoreCoordinator) extends RDD[OUTPUT](dataRDD) {
+
+ val nextVersion = storeVersion + 1
+
+ override protected def getPartitions: Array[Partition] = dataRDD.partitions
+ override def getPreferredLocations(partition: Partition): Seq[String] = {
+ Seq.empty
--- End diff --
Yes. Its still WIP. I need to add enable StateStoreCoordinator for this, which is what I am working on now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195645857
**[Test build #52988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52988/consoleFull)** for PR 11645 at commit [`7d74c67`](https://github.com/apache/spark/commit/7d74c67f999d53e9f02cf9ed5d2c8d2e951144ca).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56373034
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,465 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
--- End diff --
Maybe even say that if they are different, which one wins is undefined? (It is at least atomic right?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197128814
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53258/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56376088
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDDSuite.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+import java.nio.file.Files
+
+import scala.util.Random
+
+import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.rdd.RDD
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.util.Utils
+
+class StateStoreRDDSuite extends SparkFunSuite with BeforeAndAfter with BeforeAndAfterAll {
+
+ private val conf = new SparkConf().setMaster("local").setAppName(this.getClass.getCanonicalName)
+ private var tempDir = Files.createTempDirectory("StateStoreRDDSuite").toString
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ after {
+ StateStore.stop()
+ }
+
+ override def afterAll(): Unit = {
+ super.afterAll()
+ Utils.deleteRecursively(new File(tempDir))
+ }
+
+ test("versioning and immutability") {
+ quietly {
+ withSpark(new SparkContext(conf)) { sc =>
+ val path = Utils.createDirectory(tempDir, Random.nextString(10)).toString
+ val increment = (store: StateStore, iter: Iterator[String]) => {
+ iter.foreach { s =>
+ store.update(
+ wrapKey(s), oldRow => {
+ val oldValue = oldRow.map(unwrapValue).getOrElse(0)
+ wrapValue(oldValue + 1)
+ })
+ }
+ store.commit()
+ store.iterator().map(unwrapKeyValue)
+ }
+ val opId = 0
+ val rdd1 = makeRDD(sc, Seq("a", "b", "a"))
+ .withStateStores(increment, opId, storeVersion = 0, path)
+ assert(rdd1.collect().toSet === Set("a" -> 2, "b" -> 1))
+
+ // Generate next version of stores
+ val rdd2 = makeRDD(sc, Seq("a", "c"))
+ .withStateStores(increment, opId, storeVersion = 1, path)
+ assert(rdd2.collect().toSet === Set("a" -> 3, "b" -> 1, "c" -> 1))
+
+ // Make sure the previous RDD still has the same data.
+ assert(rdd1.collect().toSet === Set("a" -> 2, "b" -> 1))
+ }
+ }
+ }
+
+ test("recovering from files") {
+ quietly {
+ val opId = 0
+ val path = Utils.createDirectory(tempDir, Random.nextString(10)).toString
+
+ def makeStoreRDD(
+ sc: SparkContext,
+ seq: Seq[String],
+ storeVersion: Int): RDD[(String, Int)] = {
+ makeRDD(sc, Seq("a")).withStateStores(increment, opId, storeVersion, path)
+ }
+
+ // Generate RDDs and state store data
+ withSpark(new SparkContext(conf)) { sc =>
+ for (i <- 1 to 20) {
+ require(makeStoreRDD(sc, Seq("a"), i - 1).collect().toSet === Set("a" -> i))
+ }
+ }
+
+ // With a new context, try using the earlier state store data
+ withSpark(new SparkContext(conf)) { sc =>
+ assert(makeStoreRDD(sc, Seq("a"), 20).collect().toSet === Set("a" -> 21))
+ }
+ }
+ }
+
+ test("preferred locations using StateStoreCoordinator") {
+ val opId = 0
+ val path = Utils.createDirectory(tempDir, Random.nextString(10)).toString
--- End diff --
createTempDir
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57073603
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ storeConf: StateStoreConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ private val allUpdates = new java.util.HashMap[UnsafeRow, StoreUpdate]()
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ override def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = Option(mapToUpdate.get(key))
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ writeToDeltaFile(tempDeltaFileStream, ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: UnsafeRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keySet().iterator()
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ keyIter.remove()
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ writeToDeltaFile(tempDeltaFileStream, KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ finalizeDeltaFile(tempDeltaFileStream)
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ logInfo(s"Committed version $newVersion for $this")
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ logInfo("Canceled ")
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[(UnsafeRow, UnsafeRow)] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.values().asScala.toIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ val time = System.nanoTime()
--- End diff --
nit: time is not used
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56918468
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,597 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.serializer.KryoSerializer
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ sparkConf: SparkConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ import HDFSBackedStateStoreProvider._
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ // private val tempDeltaFileStream = fs.create(tempDeltaFile, true)
--- End diff --
dang! removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199538076
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199547433
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57066767
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEndpointRef, RpcEnv}
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.util.RpcUtils
+
+/** Trait representing all messages to [[StateStoreCoordinator]] */
+private sealed trait StateStoreCoordinatorMessage extends Serializable
+
+/** Classes representing messages */
+private case class ReportActiveInstance(storeId: StateStoreId, host: String, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class VerifyIfInstanceActive(storeId: StateStoreId, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class GetLocation(storeId: StateStoreId)
+ extends StateStoreCoordinatorMessage
+
+private case class DeactivateInstances(storeRootLocation: String)
+ extends StateStoreCoordinatorMessage
+
+private object StopCoordinator
+ extends StateStoreCoordinatorMessage
+
+/** Helper object used to create reference to [[StateStoreCoordinator]]. */
+private[sql] object StateStoreCoordinatorRef extends Logging {
+
+ private val endpointName = "StateStoreCoordinator"
+
+ /**
+ * Create a reference to a [[StateStoreCoordinator]], This can be called from driver as well as
+ * executors.
+ */
+ def apply(env: SparkEnv): StateStoreCoordinatorRef = synchronized {
+ try {
+ val coordinator = new StateStoreCoordinator(env.rpcEnv)
+ val coordinatorRef = env.rpcEnv.setupEndpoint(endpointName, coordinator)
+ logInfo("Registered StateStoreCoordinator endpoint")
+ new StateStoreCoordinatorRef(coordinatorRef)
+ } catch {
+ case e: IllegalArgumentException =>
--- End diff --
I think we can add a new API to RPC. This can be done later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200128527
**[Test build #53851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53851/consoleFull)** for PR 11645 at commit [`819ca17`](https://github.com/apache/spark/commit/819ca17412b60c6afeaa7ac0624abf80ebd422b9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199940980
**[Test build #53793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53793/consoleFull)** for PR 11645 at commit [`24fb325`](https://github.com/apache/spark/commit/24fb325a44fb63cb7021d10b44f516a2d40d02c6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56386237
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,465 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends StateStoreProvider with Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore( val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream =
+ serializer.newInstance().serializeStream(fs.create(tempDeltaFile, true))
+ private val allUpdates = new mutable.HashMap[InternalRow, StoreUpdate]
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /** Update the value of a key using the value generated by the update function */
+ override def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = mapToUpdate.get(key)
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+ allUpdates.get(key) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ tempDeltaFileStream.writeObject(ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: InternalRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keysIterator
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ mapToUpdate.remove(key)
+
+ allUpdates.get(key) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ tempDeltaFileStream.writeObject(KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ tempDeltaFileStream.close()
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[InternalRow] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.valuesIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ newMap ++= loadMap(version)
+ }
+ new HDFSBackedStateStore(version, newMap)
+ }
+
+ /** Manage backing files, including creating snapshots and cleaning up old files */
+ override def manage(): Unit = {
--- End diff --
No strong feelings either way, but this is kind of long. How about `doMaintenance`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199614268
**[Test build #53743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53743/consoleFull)** for PR 11645 at commit [`f4f3838`](https://github.com/apache/spark/commit/f4f383803e45aaea182b09c4e3c4b64415b9461e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196494409
**[Test build #53085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53085/consoleFull)** for PR 11645 at commit [`48afbe6`](https://github.com/apache/spark/commit/48afbe6b64ba07e28cc1e56d9aa3d850e03906e5).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199525639
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197031923
**[Test build #53225 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53225/consoleFull)** for PR 11645 at commit [`d5e2b10`](https://github.com/apache/spark/commit/d5e2b10ad11b22ea020dd3f3bb13a1a64aa33b50).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56918443
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(checkpointLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[(UnsafeRow, UnsafeRow)]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+/** Trait representing a provider of a specific version of a [[StateStore]]. */
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background maintenance */
+ def doMaintenance(): Unit = { }
+}
+
+/** Trait representing updates made to a [[StateStore]]. */
+sealed trait StoreUpdate
+
+case class ValueAdded(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+
+case class ValueUpdated(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+
+case class KeyRemoved(key: UnsafeRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrieve stores
+ * by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null),
+ * it also runs a periodic background tasks to do maintenance on the loaded stores. For each
+ * store, tt uses the [[StateStoreCoordinator]] to ensure whether the current loaded instance of
+ * the store is the active instance. Accordingly, it either keeps it loaded and performs
+ * maintenance, or unloads the store.
+ */
+private[state] object StateStore extends Logging {
+
+ val MAINTENANCE_INTERVAL_CONFIG = "spark.sql.streaming.stateStore.maintenanceInterval"
+ val MAINTENANCE_INTERVAL_DEFAULT_SECS = 60
+
+ private val loadedProviders = new mutable.HashMap[StateStoreId, StateStoreProvider]()
+ private val maintenanceTimer = new Timer("StateStore Timer", true)
+
+ @volatile private var maintenanceTask: TimerTask = null
+ @volatile private var _coordRef: StateStoreCoordinatorRef = null
+
+ /** Get or create a store associated with the id. */
+ def get(
+ storeId: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ version: Long,
+ hadoopConf: Configuration): StateStore = {
+ require(version >= 0)
+ val storeProvider = loadedProviders.synchronized {
+ startMaintenanceIfNeeded()
+ val sparkConf = Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
--- End diff --
We typically thread configuration through explicitly instead of getting it from the singleton.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196637004
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195181294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52898/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200051433
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53791/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56227417
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends StateStoreProvider with Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+
+ class HDFSBackedStateStore( val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream =
+ serializer.newInstance().serializeStream(fs.create(tempDeltaFile, true))
+ private val allUpdates = new mutable.HashMap[InternalRow, StoreUpdate]
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /** Update the value of a key using the value generated by the update function */
+ override def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = mapToUpdate.get(key)
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+ allUpdates.get(key) match {
+ case Some(ValueAdded(_, _)) =>
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
--- End diff --
add more inline comments to explain the cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200088994
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199511917
**[Test build #53718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53718/consoleFull)** for PR 11645 at commit [`3824053`](https://github.com/apache/spark/commit/3824053324aae89956601db060bfb77b013a2303).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196245523
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53070/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197128647
**[Test build #53258 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53258/consoleFull)** for PR 11645 at commit [`76dd988`](https://github.com/apache/spark/commit/76dd988fcac508609be3f32754284bc07524e481).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198621792
**[Test build #53593 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53593/consoleFull)** for PR 11645 at commit [`8cb0da8`](https://github.com/apache/spark/commit/8cb0da8a70e415297f7ee1d25d25eae1cc095b53).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200144942
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197121323
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53255/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197113154
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53250/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198626828
**[Test build #53596 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53596/consoleFull)** for PR 11645 at commit [`e6b1fb3`](https://github.com/apache/spark/commit/e6b1fb3f5ec4cdd711f0f8048933e745d8bf3d81).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198636672
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199442742
**[Test build #53690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53690/consoleFull)** for PR 11645 at commit [`25afe31`](https://github.com/apache/spark/commit/25afe31080451c16e74bd17bf9f81e5b8aab55f5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56376159
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala ---
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+
+import scala.collection.mutable
+import scala.util.Random
+
+import org.apache.hadoop.fs.Path
+import org.scalatest.{BeforeAndAfter, PrivateMethodTester}
+import org.scalatest.concurrent.Eventually._
+import org.scalatest.time.SpanSugar._
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+class StateStoreSuite extends SparkFunSuite with BeforeAndAfter with PrivateMethodTester {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ private val tempDir = Utils.createTempDir().toString
+
+ after {
+ StateStore.stop()
+ }
+
+ test("update, remove, commit, and all data iterator") {
+ val provider = newStoreProvider()
+
+ // Verify state before starting a new set of updates
+ assert(provider.latestIterator().isEmpty)
+
+ val store = provider.getStore(0)
+ assert(!store.hasCommitted)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+
+ // Verify state after updating
+ update(store, "a", 1)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+ assert(provider.latestIterator().isEmpty)
+
+ // Make updates, commit and then verify state
+ update(store, "b", 2)
+ update(store, "aa", 3)
+ remove(store, _.startsWith("a"))
+ assert(store.commit() === 1)
+
+ assert(store.hasCommitted)
+ assert(unwrapToSet(store.iterator()) === Set("b" -> 2))
+ assert(unwrapToSet(provider.latestIterator()) === Set("b" -> 2))
+ assert(fileExists(provider, version = 1, isSnapshot = false))
+ assert(getDataFromFiles(provider) === Set("b" -> 2))
+
+ // Trying to get newer versions should fail
+ intercept[Exception] {
+ provider.getStore(2)
+ }
+ intercept[Exception] {
+ getDataFromFiles(provider, 2)
+ }
+
+ // New updates to the reloaded store with new version, and does not change old version
+ val reloadedStore = new HDFSBackedStateStoreProvider(store.id, provider.directory).getStore(1)
+ update(reloadedStore, "c", 4)
+ assert(reloadedStore.commit() === 2)
+ assert(unwrapToSet(reloadedStore.iterator()) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider, version = 1) === Set("b" -> 2))
+ assert(getDataFromFiles(provider, version = 2) === Set("b" -> 2, "c" -> 4))
+ }
+
+ test("updates iterator with all combos of updates and removes") {
+ val provider = newStoreProvider()
+ var currentVersion: Int = 0
+ def withStore(body: StateStore => Unit): Unit = {
+ val store = provider.getStore(currentVersion)
+ body(store)
+ currentVersion += 1
+ }
+
+ // New data should be seen in updates as value added, even if they had multiple updates
+ withStore { store =>
+ update(store, "a", 1)
+ update(store, "aa", 1)
+ update(store, "aa", 2)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Added("a", 1), Added("aa", 2)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1, "aa" -> 2))
+ }
+
+ // Multiple updates to same key should be collapsed in the updates as a single value update
+ // Keys that have not been updated should not appear in the updates
+ withStore { store =>
+ update(store, "a", 4)
+ update(store, "a", 6)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 6)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Keys added, updated and finally removed before commit should not appear in updates
+ withStore { store =>
+ update(store, "b", 4) // Added, finally removed
+ update(store, "bb", 5) // Added, updated, finally removed
+ update(store, "bb", 6)
+ remove(store, _.startsWith("b"))
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set.empty)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Removed data should be seen in updates as a key removed
+ // Removed, but re-added data should be seen in updates as a value update
+ withStore { store =>
+ remove(store, _.startsWith("a"))
+ update(store, "a", 10)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 10), Removed("aa")))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 10))
+ }
+ }
+
+ test("cancel") {
+ val provider = newStoreProvider()
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ store.commit()
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ // cancelUpdates should not change the data in the files
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ store1.cancel()
+ assert(getDataFromFiles(provider) === Set("a" -> 1))
+ }
+
+ test("getStore with unexpected versions") {
+ val provider = newStoreProvider()
+
+ intercept[IllegalArgumentException] {
+ provider.getStore(-1)
+ }
+
+ // Prepare some data in the stoer
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ assert(store.commit() === 1)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ intercept[IllegalStateException] {
+ provider.getStore(2)
+ }
+
+ // Update store version with some data
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ assert(store1.commit() === 2)
+ assert(unwrapToSet(store1.iterator()) === Set("a" -> 1, "b" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "b" -> 1))
+
+ // Overwrite the version with other data
+ val store2 = provider.getStore(1)
+ update(store2, "c", 1)
+ assert(store2.commit() === 2)
+ assert(unwrapToSet(store2.iterator()) === Set("a" -> 1, "c" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "c" -> 1))
+ }
+
+ test("snapshotting") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ var currentVersion = 0
+ def updateVersionTo(targetVersion: Int): Unit = {
+ for (i <- currentVersion + 1 to targetVersion) {
+ val store = provider.getStore(currentVersion)
+ update(store, "a", i)
+ store.commit()
+ currentVersion += 1
+ }
+ require(currentVersion === targetVersion)
+ }
+
+ updateVersionTo(2)
+ require(getDataFromFiles(provider) === Set("a" -> 2))
+ provider.manage() // should not generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 2))
+
+ for (i <- 1 to currentVersion) {
+ assert(fileExists(provider, i, isSnapshot = false)) // all delta files present
+ assert(!fileExists(provider, i, isSnapshot = true)) // no snapshot files present
+ }
+
+ // After version 6, snapshotting should generate one snapshot file
+ updateVersionTo(6)
+ require(getDataFromFiles(provider) === Set("a" -> 6), "store not updated correctly")
+ provider.manage() // should generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 6), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 6))
+
+ val snapshotVersion = (0 to 6).find(version => fileExists(provider, version, isSnapshot = true))
+ assert(snapshotVersion.nonEmpty, "snapshot file not generated")
+
+ // After version 20, snapshotting should generate newer snapshot files
+ updateVersionTo(20)
+ require(getDataFromFiles(provider) === Set("a" -> 20), "store not updated correctly")
+ provider.manage() // do snapshot
+ assert(getDataFromFiles(provider) === Set("a" -> 20), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 20))
+
+ val latestSnapshotVersion = (0 to 20).filter(version =>
+ fileExists(provider, version, isSnapshot = true)).lastOption
+ assert(latestSnapshotVersion.nonEmpty, "no snapshot file found")
+ assert(latestSnapshotVersion.get > snapshotVersion.get, "newer snapshot not generated")
+
+ }
+
+ test("cleaning") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ for (i <- 1 to 20) {
+ val store = provider.getStore(i - 1)
+ update(store, "a", i)
+ store.commit()
+ provider.manage() // do cleanup
+ }
+ require(
+ unwrapToSet(provider.latestIterator()) === Set("a" -> 20),
+ "store not updated correctly")
+
+ assert(!fileExists(provider, version = 1, isSnapshot = false)) // first file should be deleted
+
+ // last couple of versions should be retrievable
+ assert(getDataFromFiles(provider, 20) === Set("a" -> 20))
+ assert(getDataFromFiles(provider, 19) === Set("a" -> 19))
+ }
+
+ test("StateStore.get") {
+ quietly {
+ val dir = Utils.createDirectory(tempDir, Random.nextString(5)).toString
+ val storeId = StateStoreId(0, 0)
+
+ // Verify that trying to get incorrect versions throw errors
+ intercept[IllegalArgumentException] {
+ StateStore.get(storeId, dir, -1)
+ }
+ assert(!StateStore.isLoaded(storeId)) // version -1 should not attempt to load the store
+
+ intercept[IllegalStateException] {
+ StateStore.get(storeId, dir, 1)
+ }
+
+ // Increase version of the store
+ val store0 = StateStore.get(storeId, dir, 0)
+ assert(store0.version === 0)
+ update(store0, "a", 1)
+ store0.commit()
+
+ assert(StateStore.get(storeId, dir, 1).version == 1)
+ assert(StateStore.get(storeId, dir, 0).version == 0)
+
+ // Verify that you can remove the store and still reload and use it
+ StateStore.remove(storeId)
+ assert(!StateStore.isLoaded(storeId))
+
+ val store1 = StateStore.get(storeId, dir, 1)
+ assert(StateStore.isLoaded(storeId))
+ update(store1, "a", 2)
+ assert(store1.commit() === 2)
+ assert(unwrapToSet(store1.iterator()) === Set("a" -> 2))
+ }
+ }
+
+ test("background management") {
+ val conf = new SparkConf()
+ .setMaster("local")
+ .setAppName("test")
+ .set("spark.sql.streaming.stateStore.managementInterval", "10ms")
+ val opId = 0
+ val storeId = StateStoreId(opId, 0)
+ val dir = Utils.createDirectory(tempDir, Random.nextString(5)).toString
--- End diff --
createTempDir
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195181291
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199983893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53793/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199443028
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198596903
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53590/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199965006
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198622504
**[Test build #53594 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53594/consoleFull)** for PR 11645 at commit [`9fb9c43`](https://github.com/apache/spark/commit/9fb9c43b0d8ebef0b437226698263f058c04cf18).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199630830
**[Test build #53740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53740/consoleFull)** for PR 11645 at commit [`5b7cf53`](https://github.com/apache/spark/commit/5b7cf538cf4b6485ddee102568e931d66f9c1a75).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199943883
**[Test build #53794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53794/consoleFull)** for PR 11645 at commit [`4752d73`](https://github.com/apache/spark/commit/4752d73af2b7257e0160d1eac81ee9b9279d3109).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56374099
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,465 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends StateStoreProvider with Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore( val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream =
+ serializer.newInstance().serializeStream(fs.create(tempDeltaFile, true))
+ private val allUpdates = new mutable.HashMap[InternalRow, StoreUpdate]
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /** Update the value of a key using the value generated by the update function */
+ override def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = mapToUpdate.get(key)
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+ allUpdates.get(key) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ tempDeltaFileStream.writeObject(ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: InternalRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keysIterator
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ mapToUpdate.remove(key)
+
+ allUpdates.get(key) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ tempDeltaFileStream.writeObject(KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ tempDeltaFileStream.close()
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[InternalRow] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.valuesIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ newMap ++= loadMap(version)
+ }
+ new HDFSBackedStateStore(version, newMap)
+ }
+
+ /** Manage backing files, including creating snapshots and cleaning up old files */
+ override def manage(): Unit = {
--- End diff --
`performMaintenance()` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197671947
**[Test build #53388 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53388/consoleFull)** for PR 11645 at commit [`15e1780`](https://github.com/apache/spark/commit/15e178059d86b8284da4c7e095f70d63c1b7acbe).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200088824
**[Test build #53810 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53810/consoleFull)** for PR 11645 at commit [`b147f59`](https://github.com/apache/spark/commit/b147f599362ca4a6851f2b2d2140330bcdc6de0c).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195645941
**[Test build #52988 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52988/consoleFull)** for PR 11645 at commit [`7d74c67`](https://github.com/apache/spark/commit/7d74c67f999d53e9f02cf9ed5d2c8d2e951144ca).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195645943
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52988/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56376208
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala ---
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+
+import scala.collection.mutable
+import scala.util.Random
+
+import org.apache.hadoop.fs.Path
+import org.scalatest.{BeforeAndAfter, PrivateMethodTester}
+import org.scalatest.concurrent.Eventually._
+import org.scalatest.time.SpanSugar._
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+class StateStoreSuite extends SparkFunSuite with BeforeAndAfter with PrivateMethodTester {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ private val tempDir = Utils.createTempDir().toString
+
+ after {
+ StateStore.stop()
+ }
+
+ test("update, remove, commit, and all data iterator") {
+ val provider = newStoreProvider()
+
+ // Verify state before starting a new set of updates
+ assert(provider.latestIterator().isEmpty)
+
+ val store = provider.getStore(0)
+ assert(!store.hasCommitted)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+
+ // Verify state after updating
+ update(store, "a", 1)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+ assert(provider.latestIterator().isEmpty)
+
+ // Make updates, commit and then verify state
+ update(store, "b", 2)
+ update(store, "aa", 3)
+ remove(store, _.startsWith("a"))
+ assert(store.commit() === 1)
+
+ assert(store.hasCommitted)
+ assert(unwrapToSet(store.iterator()) === Set("b" -> 2))
+ assert(unwrapToSet(provider.latestIterator()) === Set("b" -> 2))
+ assert(fileExists(provider, version = 1, isSnapshot = false))
+ assert(getDataFromFiles(provider) === Set("b" -> 2))
+
+ // Trying to get newer versions should fail
+ intercept[Exception] {
+ provider.getStore(2)
+ }
+ intercept[Exception] {
+ getDataFromFiles(provider, 2)
+ }
+
+ // New updates to the reloaded store with new version, and does not change old version
+ val reloadedStore = new HDFSBackedStateStoreProvider(store.id, provider.directory).getStore(1)
+ update(reloadedStore, "c", 4)
+ assert(reloadedStore.commit() === 2)
+ assert(unwrapToSet(reloadedStore.iterator()) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider, version = 1) === Set("b" -> 2))
+ assert(getDataFromFiles(provider, version = 2) === Set("b" -> 2, "c" -> 4))
+ }
+
+ test("updates iterator with all combos of updates and removes") {
+ val provider = newStoreProvider()
+ var currentVersion: Int = 0
+ def withStore(body: StateStore => Unit): Unit = {
+ val store = provider.getStore(currentVersion)
+ body(store)
+ currentVersion += 1
+ }
+
+ // New data should be seen in updates as value added, even if they had multiple updates
+ withStore { store =>
+ update(store, "a", 1)
+ update(store, "aa", 1)
+ update(store, "aa", 2)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Added("a", 1), Added("aa", 2)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1, "aa" -> 2))
+ }
+
+ // Multiple updates to same key should be collapsed in the updates as a single value update
+ // Keys that have not been updated should not appear in the updates
+ withStore { store =>
+ update(store, "a", 4)
+ update(store, "a", 6)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 6)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Keys added, updated and finally removed before commit should not appear in updates
+ withStore { store =>
+ update(store, "b", 4) // Added, finally removed
+ update(store, "bb", 5) // Added, updated, finally removed
+ update(store, "bb", 6)
+ remove(store, _.startsWith("b"))
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set.empty)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Removed data should be seen in updates as a key removed
+ // Removed, but re-added data should be seen in updates as a value update
+ withStore { store =>
+ remove(store, _.startsWith("a"))
+ update(store, "a", 10)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 10), Removed("aa")))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 10))
+ }
+ }
+
+ test("cancel") {
+ val provider = newStoreProvider()
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ store.commit()
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ // cancelUpdates should not change the data in the files
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ store1.cancel()
+ assert(getDataFromFiles(provider) === Set("a" -> 1))
+ }
+
+ test("getStore with unexpected versions") {
+ val provider = newStoreProvider()
+
+ intercept[IllegalArgumentException] {
+ provider.getStore(-1)
+ }
+
+ // Prepare some data in the stoer
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ assert(store.commit() === 1)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ intercept[IllegalStateException] {
+ provider.getStore(2)
+ }
+
+ // Update store version with some data
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ assert(store1.commit() === 2)
+ assert(unwrapToSet(store1.iterator()) === Set("a" -> 1, "b" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "b" -> 1))
+
+ // Overwrite the version with other data
+ val store2 = provider.getStore(1)
+ update(store2, "c", 1)
+ assert(store2.commit() === 2)
+ assert(unwrapToSet(store2.iterator()) === Set("a" -> 1, "c" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "c" -> 1))
+ }
+
+ test("snapshotting") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ var currentVersion = 0
+ def updateVersionTo(targetVersion: Int): Unit = {
+ for (i <- currentVersion + 1 to targetVersion) {
+ val store = provider.getStore(currentVersion)
+ update(store, "a", i)
+ store.commit()
+ currentVersion += 1
+ }
+ require(currentVersion === targetVersion)
+ }
+
+ updateVersionTo(2)
+ require(getDataFromFiles(provider) === Set("a" -> 2))
+ provider.manage() // should not generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 2))
+
+ for (i <- 1 to currentVersion) {
+ assert(fileExists(provider, i, isSnapshot = false)) // all delta files present
+ assert(!fileExists(provider, i, isSnapshot = true)) // no snapshot files present
+ }
+
+ // After version 6, snapshotting should generate one snapshot file
+ updateVersionTo(6)
+ require(getDataFromFiles(provider) === Set("a" -> 6), "store not updated correctly")
+ provider.manage() // should generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 6), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 6))
+
+ val snapshotVersion = (0 to 6).find(version => fileExists(provider, version, isSnapshot = true))
+ assert(snapshotVersion.nonEmpty, "snapshot file not generated")
+
+ // After version 20, snapshotting should generate newer snapshot files
+ updateVersionTo(20)
+ require(getDataFromFiles(provider) === Set("a" -> 20), "store not updated correctly")
+ provider.manage() // do snapshot
+ assert(getDataFromFiles(provider) === Set("a" -> 20), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 20))
+
+ val latestSnapshotVersion = (0 to 20).filter(version =>
+ fileExists(provider, version, isSnapshot = true)).lastOption
+ assert(latestSnapshotVersion.nonEmpty, "no snapshot file found")
+ assert(latestSnapshotVersion.get > snapshotVersion.get, "newer snapshot not generated")
+
+ }
+
+ test("cleaning") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ for (i <- 1 to 20) {
+ val store = provider.getStore(i - 1)
+ update(store, "a", i)
+ store.commit()
+ provider.manage() // do cleanup
+ }
+ require(
+ unwrapToSet(provider.latestIterator()) === Set("a" -> 20),
+ "store not updated correctly")
+
+ assert(!fileExists(provider, version = 1, isSnapshot = false)) // first file should be deleted
+
+ // last couple of versions should be retrievable
+ assert(getDataFromFiles(provider, 20) === Set("a" -> 20))
+ assert(getDataFromFiles(provider, 19) === Set("a" -> 19))
+ }
+
+ test("StateStore.get") {
+ quietly {
+ val dir = Utils.createDirectory(tempDir, Random.nextString(5)).toString
+ val storeId = StateStoreId(0, 0)
+
+ // Verify that trying to get incorrect versions throw errors
+ intercept[IllegalArgumentException] {
+ StateStore.get(storeId, dir, -1)
+ }
+ assert(!StateStore.isLoaded(storeId)) // version -1 should not attempt to load the store
+
+ intercept[IllegalStateException] {
+ StateStore.get(storeId, dir, 1)
+ }
+
+ // Increase version of the store
+ val store0 = StateStore.get(storeId, dir, 0)
+ assert(store0.version === 0)
+ update(store0, "a", 1)
+ store0.commit()
+
+ assert(StateStore.get(storeId, dir, 1).version == 1)
+ assert(StateStore.get(storeId, dir, 0).version == 0)
+
+ // Verify that you can remove the store and still reload and use it
+ StateStore.remove(storeId)
+ assert(!StateStore.isLoaded(storeId))
+
+ val store1 = StateStore.get(storeId, dir, 1)
+ assert(StateStore.isLoaded(storeId))
+ update(store1, "a", 2)
+ assert(store1.commit() === 2)
+ assert(unwrapToSet(store1.iterator()) === Set("a" -> 2))
+ }
+ }
+
+ test("background management") {
+ val conf = new SparkConf()
+ .setMaster("local")
+ .setAppName("test")
+ .set("spark.sql.streaming.stateStore.managementInterval", "10ms")
+ val opId = 0
+ val storeId = StateStoreId(opId, 0)
+ val dir = Utils.createDirectory(tempDir, Random.nextString(5)).toString
+ val provider = new HDFSBackedStateStoreProvider(storeId, dir)
+
+ quietly {
+ withSpark(new SparkContext(conf)) { sc =>
+ withCoordinator(sc) { coordinator =>
+ for (i <- 1 to 20) {
+ val store = StateStore.get(storeId, dir, i - 1)
+ update(store, "a", i)
+ store.commit()
+ }
+
+ // Background management should clean up and generate snapshots
+ eventually(timeout(4 seconds)) {
+ // Earliest delta file should get cleaned up
+ assert(!fileExists(provider, 1, isSnapshot = false), "earliest file not deleted")
+
+ // Some snapshots should have been generated
+ val snapshotVersions = (0 to 20).filter { version =>
+ fileExists(provider, version, isSnapshot = true)
+ }
+ assert(snapshotVersions.nonEmpty, "no snapshot file found")
+ }
+
+ // If driver decides to deactivate all instances of the store, then this instance
+ // should be unloaded
+ coordinator.deactivateInstances(Set(opId))
+ eventually(timeout(4 seconds)) {
+ assert(!StateStore.isLoaded(storeId))
+ }
+
+ // Reload the store and verify
+ StateStore.get(storeId, dir, 20)
+ assert(StateStore.isLoaded(storeId))
+
+ // If some other executor loads the store, then this instance should be unloaded
+ coordinator.reportActiveInstance(storeId, "other-host", "other-exec")
+ eventually(timeout(4 seconds)) {
+ assert(!StateStore.isLoaded(storeId))
+ }
+
+ // Reload the store and verify
+ StateStore.get(storeId, dir, 20)
+ assert(StateStore.isLoaded(storeId))
+ }
+ }
+
+ // Verify if instance is unloaded if SparkContext is stopped
+ eventually(timeout(4 seconds)) {
+ assert(!StateStore.isLoaded(storeId))
+ }
+ }
+ }
+
+ def getDataFromFiles(
+ provider: HDFSBackedStateStoreProvider,
+ version: Int = -1): Set[(String, Int)] = {
+ val reloadedProvider = new HDFSBackedStateStoreProvider(provider.id, provider.directory)
+ if (version < 0) {
+ reloadedProvider.latestIterator().map(unwrapKeyValue).toSet
+ } else {
+ reloadedProvider.iterator(version).map(unwrapKeyValue).toSet
+ }
+ }
+
+ def assertMap(
+ testMapOption: Option[MapType],
+ expectedMap: Map[String, Int]): Unit = {
+ assert(testMapOption.nonEmpty, "no map present")
+ val convertedMap = testMapOption.get.map(unwrapKeyValue)
+ assert(convertedMap === expectedMap)
+ }
+
+ def fileExists(
+ provider: HDFSBackedStateStoreProvider,
+ version: Long,
+ isSnapshot: Boolean): Boolean = {
+ val method = PrivateMethod[Path]('baseDir)
+ val basePath = provider invokePrivate method()
+ val fileName = if (isSnapshot) s"$version.snapshot" else s"$version.delta"
+ val filePath = new File(basePath.toString, fileName)
+ filePath.exists
+ }
+
+ def storeLoaded(storeId: StateStoreId): Boolean = {
+ val method = PrivateMethod[mutable.HashMap[StateStoreId, StateStore]]('loadedStores)
+ val loadedStores = StateStore invokePrivate method()
+ loadedStores.contains(storeId)
+ }
+
+ def unloadStore(storeId: StateStoreId): Boolean = {
+ val method = PrivateMethod('remove)
+ StateStore invokePrivate method(storeId)
+ }
+
+ def newStoreProvider(
+ opId: Long = Random.nextLong,
+ partition: Int = 0,
+ maxDeltaChainForSnapshots: Int = 10
+ ): HDFSBackedStateStoreProvider = {
+ new HDFSBackedStateStoreProvider(
+ StateStoreId(opId, partition),
+ Utils.createDirectory(tempDir, Random.nextString(5)).toString,
--- End diff --
createTempDir
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56740432
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala ---
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+
+import scala.collection.mutable
+import scala.util.Random
+
+import org.apache.hadoop.fs.Path
+import org.scalatest.{BeforeAndAfter, PrivateMethodTester}
+import org.scalatest.concurrent.Eventually._
+import org.scalatest.time.SpanSugar._
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+class StateStoreSuite extends SparkFunSuite with BeforeAndAfter with PrivateMethodTester {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ private val tempDir = Utils.createTempDir().toString
+
+ after {
+ StateStore.stop()
+ }
+
+ test("update, remove, commit, and all data iterator") {
+ val provider = newStoreProvider()
+
+ // Verify state before starting a new set of updates
+ assert(provider.latestIterator().isEmpty)
+
+ val store = provider.getStore(0)
+ assert(!store.hasCommitted)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+
+ // Verify state after updating
+ update(store, "a", 1)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+ assert(provider.latestIterator().isEmpty)
+
+ // Make updates, commit and then verify state
+ update(store, "b", 2)
+ update(store, "aa", 3)
+ remove(store, _.startsWith("a"))
+ assert(store.commit() === 1)
+
+ assert(store.hasCommitted)
+ assert(unwrapToSet(store.iterator()) === Set("b" -> 2))
+ assert(unwrapToSet(provider.latestIterator()) === Set("b" -> 2))
+ assert(fileExists(provider, version = 1, isSnapshot = false))
+ assert(getDataFromFiles(provider) === Set("b" -> 2))
+
+ // Trying to get newer versions should fail
+ intercept[Exception] {
+ provider.getStore(2)
+ }
+ intercept[Exception] {
+ getDataFromFiles(provider, 2)
+ }
+
+ // New updates to the reloaded store with new version, and does not change old version
+ val reloadedStore = new HDFSBackedStateStoreProvider(store.id, provider.directory).getStore(1)
+ update(reloadedStore, "c", 4)
+ assert(reloadedStore.commit() === 2)
+ assert(unwrapToSet(reloadedStore.iterator()) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider, version = 1) === Set("b" -> 2))
+ assert(getDataFromFiles(provider, version = 2) === Set("b" -> 2, "c" -> 4))
+ }
+
+ test("updates iterator with all combos of updates and removes") {
+ val provider = newStoreProvider()
+ var currentVersion: Int = 0
+ def withStore(body: StateStore => Unit): Unit = {
+ val store = provider.getStore(currentVersion)
+ body(store)
+ currentVersion += 1
+ }
+
+ // New data should be seen in updates as value added, even if they had multiple updates
+ withStore { store =>
+ update(store, "a", 1)
+ update(store, "aa", 1)
+ update(store, "aa", 2)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Added("a", 1), Added("aa", 2)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1, "aa" -> 2))
+ }
+
+ // Multiple updates to same key should be collapsed in the updates as a single value update
+ // Keys that have not been updated should not appear in the updates
+ withStore { store =>
+ update(store, "a", 4)
+ update(store, "a", 6)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 6)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Keys added, updated and finally removed before commit should not appear in updates
+ withStore { store =>
+ update(store, "b", 4) // Added, finally removed
+ update(store, "bb", 5) // Added, updated, finally removed
+ update(store, "bb", 6)
+ remove(store, _.startsWith("b"))
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set.empty)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Removed data should be seen in updates as a key removed
+ // Removed, but re-added data should be seen in updates as a value update
+ withStore { store =>
+ remove(store, _.startsWith("a"))
+ update(store, "a", 10)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 10), Removed("aa")))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 10))
+ }
+ }
+
+ test("cancel") {
+ val provider = newStoreProvider()
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ store.commit()
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ // cancelUpdates should not change the data in the files
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ store1.cancel()
+ assert(getDataFromFiles(provider) === Set("a" -> 1))
+ }
+
+ test("getStore with unexpected versions") {
+ val provider = newStoreProvider()
+
+ intercept[IllegalArgumentException] {
+ provider.getStore(-1)
+ }
+
+ // Prepare some data in the stoer
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ assert(store.commit() === 1)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ intercept[IllegalStateException] {
+ provider.getStore(2)
+ }
+
+ // Update store version with some data
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ assert(store1.commit() === 2)
+ assert(unwrapToSet(store1.iterator()) === Set("a" -> 1, "b" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "b" -> 1))
+
+ // Overwrite the version with other data
+ val store2 = provider.getStore(1)
+ update(store2, "c", 1)
+ assert(store2.commit() === 2)
+ assert(unwrapToSet(store2.iterator()) === Set("a" -> 1, "c" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "c" -> 1))
+ }
+
+ test("snapshotting") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ var currentVersion = 0
+ def updateVersionTo(targetVersion: Int): Unit = {
+ for (i <- currentVersion + 1 to targetVersion) {
+ val store = provider.getStore(currentVersion)
+ update(store, "a", i)
+ store.commit()
+ currentVersion += 1
+ }
+ require(currentVersion === targetVersion)
+ }
+
+ updateVersionTo(2)
+ require(getDataFromFiles(provider) === Set("a" -> 2))
+ provider.manage() // should not generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 2))
+
+ for (i <- 1 to currentVersion) {
+ assert(fileExists(provider, i, isSnapshot = false)) // all delta files present
+ assert(!fileExists(provider, i, isSnapshot = true)) // no snapshot files present
+ }
+
+ // After version 6, snapshotting should generate one snapshot file
+ updateVersionTo(6)
+ require(getDataFromFiles(provider) === Set("a" -> 6), "store not updated correctly")
+ provider.manage() // should generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 6), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 6))
+
+ val snapshotVersion = (0 to 6).find(version => fileExists(provider, version, isSnapshot = true))
+ assert(snapshotVersion.nonEmpty, "snapshot file not generated")
+
+ // After version 20, snapshotting should generate newer snapshot files
+ updateVersionTo(20)
+ require(getDataFromFiles(provider) === Set("a" -> 20), "store not updated correctly")
+ provider.manage() // do snapshot
+ assert(getDataFromFiles(provider) === Set("a" -> 20), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 20))
+
+ val latestSnapshotVersion = (0 to 20).filter(version =>
+ fileExists(provider, version, isSnapshot = true)).lastOption
+ assert(latestSnapshotVersion.nonEmpty, "no snapshot file found")
+ assert(latestSnapshotVersion.get > snapshotVersion.get, "newer snapshot not generated")
+
+ }
+
+ test("cleaning") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ for (i <- 1 to 20) {
+ val store = provider.getStore(i - 1)
+ update(store, "a", i)
+ store.commit()
+ provider.manage() // do cleanup
+ }
+ require(
+ unwrapToSet(provider.latestIterator()) === Set("a" -> 20),
+ "store not updated correctly")
+
+ assert(!fileExists(provider, version = 1, isSnapshot = false)) // first file should be deleted
+
+ // last couple of versions should be retrievable
+ assert(getDataFromFiles(provider, 20) === Set("a" -> 20))
+ assert(getDataFromFiles(provider, 19) === Set("a" -> 19))
+ }
+
+ test("StateStore.get") {
+ quietly {
+ val dir = Utils.createDirectory(tempDir, Random.nextString(5)).toString
--- End diff --
The root `tempDir` is already a temporary directory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195645942
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198596900
**[Test build #53590 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53590/consoleFull)** for PR 11645 at commit [`3fe34e9`](https://github.com/apache/spark/commit/3fe34e90bf7820c73ede7901422a48b83c954f27).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200144946
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53857/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199571104
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57094759
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ storeConf: StateStoreConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ private val allUpdates = new java.util.HashMap[UnsafeRow, StoreUpdate]()
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ override def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = Option(mapToUpdate.get(key))
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ writeToDeltaFile(tempDeltaFileStream, ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: UnsafeRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keySet().iterator()
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ keyIter.remove()
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ writeToDeltaFile(tempDeltaFileStream, KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ finalizeDeltaFile(tempDeltaFileStream)
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ logInfo(s"Committed version $newVersion for $this")
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ logInfo("Canceled ")
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[(UnsafeRow, UnsafeRow)] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.values().asScala.toIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ val time = System.nanoTime()
+ newMap.putAll(loadMap(version))
+ }
+ val store = new HDFSBackedStateStore(version, newMap)
+ logInfo(s"Retrieved version $version of $this for update")
+ store
+ }
+
+ /** Do maintenance backing data files, including creating snapshots and cleaning up old files */
+ override def doMaintenance(): Unit = {
+ try {
+ doSnapshot()
+ cleanup()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up $this")
+ }
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ /* Internal classes and methods */
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir =
+ new Path(id.checkpointLocation, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(hadoopConf)
+ private val sparkConf = Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
+
+ initialize()
+
+ private case class StoreFile(version: Long, path: Path, isSnapshot: Boolean)
+
+ /** Commit a set of updates to the store with the given new version */
+ private def commitUpdates(newVersion: Long, map: MapType, tempDeltaFile: Path): Path = {
+ synchronized {
+ val finalDeltaFile = deltaFile(newVersion)
+ fs.rename(tempDeltaFile, finalDeltaFile)
+ loadedMaps.put(newVersion, map)
+ finalDeltaFile
+ }
+ }
+
+ /**
+ * Get iterator of all the data of the latest version of the store.
+ * Note that this will look up the files to determined the latest known version.
+ */
+ private[state] def latestIterator(): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ val versionsInFiles = fetchFiles().map(_.version).toSet
+ val versionsLoaded = loadedMaps.keySet
+ val allKnownVersions = versionsInFiles ++ versionsLoaded
+ if (allKnownVersions.nonEmpty) {
+ loadMap(allKnownVersions.max).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ } else Iterator.empty
+ }
+
+ /** Get iterator of a specific version of the store */
+ private[state] def iterator(version: Long): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ loadMap(version).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ }
+
+ /** Initialize the store provider */
+ private def initialize(): Unit = {
+ if (!fs.exists(baseDir)) {
+ fs.mkdirs(baseDir)
+ } else {
+ if (!fs.isDirectory(baseDir)) {
+ throw new IllegalStateException(
+ s"Cannot use ${id.checkpointLocation} for storing state data for $this as" +
+ s"$baseDir already exists and is not a directory")
+ }
+ }
+ }
+
+ /** Load the required version of the map data from the backing files */
+ private def loadMap(version: Long): MapType = {
+ if (version <= 0) return new MapType
+ synchronized { loadedMaps.get(version) }.getOrElse {
+ val mapFromFile = readSnapshotFile(version).getOrElse {
+ val prevMap = loadMap(version - 1)
+ val newMap = new MapType(prevMap)
+ newMap.putAll(prevMap)
+ updateFromDeltaFile(version, newMap)
+ newMap
+ }
+ loadedMaps.put(version, mapFromFile)
+ mapFromFile
+ }
+ }
+
+ private def writeToDeltaFile(output: DataOutputStream, update: StoreUpdate): Unit = {
+
+ def writeUpdate(key: UnsafeRow, value: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ val valueBytes = value.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+
+ def writeRemove(key: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(-1)
+ }
+
+ update match {
+ case ValueAdded(key, value) =>
+ writeUpdate(key, value)
+ case ValueUpdated(key, value) =>
+ writeUpdate(key, value)
+ case KeyRemoved(key) =>
+ writeRemove(key)
+ }
+ }
+
+ private def finalizeDeltaFile(output: DataOutputStream): Unit = {
+ output.writeInt(-1) // Write this magic number to signify end of file
+ output.close()
+ }
+
+ private def updateFromDeltaFile(version: Long, map: MapType): Unit = {
+ val fileToRead = deltaFile(version)
+ if (!fs.exists(fileToRead)) {
+ throw new IllegalStateException(
+ s"Error reading delta file $fileToRead of $this: $fileToRead does not exist")
+ }
+ var input: DataInputStream = null
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while(!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading delta file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ map.remove(keyRow)
+ } else {
+ val valueRowBuffer = new Array[Byte](valueSize)
+ ByteStreams.readFully(input, valueRowBuffer, 0, valueSize)
+ val valueRow = new UnsafeRow(valueSchema.fields.length)
+ valueRow.pointTo(valueRowBuffer, valueSize)
+ map.put(keyRow, valueRow)
+ }
+ }
+ }
+ } finally {
+ if (input != null) input.close()
+ }
+ logInfo(s"Read delta file for version $version of $this from $fileToRead")
+ }
+
+ private def writeSnapshotFile(version: Long, map: MapType): Unit = {
+ val fileToWrite = snapshotFile(version)
+ var output: DataOutputStream = null
+ Utils.tryWithSafeFinally {
+ output = compressStream(fs.create(fileToWrite, false))
+ val iter = map.entrySet().iterator()
+ while(iter.hasNext) {
+ val entry = iter.next()
+ val keyBytes = entry.getKey.getBytes()
+ val valueBytes = entry.getValue.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+ output.writeInt(-1)
+ } {
+ if (output != null) output.close()
+ }
+ logInfo(s"Written snapshot file for version $version of $this at $fileToWrite")
+ }
+
+ private def readSnapshotFile(version: Long): Option[MapType] = {
+ val fileToRead = snapshotFile(version)
+ if (!fs.exists(fileToRead)) return None
+
+ val map = new MapType()
+ var input: DataInputStream = null
+
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while (!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
--- End diff --
Removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57067817
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{Partition, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.{SerializableConfiguration, Utils}
+
+/**
+ * An RDD that allows computations to be executed against [[StateStore]]s. It
+ * uses the [[StateStoreCoordinator]] to use the locations of loaded state stores as
+ * preferred locations.
+ */
+class StateStoreRDD[T: ClassTag, U: ClassTag](
+ dataRDD: RDD[T],
+ storeUpdateFunction: (StateStore, Iterator[T]) => Iterator[U],
--- End diff --
nit: We should have a doc to say that `StateStore` should be committed in `storeUpdateFunction`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56919469
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEndpointRef, RpcEnv}
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.util.RpcUtils
+
+/** Trait representing all messages to [[StateStoreCoordinator]] */
+private sealed trait StateStoreCoordinatorMessage extends Serializable
+
+/** Classes representing messages */
+private case class ReportActiveInstance(storeId: StateStoreId, host: String, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class VerifyIfInstanceActive(storeId: StateStoreId, executorId: String)
+ extends StateStoreCoordinatorMessage
+
+private case class GetLocation(storeId: StateStoreId)
+ extends StateStoreCoordinatorMessage
+
+private case class DeactivateInstances(storeRootLocation: String)
+ extends StateStoreCoordinatorMessage
+
+private object StopCoordinator
+ extends StateStoreCoordinatorMessage
+
+/** Helper object used to create reference to [[StateStoreCoordinator]]. */
+private[sql] object StateStoreCoordinatorRef extends Logging {
+
+ private val endpointName = "StateStoreCoordinator"
+
+ /**
+ * Create a reference to a [[StateStoreCoordinator]], This can be called from driver as well as
+ * executors.
+ */
+ def apply(env: SparkEnv): StateStoreCoordinatorRef = synchronized {
+ try {
+ val coordinator = new StateStoreCoordinator(env.rpcEnv)
+ val coordinatorRef = env.rpcEnv.setupEndpoint(endpointName, coordinator)
+ logInfo("Registered StateStoreCoordinator endpoint")
+ new StateStoreCoordinatorRef(coordinatorRef)
+ } catch {
+ case e: IllegalArgumentException =>
--- End diff --
This is a pretty general exception to catch. Can we not check to see if it already exists instead of relying on an exception to get the already existing coordinator.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197090248
**[Test build #53252 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53252/consoleFull)** for PR 11645 at commit [`e89f4d0`](https://github.com/apache/spark/commit/e89f4d0d1a45d5a4d8c59a433cf55ce5d19010da).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200144193
**[Test build #53857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53857/consoleFull)** for PR 11645 at commit [`70cc7b1`](https://github.com/apache/spark/commit/70cc7b11c0d502d4a741ccdcab5e7aa006b5b311).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199490879
**[Test build #53714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53714/consoleFull)** for PR 11645 at commit [`32c0139`](https://github.com/apache/spark/commit/32c013918677cb2c93385fbfa6a38b4b8594bea1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199631002
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53740/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197117447
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53252/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56916936
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[(UnsafeRow, UnsafeRow)]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background management */
+ def doMaintenance(): Unit = { }
+}
+
+sealed trait StoreUpdate
+case class ValueAdded(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class ValueUpdated(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class KeyRemoved(key: UnsafeRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrieve stores
+ * by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null),
+ * it also runs a periodic background tasks to do maintenance on the loaded stores. For each
+ * store, tt uses the [[StateStoreCoordinator]] to ensure whether the current loaded instance of
+ * the store is the active instance. Accordingly, it either keeps it loaded and performance
+ * maintenance, or unloads the store.
+ */
+private[state] object StateStore extends Logging {
+
+ val MAINTENANCE_INTERVAL_CONFIG = "spark.sql.streaming.stateStore.maintenanceInterval"
+ val MAINTENANCE_INTERVAL_DEFAULT_SECS = 60
+
+ private val loadedProviders = new mutable.HashMap[StateStoreId, StateStoreProvider]()
+ private val managementTimer = new Timer("StateStore Timer", true)
+
+ @volatile private var managementTask: TimerTask = null
+ @volatile private var _coordRef: StateStoreCoordinatorRef = null
+
+ /** Get or create a store associated with the id. */
+ def get(
+ storeId: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ version: Long,
+ hadoopConf: Configuration
+ ): StateStore = {
--- End diff --
Indent. Also I don't think we typically wrap the return.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56918188
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,597 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.serializer.KryoSerializer
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ sparkConf: SparkConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ import HDFSBackedStateStoreProvider._
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ // private val tempDeltaFileStream = fs.create(tempDeltaFile, true)
+ private val allUpdates = new java.util.HashMap[UnsafeRow, StoreUpdate]()
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /** Update the value of a key using the value generated by the update function */
+ override def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = Option(mapToUpdate.get(key))
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ writeToDeltaFile(tempDeltaFileStream, ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: UnsafeRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keySet().iterator()
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ keyIter.remove()
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ writeToDeltaFile(tempDeltaFileStream, KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ finalizeDeltaFile(tempDeltaFileStream)
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ logInfo(s"Committed version $newVersion for $this")
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ logInfo("Canceled ")
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[(UnsafeRow, UnsafeRow)] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.values().asScala.toIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ val time = System.nanoTime()
+ newMap.putAll(loadMap(version))
+ }
+ val store = new HDFSBackedStateStore(version, newMap)
+ logInfo(s"Retrieved version $version of $this for update")
+ store
+ }
+
+ /** Do maintenance backing data files, including creating snapshots and cleaning up old files */
+ override def doMaintenance(): Unit = {
+ try {
+ doSnapshot()
+ cleanup()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up $this")
+ }
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ /* Internal classes and methods */
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir = new Path(id.checkpointLocation, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(hadoopConf)
+ private val serializer = new KryoSerializer(sparkConf)
--- End diff --
Is this still used?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197032651
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53225/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56227475
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends StateStoreProvider with Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+
+ class HDFSBackedStateStore( val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream =
+ serializer.newInstance().serializeStream(fs.create(tempDeltaFile, true))
+ private val allUpdates = new mutable.HashMap[InternalRow, StoreUpdate]
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /** Update the value of a key using the value generated by the update function */
+ override def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = mapToUpdate.get(key)
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+ allUpdates.get(key) match {
+ case Some(ValueAdded(_, _)) =>
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ tempDeltaFileStream.writeObject(ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: InternalRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keysIterator
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ mapToUpdate.remove(key)
+ allUpdates.get(key) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ tempDeltaFileStream.writeObject(KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ tempDeltaFileStream.close()
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[InternalRow] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.valuesIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ newMap ++= loadMap(version)
+ }
+ new HDFSBackedStateStore(version, newMap)
+ }
+
+ override def manage(): Unit = {
+ try {
+ doSnapshot()
+ cleanup()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up $this")
+ }
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ /* Internal classes and methods */
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir = new Path(directory, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(new Configuration())
+ private val serializer = new KryoSerializer(new SparkConf)
+
+ initialize()
+
+ private case class StoreFile(version: Long, path: Path, isSnapshot: Boolean)
+
+ /** Commit a set of updates to the store with the given new version */
+ private def commitUpdates(newVersion: Long, map: MapType, tempDeltaFile: Path): Path = {
+ synchronized {
+ val finalDeltaFile = deltaFile(newVersion)
+ fs.rename(tempDeltaFile, finalDeltaFile)
+ loadedMaps.put(newVersion, map)
+ finalDeltaFile
+ }
+ }
+
+ /**
+ * Get iterator of all the data of the latest version of the store.
+ * Note that this will look up the files to determined the latest known version.
+ */
+ private[state] def latestIterator(): Iterator[InternalRow] = synchronized {
+ val versionsInFiles = fetchFiles().map(_.version).toSet
+ val versionsLoaded = loadedMaps.keySet
+ val allKnownVersions = versionsInFiles ++ versionsLoaded
+ if (allKnownVersions.nonEmpty) {
+ loadMap(allKnownVersions.max)
+ .iterator
+ .map { case (key, value) => new JoinedRow(key, value) }
+ } else Iterator.empty
+ }
+
+ /** Get iterator of a specific version of the store */
+ private[state] def iterator(version: Long): Iterator[InternalRow] = synchronized {
+ loadMap(version)
+ .iterator
+ .map { case (key, value) => new JoinedRow(key, value) }
+ }
+
+ /** Initialize the store provider */
+ private def initialize(): Unit = {
+ if (!fs.exists(baseDir)) {
+ fs.mkdirs(baseDir)
+ } else {
+ if (!fs.isDirectory(baseDir)) {
+ throw new IllegalStateException(
+ s"Cannot use $directory for storing state data as" +
+ s"$baseDir already exists and is not a directory")
+ }
+ }
+ }
+
+ /** Load the required version of the map data from the backing files */
+ private def loadMap(version: Long): MapType = {
+ if (version <= 0) return new MapType
+ synchronized { loadedMaps.get(version) }.getOrElse {
+ val mapFromFile = readSnapshotFile(version).getOrElse {
+ val prevMap = loadMap(version - 1)
+ val deltaUpdates = readDeltaFile(version)
+ val newMap = new MapType()
+ newMap ++= prevMap
+ newMap.sizeHint(prevMap.size)
+ while (deltaUpdates.hasNext) {
+ deltaUpdates.next match {
+ case ValueAdded(key, value) => newMap.put(key, value)
+ case ValueUpdated(key, value) => newMap.put(key, value)
+ case KeyRemoved(key) => newMap.remove(key)
+ }
+ }
+ newMap
+ }
+ loadedMaps.put(version, mapFromFile)
+ mapFromFile
+ }
+ }
+
+ private def readDeltaFile(version: Long): Iterator[StoreUpdate] = {
+ val fileToRead = deltaFile(version)
+ if (!fs.exists(fileToRead)) {
+ throw new IllegalStateException(
+ s"Cannot read delta file $fileToRead of $this: $fileToRead does not exist")
+ }
+ val deser = serializer.newInstance()
+ var deserStream: DeserializationStream = null
+ deserStream = deser.deserializeStream(fs.open(fileToRead))
+ val iter = deserStream.asIterator.asInstanceOf[Iterator[StoreUpdate]]
+ CompletionIterator[StoreUpdate, Iterator[StoreUpdate]](
+ iter, {
+ deserStream.close()
+ })
+ }
+
+ private def writeSnapshotFile(version: Long, map: MapType): Unit = {
+ val fileToWrite = snapshotFile(version)
+ val ser = serializer.newInstance()
+ var outputStream: SerializationStream = null
+ Utils.tryWithSafeFinally {
+ outputStream = ser.serializeStream(fs.create(fileToWrite, false))
+ outputStream.writeAll(map.iterator)
+ } {
+ if (outputStream != null) outputStream.close()
+ }
+ }
+
+ private def readSnapshotFile(version: Long): Option[MapType] = {
+ val fileToRead = snapshotFile(version)
+ if (!fs.exists(fileToRead)) return None
+
+ val deser = serializer.newInstance()
+ val map = new MapType()
+ var deserStream: DeserializationStream = null
+
+ try {
+ deserStream = deser.deserializeStream(fs.open(fileToRead))
+ val iter = deserStream.asIterator.asInstanceOf[Iterator[(InternalRow, InternalRow)]]
+ while (iter.hasNext) {
+ map += iter.next()
+ }
+ Some(map)
+ } finally {
+ if (deserStream != null) deserStream.close()
+ }
+ }
+
+
+ /** Perform a snapshot of the store to allow delta files to be consolidated */
+ private def doSnapshot(): Unit = {
+ try {
+ val files = fetchFiles()
+ if (files.nonEmpty) {
+ val lastVersion = files.last.version
+ val deltaFilesForLastVersion =
+ filesForVersion(files, lastVersion).filter(_.isSnapshot == false)
+ synchronized { loadedMaps.get(lastVersion) } match {
+ case Some(map) =>
+ if (deltaFilesForLastVersion.size > maxDeltaChainForSnapshots) {
+ writeSnapshotFile(lastVersion, map)
+ }
+ case None =>
+ // The last map is not loaded, probably some other instance is incharge
+ }
+
+ }
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error doing snapshots for $this", e)
+ }
+ }
+
+ /**
+ * Clean up old snapshots and delta files that are not needed any more. It ensures that last
+ * few versions of the store can be recovered from the files, so re-executed RDD operations
+ * can re-apply updates on the past versions of the store.
+ */
+ private[state] def cleanup(): Unit = {
+ try {
+ val files = fetchFiles()
+ if (files.nonEmpty) {
+ val earliestVersionToRetain = files.last.version - numBatchesToRetain
+ if (earliestVersionToRetain > 0) {
+ val earliestFileToRetain = filesForVersion(files, earliestVersionToRetain).head
+ synchronized {
+ loadedMaps.keys.filter(_ < earliestVersionToRetain).foreach(loadedMaps.remove)
+ }
+ files.filter(_.version < earliestFileToRetain.version).foreach { f =>
+ fs.delete(f.path, true)
+ }
+ }
+ }
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error cleaning up files for $this", e)
+ }
+ }
+
+ private def filesForVersion(allFiles: Seq[StoreFile], version: Long): Seq[StoreFile] = {
--- End diff --
add docs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198636209
**[Test build #53598 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53598/consoleFull)** for PR 11645 at commit [`2bd6cbd`](https://github.com/apache/spark/commit/2bd6cbd4a75ed4df752b323d35d3a86eaf74b241).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197068487
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53241/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198622506
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53594/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56919238
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{Partition, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.{SerializableConfiguration, Utils}
+
+/**
+ * An RDD that allows computations to be executed against [[StateStore]]s. It
+ * uses the [[StateStoreCoordinator]] to use the locations of loaded state stores as
+ * preferred locations.
+ */
+class StateStoreRDD[T: ClassTag, U: ClassTag](
+ dataRDD: RDD[T],
+ storeUpdateFunction: (StateStore, Iterator[T]) => Iterator[U],
+ checkpointLocation: String,
+ operatorId: Long,
+ storeVersion: Long,
+ keySchema: StructType,
+ valueSchema: StructType,
+ @transient private val storeCoordinator: Option[StateStoreCoordinatorRef])
+ extends RDD[U](dataRDD) {
+
+ // A Hadoop Configuration can be about 10 KB, which is pretty big, so broadcast it
+ private val confBroadcast = dataRDD.context.broadcast(
--- End diff --
Would it be okay if I do this optimization in a future PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57030569
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
--- End diff --
Its still there?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199525087
**[Test build #53714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53714/consoleFull)** for PR 11645 at commit [`32c0139`](https://github.com/apache/spark/commit/32c013918677cb2c93385fbfa6a38b4b8594bea1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56375631
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala ---
@@ -0,0 +1,471 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+
+import scala.collection.mutable
+import scala.util.Random
+
+import org.apache.hadoop.fs.Path
+import org.scalatest.{BeforeAndAfter, PrivateMethodTester}
+import org.scalatest.concurrent.Eventually._
+import org.scalatest.time.SpanSugar._
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+class StateStoreSuite extends SparkFunSuite with BeforeAndAfter with PrivateMethodTester {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ private val tempDir = Utils.createTempDir().toString
+
+ after {
+ StateStore.stop()
+ }
+
+ test("update, remove, commit, and all data iterator") {
+ val provider = newStoreProvider()
+
+ // Verify state before starting a new set of updates
+ assert(provider.latestIterator().isEmpty)
+
+ val store = provider.getStore(0)
+ assert(!store.hasCommitted)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+
+ // Verify state after updating
+ update(store, "a", 1)
+ intercept[IllegalStateException] {
+ store.iterator()
+ }
+ intercept[IllegalStateException] {
+ store.updates()
+ }
+ assert(provider.latestIterator().isEmpty)
+
+ // Make updates, commit and then verify state
+ update(store, "b", 2)
+ update(store, "aa", 3)
+ remove(store, _.startsWith("a"))
+ assert(store.commit() === 1)
+
+ assert(store.hasCommitted)
+ assert(unwrapToSet(store.iterator()) === Set("b" -> 2))
+ assert(unwrapToSet(provider.latestIterator()) === Set("b" -> 2))
+ assert(fileExists(provider, version = 1, isSnapshot = false))
+ assert(getDataFromFiles(provider) === Set("b" -> 2))
+
+ // Trying to get newer versions should fail
+ intercept[Exception] {
+ provider.getStore(2)
+ }
+ intercept[Exception] {
+ getDataFromFiles(provider, 2)
+ }
+
+ // New updates to the reloaded store with new version, and does not change old version
+ val reloadedStore = new HDFSBackedStateStoreProvider(store.id, provider.directory).getStore(1)
+ update(reloadedStore, "c", 4)
+ assert(reloadedStore.commit() === 2)
+ assert(unwrapToSet(reloadedStore.iterator()) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider) === Set("b" -> 2, "c" -> 4))
+ assert(getDataFromFiles(provider, version = 1) === Set("b" -> 2))
+ assert(getDataFromFiles(provider, version = 2) === Set("b" -> 2, "c" -> 4))
+ }
+
+ test("updates iterator with all combos of updates and removes") {
+ val provider = newStoreProvider()
+ var currentVersion: Int = 0
+ def withStore(body: StateStore => Unit): Unit = {
+ val store = provider.getStore(currentVersion)
+ body(store)
+ currentVersion += 1
+ }
+
+ // New data should be seen in updates as value added, even if they had multiple updates
+ withStore { store =>
+ update(store, "a", 1)
+ update(store, "aa", 1)
+ update(store, "aa", 2)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Added("a", 1), Added("aa", 2)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1, "aa" -> 2))
+ }
+
+ // Multiple updates to same key should be collapsed in the updates as a single value update
+ // Keys that have not been updated should not appear in the updates
+ withStore { store =>
+ update(store, "a", 4)
+ update(store, "a", 6)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 6)))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Keys added, updated and finally removed before commit should not appear in updates
+ withStore { store =>
+ update(store, "b", 4) // Added, finally removed
+ update(store, "bb", 5) // Added, updated, finally removed
+ update(store, "bb", 6)
+ remove(store, _.startsWith("b"))
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set.empty)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 6, "aa" -> 2))
+ }
+
+ // Removed data should be seen in updates as a key removed
+ // Removed, but re-added data should be seen in updates as a value update
+ withStore { store =>
+ remove(store, _.startsWith("a"))
+ update(store, "a", 10)
+ store.commit()
+ assert(unwrapUpdates(store.updates()) === Set(Updated("a", 10), Removed("aa")))
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 10))
+ }
+ }
+
+ test("cancel") {
+ val provider = newStoreProvider()
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ store.commit()
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ // cancelUpdates should not change the data in the files
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ store1.cancel()
+ assert(getDataFromFiles(provider) === Set("a" -> 1))
+ }
+
+ test("getStore with unexpected versions") {
+ val provider = newStoreProvider()
+
+ intercept[IllegalArgumentException] {
+ provider.getStore(-1)
+ }
+
+ // Prepare some data in the stoer
+ val store = provider.getStore(0)
+ update(store, "a", 1)
+ assert(store.commit() === 1)
+ assert(unwrapToSet(store.iterator()) === Set("a" -> 1))
+
+ intercept[IllegalStateException] {
+ provider.getStore(2)
+ }
+
+ // Update store version with some data
+ val store1 = provider.getStore(1)
+ update(store1, "b", 1)
+ assert(store1.commit() === 2)
+ assert(unwrapToSet(store1.iterator()) === Set("a" -> 1, "b" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "b" -> 1))
+
+ // Overwrite the version with other data
+ val store2 = provider.getStore(1)
+ update(store2, "c", 1)
+ assert(store2.commit() === 2)
+ assert(unwrapToSet(store2.iterator()) === Set("a" -> 1, "c" -> 1))
+ assert(getDataFromFiles(provider) === Set("a" -> 1, "c" -> 1))
+ }
+
+ test("snapshotting") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ var currentVersion = 0
+ def updateVersionTo(targetVersion: Int): Unit = {
+ for (i <- currentVersion + 1 to targetVersion) {
+ val store = provider.getStore(currentVersion)
+ update(store, "a", i)
+ store.commit()
+ currentVersion += 1
+ }
+ require(currentVersion === targetVersion)
+ }
+
+ updateVersionTo(2)
+ require(getDataFromFiles(provider) === Set("a" -> 2))
+ provider.manage() // should not generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 2))
+
+ for (i <- 1 to currentVersion) {
+ assert(fileExists(provider, i, isSnapshot = false)) // all delta files present
+ assert(!fileExists(provider, i, isSnapshot = true)) // no snapshot files present
+ }
+
+ // After version 6, snapshotting should generate one snapshot file
+ updateVersionTo(6)
+ require(getDataFromFiles(provider) === Set("a" -> 6), "store not updated correctly")
+ provider.manage() // should generate snapshot files
+ assert(getDataFromFiles(provider) === Set("a" -> 6), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 6))
+
+ val snapshotVersion = (0 to 6).find(version => fileExists(provider, version, isSnapshot = true))
+ assert(snapshotVersion.nonEmpty, "snapshot file not generated")
+
+ // After version 20, snapshotting should generate newer snapshot files
+ updateVersionTo(20)
+ require(getDataFromFiles(provider) === Set("a" -> 20), "store not updated correctly")
+ provider.manage() // do snapshot
+ assert(getDataFromFiles(provider) === Set("a" -> 20), "snapshotting messed up the data")
+ assert(getDataFromFiles(provider) === Set("a" -> 20))
+
+ val latestSnapshotVersion = (0 to 20).filter(version =>
+ fileExists(provider, version, isSnapshot = true)).lastOption
+ assert(latestSnapshotVersion.nonEmpty, "no snapshot file found")
+ assert(latestSnapshotVersion.get > snapshotVersion.get, "newer snapshot not generated")
+
+ }
+
+ test("cleaning") {
+ val provider = newStoreProvider(maxDeltaChainForSnapshots = 5)
+
+ for (i <- 1 to 20) {
+ val store = provider.getStore(i - 1)
+ update(store, "a", i)
+ store.commit()
+ provider.manage() // do cleanup
+ }
+ require(
+ unwrapToSet(provider.latestIterator()) === Set("a" -> 20),
+ "store not updated correctly")
+
+ assert(!fileExists(provider, version = 1, isSnapshot = false)) // first file should be deleted
+
+ // last couple of versions should be retrievable
+ assert(getDataFromFiles(provider, 20) === Set("a" -> 20))
+ assert(getDataFromFiles(provider, 19) === Set("a" -> 19))
+ }
+
+ test("StateStore.get") {
+ quietly {
+ val dir = Utils.createDirectory(tempDir, Random.nextString(5)).toString
+ val storeId = StateStoreId(0, 0)
+
+ // Verify that trying to get incorrect versions throw errors
+ intercept[IllegalArgumentException] {
+ StateStore.get(storeId, dir, -1)
+ }
+ assert(!StateStore.isLoaded(storeId)) // version -1 should not attempt to load the store
+
+ intercept[IllegalStateException] {
+ StateStore.get(storeId, dir, 1)
+ }
+
+ // Increase version of the store
+ val store0 = StateStore.get(storeId, dir, 0)
+ assert(store0.version === 0)
+ update(store0, "a", 1)
+ store0.commit()
+
+ assert(StateStore.get(storeId, dir, 1).version == 1)
+ assert(StateStore.get(storeId, dir, 0).version == 0)
+
+ // Verify that you can remove the store and still reload and use it
+ StateStore.remove(storeId)
+ assert(!StateStore.isLoaded(storeId))
+
+ val store1 = StateStore.get(storeId, dir, 1)
+ assert(StateStore.isLoaded(storeId))
+ update(store1, "a", 2)
+ assert(store1.commit() === 2)
+ assert(unwrapToSet(store1.iterator()) === Set("a" -> 2))
+ }
+ }
+
+ test("background management") {
+ val conf = new SparkConf()
+ .setMaster("local")
+ .setAppName("test")
+ .set("spark.sql.streaming.stateStore.managementInterval", "10ms")
+ val opId = 0
+ val storeId = StateStoreId(opId, 0)
+ val dir = Utils.createDirectory(tempDir, Random.nextString(5)).toString
+ val provider = new HDFSBackedStateStoreProvider(storeId, dir)
+
+ quietly {
+ withSpark(new SparkContext(conf)) { sc =>
+ withCoordinator(sc) { coordinator =>
+ for (i <- 1 to 20) {
+ val store = StateStore.get(storeId, dir, i - 1)
+ update(store, "a", i)
+ store.commit()
+ }
+
+ // Background management should clean up and generate snapshots
+ eventually(timeout(4 seconds)) {
+ // Earliest delta file should get cleaned up
+ assert(!fileExists(provider, 1, isSnapshot = false), "earliest file not deleted")
+
+ // Some snapshots should have been generated
+ val snapshotVersions = (0 to 20).filter { version =>
+ fileExists(provider, version, isSnapshot = true)
+ }
+ assert(snapshotVersions.nonEmpty, "no snapshot file found")
+ }
+
+ // If driver decides to deactivate all instances of the store, then this instance
+ // should be unloaded
+ coordinator.deactivateInstances(Set(opId))
+ eventually(timeout(4 seconds)) {
+ assert(!StateStore.isLoaded(storeId))
+ }
+
+ // Reload the store and verify
+ StateStore.get(storeId, dir, 20)
+ assert(StateStore.isLoaded(storeId))
+
+ // If some other executor loads the store, then this instance should be unloaded
+ coordinator.reportActiveInstance(storeId, "other-host", "other-exec")
+ eventually(timeout(4 seconds)) {
+ assert(!StateStore.isLoaded(storeId))
+ }
+
+ // Reload the store and verify
+ StateStore.get(storeId, dir, 20)
+ assert(StateStore.isLoaded(storeId))
+ }
+ }
+
+ // Verify if instance is unloaded if SparkContext is stopped
+ eventually(timeout(4 seconds)) {
+ assert(!StateStore.isLoaded(storeId))
+ }
+ }
+ }
+
+ def getDataFromFiles(
+ provider: HDFSBackedStateStoreProvider,
+ version: Int = -1): Set[(String, Int)] = {
+ val reloadedProvider = new HDFSBackedStateStoreProvider(provider.id, provider.directory)
+ if (version < 0) {
+ reloadedProvider.latestIterator().map(unwrapKeyValue).toSet
+ } else {
+ reloadedProvider.iterator(version).map(unwrapKeyValue).toSet
+ }
+ }
+
+ def assertMap(
+ testMapOption: Option[MapType],
+ expectedMap: Map[String, Int]): Unit = {
+ assert(testMapOption.nonEmpty, "no map present")
+ val convertedMap = testMapOption.get.map(unwrapKeyValue)
+ assert(convertedMap === expectedMap)
+ }
+
+ def fileExists(
+ provider: HDFSBackedStateStoreProvider,
+ version: Long,
+ isSnapshot: Boolean): Boolean = {
+ val method = PrivateMethod[Path]('baseDir)
+ val basePath = provider invokePrivate method()
+ val fileName = if (isSnapshot) s"$version.snapshot" else s"$version.delta"
+ val filePath = new File(basePath.toString, fileName)
+ filePath.exists
+ }
+
+ def storeLoaded(storeId: StateStoreId): Boolean = {
+ val method = PrivateMethod[mutable.HashMap[StateStoreId, StateStore]]('loadedStores)
+ val loadedStores = StateStore invokePrivate method()
+ loadedStores.contains(storeId)
+ }
+
+ def unloadStore(storeId: StateStoreId): Boolean = {
+ val method = PrivateMethod('remove)
+ StateStore invokePrivate method(storeId)
+ }
+
+ def newStoreProvider(
+ opId: Long = Random.nextLong,
+ partition: Int = 0,
+ maxDeltaChainForSnapshots: Int = 10
+ ): HDFSBackedStateStoreProvider = {
+ new HDFSBackedStateStoreProvider(
+ StateStoreId(opId, partition),
+ Utils.createDirectory(tempDir, Random.nextString(5)).toString,
+ maxDeltaChainForSnapshots = maxDeltaChainForSnapshots)
+ }
+
+ def remove(store: StateStore, condition: String => Boolean): Unit = {
+ store.remove(row => condition(unwrapKey(row)))
+ }
+
+ private def update(store: StateStore, key: String, value: Int): Unit = {
+ store.update(wrapKey(key), _ => wrapValue(value))
+ }
+}
+
+private[state] object StateStoreSuite {
+
+ /** Trait and classes mirroring [[StoreUpdate]] for testing store updates iterator */
+ trait TestUpdate
+ case class Added(key: String, value: Int) extends TestUpdate
+ case class Updated(key: String, value: Int) extends TestUpdate
+ case class Removed(key: String) extends TestUpdate
+
+ def wrapValue(i: Int): InternalRow = {
+ new GenericInternalRow(Array[Any](i))
--- End diff --
nit: I think this can just be `new GenericInternalRow(i)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196494841
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196494846
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53085/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199404380
**[Test build #53690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53690/consoleFull)** for PR 11645 at commit [`25afe31`](https://github.com/apache/spark/commit/25afe31080451c16e74bd17bf9f81e5b8aab55f5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200122345
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53845/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200103658
Looks good overall. Just some bits
On Tue, Mar 22, 2016 at 5:53 PM Apache Spark QA <no...@github.com>
wrote:
> *Test build #53857 has started
> <https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53857/consoleFull>*
> for PR 11645 at commit 70cc7b1
> <https://github.com/apache/spark/commit/70cc7b11c0d502d4a741ccdcab5e7aa006b5b311>
> .
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/11645#issuecomment-200097875>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199965009
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53805/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57094742
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ storeConf: StateStoreConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ private val allUpdates = new java.util.HashMap[UnsafeRow, StoreUpdate]()
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ override def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = Option(mapToUpdate.get(key))
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ writeToDeltaFile(tempDeltaFileStream, ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: UnsafeRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keySet().iterator()
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ keyIter.remove()
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ writeToDeltaFile(tempDeltaFileStream, KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ finalizeDeltaFile(tempDeltaFileStream)
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ logInfo(s"Committed version $newVersion for $this")
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ logInfo("Canceled ")
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[(UnsafeRow, UnsafeRow)] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.values().asScala.toIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ val time = System.nanoTime()
--- End diff --
Removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917147
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
--- End diff --
What is `prepareForUpdates`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197086170
**[Test build #53250 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53250/consoleFull)** for PR 11645 at commit [`b8b4632`](https://github.com/apache/spark/commit/b8b4632b5ed9be181456da4f53156a0cbebebaf3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200089642
**[Test build #53850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53850/consoleFull)** for PR 11645 at commit [`313d8be`](https://github.com/apache/spark/commit/313d8be0aa2f70762258cdcb4092d48f6ae543c2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200129728
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53851/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199979999
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198627615
**[Test build #53597 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53597/consoleFull)** for PR 11645 at commit [`7666026`](https://github.com/apache/spark/commit/7666026526f68ecc73c3535ce74141bb423272e5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917182
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def commit(): Long
+
+ /** Cancel all the updates that have been made to the store. */
+ def cancel(): Unit
+
+ /**
+ * Iterator of store data after a set of updates have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def iterator(): Iterator[(UnsafeRow, UnsafeRow)]
+
+ /**
+ * Iterator of the updates that have been committed.
+ * This can be called only after commitUpdates() has been called in the current thread.
+ */
+ def updates(): Iterator[StoreUpdate]
+
+ /**
+ * Whether all updates have been committed
+ */
+ def hasCommitted: Boolean
+}
+
+
+trait StateStoreProvider {
+
+ /** Get the store with the existing version. */
+ def getStore(version: Long): StateStore
+
+ /** Optional method for providers to allow for background management */
+ def doMaintenance(): Unit = { }
+}
+
+sealed trait StoreUpdate
+case class ValueAdded(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class ValueUpdated(key: UnsafeRow, value: UnsafeRow) extends StoreUpdate
+case class KeyRemoved(key: UnsafeRow) extends StoreUpdate
+
+
+/**
+ * Companion object to [[StateStore]] that provides helper methods to create and retrieve stores
+ * by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null),
+ * it also runs a periodic background tasks to do maintenance on the loaded stores. For each
+ * store, tt uses the [[StateStoreCoordinator]] to ensure whether the current loaded instance of
+ * the store is the active instance. Accordingly, it either keeps it loaded and performance
--- End diff --
fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198636674
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53598/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198596578
**[Test build #53590 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53590/consoleFull)** for PR 11645 at commit [`3fe34e9`](https://github.com/apache/spark/commit/3fe34e90bf7820c73ede7901422a48b83c954f27).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200051431
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198628446
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53597/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199983550
**[Test build #53793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53793/consoleFull)** for PR 11645 at commit [`24fb325`](https://github.com/apache/spark/commit/24fb325a44fb63cb7021d10b44f516a2d40d02c6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195175721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52897/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198628649
**[Test build #53598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53598/consoleFull)** for PR 11645 at commit [`2bd6cbd`](https://github.com/apache/spark/commit/2bd6cbd4a75ed4df752b323d35d3a86eaf74b241).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56918759
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{Partition, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.{SerializableConfiguration, Utils}
+
+/**
+ * An RDD that allows computations to be executed against [[StateStore]]s. It
+ * uses the [[StateStoreCoordinator]] to use the locations of loaded state stores as
+ * preferred locations.
+ */
+class StateStoreRDD[T: ClassTag, U: ClassTag](
+ dataRDD: RDD[T],
+ storeUpdateFunction: (StateStore, Iterator[T]) => Iterator[U],
+ checkpointLocation: String,
+ operatorId: Long,
+ storeVersion: Long,
+ keySchema: StructType,
+ valueSchema: StructType,
+ @transient private val storeCoordinator: Option[StateStoreCoordinatorRef])
+ extends RDD[U](dataRDD) {
+
+ // A Hadoop Configuration can be about 10 KB, which is pretty big, so broadcast it
+ private val confBroadcast = dataRDD.context.broadcast(
--- End diff --
We probably want to broadcast this once if possible (instead of for every batch).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917703
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(rootLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * This can be called only after prepareForUpdates() has been called in the same thread.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ * This can be called only after prepareForUpdates() has been called in the current thread.
--- End diff --
Removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56918528
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,597 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.serializer.KryoSerializer
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
--- End diff --
Good point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199547439
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53729/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56376064
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDDSuite.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+import java.nio.file.Files
+
+import scala.util.Random
+
+import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.rdd.RDD
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.util.Utils
+
+class StateStoreRDDSuite extends SparkFunSuite with BeforeAndAfter with BeforeAndAfterAll {
+
+ private val conf = new SparkConf().setMaster("local").setAppName(this.getClass.getCanonicalName)
+ private var tempDir = Files.createTempDirectory("StateStoreRDDSuite").toString
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ after {
+ StateStore.stop()
+ }
+
+ override def afterAll(): Unit = {
+ super.afterAll()
+ Utils.deleteRecursively(new File(tempDir))
+ }
+
+ test("versioning and immutability") {
+ quietly {
+ withSpark(new SparkContext(conf)) { sc =>
+ val path = Utils.createDirectory(tempDir, Random.nextString(10)).toString
--- End diff --
createTempDir
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199980008
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53794/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56212504
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+class HDFSBackedStateStoreProvider(
--- End diff --
note to self: private[state]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197128812
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200090635
**[Test build #53851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53851/consoleFull)** for PR 11645 at commit [`819ca17`](https://github.com/apache/spark/commit/819ca17412b60c6afeaa7ac0624abf80ebd422b9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199578057
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53731/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56917883
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,597 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.serializer.KryoSerializer
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
--- End diff --
Perhaps note that you are not allowed to modify objects after they have been added or modify objects that are retrieved or it might affect old versions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198622448
**[Test build #53594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53594/consoleFull)** for PR 11645 at commit [`9fb9c43`](https://github.com/apache/spark/commit/9fb9c43b0d8ebef0b437226698263f058c04cf18).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200088997
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53810/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56227307
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends StateStoreProvider with Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+
+ class HDFSBackedStateStore( val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ trait STATE
--- End diff --
add doc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199539885
**[Test build #53728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53728/consoleFull)** for PR 11645 at commit [`534ad48`](https://github.com/apache/spark/commit/534ad483ed021f0e707cc27ae8adadbd2b585d83).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196733125
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197101216
**[Test build #53258 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53258/consoleFull)** for PR 11645 at commit [`76dd988`](https://github.com/apache/spark/commit/76dd988fcac508609be3f32754284bc07524e481).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196219639
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53064/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199639696
**[Test build #53743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53743/consoleFull)** for PR 11645 at commit [`f4f3838`](https://github.com/apache/spark/commit/f4f383803e45aaea182b09c4e3c4b64415b9461e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57041029
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+
+
+/** Unique identifier for a [[StateStore]] */
+case class StateStoreId(checkpointLocation: String, operatorId: Long, partitionId: Int)
+
+/**
+ * Base trait for a versioned key-value store used for streaming aggregations
+ */
+trait StateStore {
+
+ /** Unique identifier of the store */
+ def id: StateStoreId
+
+ /** Version of the data in this store before committing updates. */
+ def version: Long
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit
+
+ /**
+ * Remove keys that match the following condition.
+ */
+ def remove(condition: UnsafeRow => Boolean): Unit
+
+ /**
+ * Commit all the updates that have been made to the store.
+ */
+ def commit(): Long
--- End diff --
nit: the meaning of the return value
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199571108
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53728/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198622505
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r55794512
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ---
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.util.{Timer, TimerTask}
+
+import scala.collection.mutable
+import scala.reflect.ClassTag
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf, SparkEnv}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, RpcUtils, Utils}
+
+case class StateStoreId(operatorId: Long, partitionId: Int)
+
+private[state] object StateStore extends Logging {
+
+ sealed trait Update
+ case class ValueUpdated(key: InternalRow, value: InternalRow) extends Update
+ case class KeyRemoved(key: InternalRow) extends Update
+
+ private val loadedStores = new mutable.HashMap[StateStoreId, StateStore]()
+ private val managementTimer = new Timer("StateStore Timer", true)
+ @volatile private var managementTask: TimerTask = null
+
+ def get(storeId: StateStoreId, directory: String): StateStore = {
+ val store = loadedStores.synchronized {
+ startIfNeeded()
+ loadedStores.getOrElseUpdate(storeId, new StateStore(storeId, directory))
+ }
+ reportActiveInstance(storeId)
+ store
+ }
+
+ def clearAll(): Unit = loadedStores.synchronized {
+ loadedStores.clear()
+ if (managementTask != null) {
+ managementTask.cancel()
+ managementTask = null
+ }
+ }
+
+ private def remove(storeId: StateStoreId): Unit = {
+ loadedStores.remove(storeId)
+ }
+
+ private def reportActiveInstance(storeId: StateStoreId): Unit = {
+ val host = SparkEnv.get.blockManager.blockManagerId.host
+ val executorId = SparkEnv.get.blockManager.blockManagerId.executorId
+ askCoordinator[Boolean](ReportActiveInstance(storeId, host, executorId))
+ }
+
+ private def verifyIfInstanceActive(storeId: StateStoreId): Boolean = {
+ val executorId = SparkEnv.get.blockManager.blockManagerId.executorId
+ askCoordinator[Boolean](VerifyIfInstanceActive(storeId, executorId)).getOrElse(false)
+ }
+
+ private def askCoordinator[T: ClassTag](message: StateStoreCoordinatorMessage): Option[T] = {
+ try {
+ val env = SparkEnv.get
+ if (env != null) {
+ val coordinatorRef = RpcUtils.makeDriverRef("StateStoreCoordinator", env.conf, env.rpcEnv)
+ Some(coordinatorRef.askWithRetry[T](message))
+ } else {
+ None
+ }
+ } catch {
+ case NonFatal(e) =>
+ clearAll()
+ None
+ }
+ }
+
+ private def startIfNeeded(): Unit = loadedStores.synchronized {
+ if (managementTask == null) {
+ managementTask = new TimerTask {
+ override def run(): Unit = { manageFiles() }
+ }
+ managementTimer.schedule(managementTask, 10000, 10000)
+ }
+ }
+
+ private def manageFiles(): Unit = {
+ loadedStores.synchronized { loadedStores.values.toSeq }.foreach { store =>
+ try {
+ store.manageFiles()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up store ${store.id}")
+ }
+ }
+ }
+}
+
+private[sql] class StateStore(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir = new Path(directory, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(new Configuration())
+ private val serializer = new KryoSerializer(new SparkConf)
+
+ @volatile private var uncommittedDelta: UncommittedUpdates = null
+
+ initialize()
+
+ /**
+ * Prepare for updates to create a new `version` of the map. The store ensure that updates
+ * are made on the `version - 1` of the store data. If `version` already exists, it will
+ * be overwritten when the updates are committed.
+ */
+ private[state] def prepareForUpdates(version: Long): Unit = synchronized {
+ require(version >= 0)
+ if (uncommittedDelta != null) {
+ cancelUpdates()
+ }
+ val newMap = new MapType()
+ if (version > 0) {
+ val oldMap = loadMap(version - 1)
+ newMap ++= oldMap
+ }
+ uncommittedDelta = new UncommittedUpdates(version, newMap)
+ }
+
--- End diff --
Missing `newVersion()`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195175716
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-196241010
**[Test build #53070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53070/consoleFull)** for PR 11645 at commit [`a0ba498`](https://github.com/apache/spark/commit/a0ba498185286a921ef303dbc087dad5d226fa41).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198621842
**[Test build #53593 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53593/consoleFull)** for PR 11645 at commit [`8cb0da8`](https://github.com/apache/spark/commit/8cb0da8a70e415297f7ee1d25d25eae1cc095b53).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199538081
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53718/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197113151
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197671953
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11645
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198628434
**[Test build #53597 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53597/consoleFull)** for PR 11645 at commit [`7666026`](https://github.com/apache/spark/commit/7666026526f68ecc73c3535ce74141bb423272e5).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200133639
**[Test build #53850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53850/consoleFull)** for PR 11645 at commit [`313d8be`](https://github.com/apache/spark/commit/313d8be0aa2f70762258cdcb4092d48f6ae543c2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57074371
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ storeConf: StateStoreConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ private val allUpdates = new java.util.HashMap[UnsafeRow, StoreUpdate]()
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ override def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = Option(mapToUpdate.get(key))
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ writeToDeltaFile(tempDeltaFileStream, ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: UnsafeRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keySet().iterator()
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ keyIter.remove()
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ writeToDeltaFile(tempDeltaFileStream, KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ finalizeDeltaFile(tempDeltaFileStream)
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ logInfo(s"Committed version $newVersion for $this")
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ logInfo("Canceled ")
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[(UnsafeRow, UnsafeRow)] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.values().asScala.toIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ val time = System.nanoTime()
+ newMap.putAll(loadMap(version))
+ }
+ val store = new HDFSBackedStateStore(version, newMap)
+ logInfo(s"Retrieved version $version of $this for update")
+ store
+ }
+
+ /** Do maintenance backing data files, including creating snapshots and cleaning up old files */
+ override def doMaintenance(): Unit = {
+ try {
+ doSnapshot()
+ cleanup()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up $this")
+ }
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ /* Internal classes and methods */
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir =
+ new Path(id.checkpointLocation, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(hadoopConf)
+ private val sparkConf = Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
+
+ initialize()
+
+ private case class StoreFile(version: Long, path: Path, isSnapshot: Boolean)
+
+ /** Commit a set of updates to the store with the given new version */
+ private def commitUpdates(newVersion: Long, map: MapType, tempDeltaFile: Path): Path = {
+ synchronized {
+ val finalDeltaFile = deltaFile(newVersion)
+ fs.rename(tempDeltaFile, finalDeltaFile)
+ loadedMaps.put(newVersion, map)
+ finalDeltaFile
+ }
+ }
+
+ /**
+ * Get iterator of all the data of the latest version of the store.
+ * Note that this will look up the files to determined the latest known version.
+ */
+ private[state] def latestIterator(): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ val versionsInFiles = fetchFiles().map(_.version).toSet
+ val versionsLoaded = loadedMaps.keySet
+ val allKnownVersions = versionsInFiles ++ versionsLoaded
+ if (allKnownVersions.nonEmpty) {
+ loadMap(allKnownVersions.max).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ } else Iterator.empty
+ }
+
+ /** Get iterator of a specific version of the store */
+ private[state] def iterator(version: Long): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ loadMap(version).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ }
+
+ /** Initialize the store provider */
+ private def initialize(): Unit = {
+ if (!fs.exists(baseDir)) {
+ fs.mkdirs(baseDir)
+ } else {
+ if (!fs.isDirectory(baseDir)) {
+ throw new IllegalStateException(
+ s"Cannot use ${id.checkpointLocation} for storing state data for $this as" +
+ s"$baseDir already exists and is not a directory")
+ }
+ }
+ }
+
+ /** Load the required version of the map data from the backing files */
+ private def loadMap(version: Long): MapType = {
+ if (version <= 0) return new MapType
+ synchronized { loadedMaps.get(version) }.getOrElse {
+ val mapFromFile = readSnapshotFile(version).getOrElse {
+ val prevMap = loadMap(version - 1)
+ val newMap = new MapType(prevMap)
+ newMap.putAll(prevMap)
+ updateFromDeltaFile(version, newMap)
+ newMap
+ }
+ loadedMaps.put(version, mapFromFile)
+ mapFromFile
+ }
+ }
+
+ private def writeToDeltaFile(output: DataOutputStream, update: StoreUpdate): Unit = {
+
+ def writeUpdate(key: UnsafeRow, value: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ val valueBytes = value.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+
+ def writeRemove(key: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(-1)
+ }
+
+ update match {
+ case ValueAdded(key, value) =>
+ writeUpdate(key, value)
+ case ValueUpdated(key, value) =>
+ writeUpdate(key, value)
+ case KeyRemoved(key) =>
+ writeRemove(key)
+ }
+ }
+
+ private def finalizeDeltaFile(output: DataOutputStream): Unit = {
+ output.writeInt(-1) // Write this magic number to signify end of file
+ output.close()
+ }
+
+ private def updateFromDeltaFile(version: Long, map: MapType): Unit = {
+ val fileToRead = deltaFile(version)
+ if (!fs.exists(fileToRead)) {
+ throw new IllegalStateException(
+ s"Error reading delta file $fileToRead of $this: $fileToRead does not exist")
+ }
+ var input: DataInputStream = null
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while(!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading delta file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ map.remove(keyRow)
+ } else {
+ val valueRowBuffer = new Array[Byte](valueSize)
+ ByteStreams.readFully(input, valueRowBuffer, 0, valueSize)
+ val valueRow = new UnsafeRow(valueSchema.fields.length)
+ valueRow.pointTo(valueRowBuffer, valueSize)
+ map.put(keyRow, valueRow)
+ }
+ }
+ }
+ } finally {
+ if (input != null) input.close()
+ }
+ logInfo(s"Read delta file for version $version of $this from $fileToRead")
+ }
+
+ private def writeSnapshotFile(version: Long, map: MapType): Unit = {
+ val fileToWrite = snapshotFile(version)
+ var output: DataOutputStream = null
+ Utils.tryWithSafeFinally {
+ output = compressStream(fs.create(fileToWrite, false))
+ val iter = map.entrySet().iterator()
+ while(iter.hasNext) {
+ val entry = iter.next()
+ val keyBytes = entry.getKey.getBytes()
+ val valueBytes = entry.getValue.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+ output.writeInt(-1)
+ } {
+ if (output != null) output.close()
+ }
+ logInfo(s"Written snapshot file for version $version of $this at $fileToWrite")
+ }
+
+ private def readSnapshotFile(version: Long): Option[MapType] = {
+ val fileToRead = snapshotFile(version)
+ if (!fs.exists(fileToRead)) return None
+
+ val map = new MapType()
+ var input: DataInputStream = null
+
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while (!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
--- End diff --
nit: Using `IOException` would be better
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197671956
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53388/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198627158
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53596/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56387171
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDDSuite.scala ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.File
+import java.nio.file.Files
+
+import scala.util.Random
+
+import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
+import org.apache.spark.LocalSparkContext._
+import org.apache.spark.rdd.RDD
+import org.apache.spark.scheduler.ExecutorCacheTaskLocation
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.util.Utils
+
+class StateStoreRDDSuite extends SparkFunSuite with BeforeAndAfter with BeforeAndAfterAll {
+
+ private val conf = new SparkConf().setMaster("local").setAppName(this.getClass.getCanonicalName)
+ private var tempDir = Files.createTempDirectory("StateStoreRDDSuite").toString
+
+ import StateStoreCoordinatorSuite._
+ import StateStoreSuite._
+
+ after {
+ StateStore.stop()
+ }
+
+ override def afterAll(): Unit = {
+ super.afterAll()
+ Utils.deleteRecursively(new File(tempDir))
+ }
+
+ test("versioning and immutability") {
+ quietly {
+ withSpark(new SparkContext(conf)) { sc =>
+ val path = Utils.createDirectory(tempDir, Random.nextString(10)).toString
+ val increment = (store: StateStore, iter: Iterator[String]) => {
+ iter.foreach { s =>
+ store.update(
+ wrapKey(s), oldRow => {
+ val oldValue = oldRow.map(unwrapValue).getOrElse(0)
+ wrapValue(oldValue + 1)
+ })
+ }
+ store.commit()
+ store.iterator().map(unwrapKeyValue)
+ }
+ val opId = 0
+ val rdd1 = makeRDD(sc, Seq("a", "b", "a"))
+ .withStateStores(increment, opId, storeVersion = 0, path)
+ assert(rdd1.collect().toSet === Set("a" -> 2, "b" -> 1))
+
+ // Generate next version of stores
+ val rdd2 = makeRDD(sc, Seq("a", "c"))
+ .withStateStores(increment, opId, storeVersion = 1, path)
+ assert(rdd2.collect().toSet === Set("a" -> 3, "b" -> 1, "c" -> 1))
+
+ // Make sure the previous RDD still has the same data.
+ assert(rdd1.collect().toSet === Set("a" -> 2, "b" -> 1))
+ }
+ }
+ }
+
+ test("recovering from files") {
+ quietly {
+ val opId = 0
+ val path = Utils.createDirectory(tempDir, Random.nextString(10)).toString
--- End diff --
Temp dir gets deleted when SparkContext is shutdown, so cannot be used for this test. Rather the root tempDir gets deleted after all the tests in afterAll()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197157899
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53268/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197093878
**[Test build #53255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53255/consoleFull)** for PR 11645 at commit [`b5e2421`](https://github.com/apache/spark/commit/b5e242132ae98342113f27da4824aaf06485e291).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-197671677
**[Test build #53388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53388/consoleFull)** for PR 11645 at commit [`15e1780`](https://github.com/apache/spark/commit/15e178059d86b8284da4c7e095f70d63c1b7acbe).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-199525642
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53714/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r56227343
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.serializer.{DeserializationStream, KryoSerializer, SerializationStream}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.JoinedRow
+import org.apache.spark.util.{CompletionIterator, Utils}
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates must have the same updates.
+ * - Background management of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ val directory: String,
+ numBatchesToRetain: Int = 2,
+ maxDeltaChainForSnapshots: Int = 10
+ ) extends StateStoreProvider with Logging {
+ type MapType = mutable.HashMap[InternalRow, InternalRow]
+
+ import StateStore._
+
+
+ class HDFSBackedStateStore( val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream =
+ serializer.newInstance().serializeStream(fs.create(tempDeltaFile, true))
+ private val allUpdates = new mutable.HashMap[InternalRow, StoreUpdate]
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /** Update the value of a key using the value generated by the update function */
+ override def update(key: InternalRow, updateFunc: Option[InternalRow] => InternalRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = mapToUpdate.get(key)
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+ allUpdates.get(key) match {
+ case Some(ValueAdded(_, _)) =>
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ tempDeltaFileStream.writeObject(ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: InternalRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keysIterator
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ mapToUpdate.remove(key)
+ allUpdates.get(key) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ tempDeltaFileStream.writeObject(KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ tempDeltaFileStream.close()
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[InternalRow] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.valuesIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ newMap ++= loadMap(version)
+ }
+ new HDFSBackedStateStore(version, newMap)
+ }
+
+ override def manage(): Unit = {
--- End diff --
add doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11645#discussion_r57074381
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ---
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{DataInputStream, DataOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.util.Random
+import scala.util.control.NonFatal
+
+import com.google.common.io.ByteStreams
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.{SparkConf, SparkEnv}
+import org.apache.spark.internal.Logging
+import org.apache.spark.io.LZ4CompressionCodec
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.Utils
+
+
+/**
+ * An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed
+ * by files in a HDFS-compatible file system. All updates to the store has to be done in sets
+ * transactionally, and each set of updates increments the store's version. These versions can
+ * be used to re-execute the updates (by retries in RDD operations) on the correct version of
+ * the store, and regenerate the store version.
+ *
+ * Usage:
+ * To update the data in the state store, the following order of operations are needed.
+ *
+ * - val store = StateStore.get(operatorId, partitionId, version) // to get the right store
+ * - store.update(...)
+ * - store.remove(...)
+ * - store.commit() // commits all the updates to made with version number
+ * - store.iterator() // key-value data after last commit as an iterator
+ * - store.updates() // updates made in the last as an iterator
+ *
+ * Fault-tolerance model:
+ * - Every set of updates is written to a delta file before committing.
+ * - The state store is responsible for managing, collapsing and cleaning up of delta files.
+ * - Multiple attempts to commit the same version of updates may overwrite each other.
+ * Consistency guarantees depend on whether multiple attempts have the same updates and
+ * the overwrite semantics of underlying file system.
+ * - Background maintenance of files ensures that last versions of the store is always recoverable
+ * to ensure re-executed RDD operations re-apply updates on the correct past version of the
+ * store.
+ */
+private[state] class HDFSBackedStateStoreProvider(
+ val id: StateStoreId,
+ keySchema: StructType,
+ valueSchema: StructType,
+ storeConf: StateStoreConf,
+ hadoopConf: Configuration
+ ) extends StateStoreProvider with Logging {
+
+ type MapType = java.util.HashMap[UnsafeRow, UnsafeRow]
+
+ /** Implementation of [[StateStore]] API which is backed by a HDFS-compatible file system */
+ class HDFSBackedStateStore(val version: Long, mapToUpdate: MapType)
+ extends StateStore {
+
+ /** Trait and classes representing the internal state of the store */
+ trait STATE
+ case object UPDATING extends STATE
+ case object COMMITTED extends STATE
+ case object CANCELLED extends STATE
+
+ private val newVersion = version + 1
+ private val tempDeltaFile = new Path(baseDir, s"temp-${Random.nextLong}")
+ private val tempDeltaFileStream = compressStream(fs.create(tempDeltaFile, true))
+
+ private val allUpdates = new java.util.HashMap[UnsafeRow, StoreUpdate]()
+
+ @volatile private var state: STATE = UPDATING
+ @volatile private var finalDeltaFile: Path = null
+
+ override def id: StateStoreId = HDFSBackedStateStoreProvider.this.id
+
+ /**
+ * Update the value of a key using the value generated by the update function.
+ * @note Do not mutate the retrieved value row as it will unexpectedly affect the previous
+ * versions of the store data.
+ */
+ override def update(key: UnsafeRow, updateFunc: Option[UnsafeRow] => UnsafeRow): Unit = {
+ verify(state == UPDATING, "Cannot update after already committed or cancelled")
+ val oldValueOption = Option(mapToUpdate.get(key))
+ val value = updateFunc(oldValueOption)
+ mapToUpdate.put(key, value)
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added already, keep it marked as added
+ allUpdates.put(key, ValueAdded(key, value))
+ case Some(ValueUpdated(_, _)) | Some(KeyRemoved(_)) =>
+ // Value existed in prev version and updated/removed, mark it as updated
+ allUpdates.put(key, ValueUpdated(key, value))
+ case None =>
+ // There was no prior update, so mark this as added or updated according to its presence
+ // in previous version.
+ val update =
+ if (oldValueOption.nonEmpty) ValueUpdated(key, value) else ValueAdded(key, value)
+ allUpdates.put(key, update)
+ }
+ writeToDeltaFile(tempDeltaFileStream, ValueUpdated(key, value))
+ }
+
+ /** Remove keys that match the following condition */
+ override def remove(condition: UnsafeRow => Boolean): Unit = {
+ verify(state == UPDATING, "Cannot remove after already committed or cancelled")
+ val keyIter = mapToUpdate.keySet().iterator()
+ while (keyIter.hasNext) {
+ val key = keyIter.next
+ if (condition(key)) {
+ keyIter.remove()
+
+ Option(allUpdates.get(key)) match {
+ case Some(ValueUpdated(_, _)) | None =>
+ // Value existed in previous version and maybe was updated, mark removed
+ allUpdates.put(key, KeyRemoved(key))
+ case Some(ValueAdded(_, _)) =>
+ // Value did not exist in previous version and was added, should not appear in updates
+ allUpdates.remove(key)
+ case Some(KeyRemoved(_)) =>
+ // Remove already in update map, no need to change
+ }
+ writeToDeltaFile(tempDeltaFileStream, KeyRemoved(key))
+ }
+ }
+ }
+
+ /** Commit all the updates that have been made to the store. */
+ override def commit(): Long = {
+ verify(state == UPDATING, "Cannot commit again after already committed or cancelled")
+
+ try {
+ finalizeDeltaFile(tempDeltaFileStream)
+ finalDeltaFile = commitUpdates(newVersion, mapToUpdate, tempDeltaFile)
+ state = COMMITTED
+ logInfo(s"Committed version $newVersion for $this")
+ newVersion
+ } catch {
+ case NonFatal(e) =>
+ throw new IllegalStateException(
+ s"Error committing version $newVersion into ${HDFSBackedStateStoreProvider.this}", e)
+ }
+ }
+
+ /** Cancel all the updates made on this store. This store will not be usable any more. */
+ override def cancel(): Unit = {
+ state = CANCELLED
+ if (tempDeltaFileStream != null) {
+ tempDeltaFileStream.close()
+ }
+ if (tempDeltaFile != null && fs.exists(tempDeltaFile)) {
+ fs.delete(tempDeltaFile, true)
+ }
+ logInfo("Canceled ")
+ }
+
+ /**
+ * Get an iterator of all the store data. This can be called only after committing the
+ * updates.
+ */
+ override def iterator(): Iterator[(UnsafeRow, UnsafeRow)] = {
+ verify(state == COMMITTED, "Cannot get iterator of store data before comitting")
+ HDFSBackedStateStoreProvider.this.iterator(newVersion)
+ }
+
+ /**
+ * Get an iterator of all the updates made to the store in the current version.
+ * This can be called only after committing the updates.
+ */
+ override def updates(): Iterator[StoreUpdate] = {
+ verify(state == COMMITTED, "Cannot get iterator of updates before committing")
+ allUpdates.values().asScala.toIterator
+ }
+
+ /**
+ * Whether all updates have been committed
+ */
+ override def hasCommitted: Boolean = {
+ state == COMMITTED
+ }
+ }
+
+ /** Get the state store for making updates to create a new `version` of the store. */
+ override def getStore(version: Long): StateStore = synchronized {
+ require(version >= 0, "Version cannot be less than 0")
+ val newMap = new MapType()
+ if (version > 0) {
+ val time = System.nanoTime()
+ newMap.putAll(loadMap(version))
+ }
+ val store = new HDFSBackedStateStore(version, newMap)
+ logInfo(s"Retrieved version $version of $this for update")
+ store
+ }
+
+ /** Do maintenance backing data files, including creating snapshots and cleaning up old files */
+ override def doMaintenance(): Unit = {
+ try {
+ doSnapshot()
+ cleanup()
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Error performing snapshot and cleaning up $this")
+ }
+ }
+
+ override def toString(): String = {
+ s"StateStore[id = (op=${id.operatorId},part=${id.partitionId}), dir = $baseDir]"
+ }
+
+ /* Internal classes and methods */
+
+ private val loadedMaps = new mutable.HashMap[Long, MapType]
+ private val baseDir =
+ new Path(id.checkpointLocation, s"${id.operatorId}/${id.partitionId.toString}")
+ private val fs = baseDir.getFileSystem(hadoopConf)
+ private val sparkConf = Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
+
+ initialize()
+
+ private case class StoreFile(version: Long, path: Path, isSnapshot: Boolean)
+
+ /** Commit a set of updates to the store with the given new version */
+ private def commitUpdates(newVersion: Long, map: MapType, tempDeltaFile: Path): Path = {
+ synchronized {
+ val finalDeltaFile = deltaFile(newVersion)
+ fs.rename(tempDeltaFile, finalDeltaFile)
+ loadedMaps.put(newVersion, map)
+ finalDeltaFile
+ }
+ }
+
+ /**
+ * Get iterator of all the data of the latest version of the store.
+ * Note that this will look up the files to determined the latest known version.
+ */
+ private[state] def latestIterator(): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ val versionsInFiles = fetchFiles().map(_.version).toSet
+ val versionsLoaded = loadedMaps.keySet
+ val allKnownVersions = versionsInFiles ++ versionsLoaded
+ if (allKnownVersions.nonEmpty) {
+ loadMap(allKnownVersions.max).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ } else Iterator.empty
+ }
+
+ /** Get iterator of a specific version of the store */
+ private[state] def iterator(version: Long): Iterator[(UnsafeRow, UnsafeRow)] = synchronized {
+ loadMap(version).entrySet().iterator().asScala.map { x =>
+ (x.getKey, x.getValue)
+ }
+ }
+
+ /** Initialize the store provider */
+ private def initialize(): Unit = {
+ if (!fs.exists(baseDir)) {
+ fs.mkdirs(baseDir)
+ } else {
+ if (!fs.isDirectory(baseDir)) {
+ throw new IllegalStateException(
+ s"Cannot use ${id.checkpointLocation} for storing state data for $this as" +
+ s"$baseDir already exists and is not a directory")
+ }
+ }
+ }
+
+ /** Load the required version of the map data from the backing files */
+ private def loadMap(version: Long): MapType = {
+ if (version <= 0) return new MapType
+ synchronized { loadedMaps.get(version) }.getOrElse {
+ val mapFromFile = readSnapshotFile(version).getOrElse {
+ val prevMap = loadMap(version - 1)
+ val newMap = new MapType(prevMap)
+ newMap.putAll(prevMap)
+ updateFromDeltaFile(version, newMap)
+ newMap
+ }
+ loadedMaps.put(version, mapFromFile)
+ mapFromFile
+ }
+ }
+
+ private def writeToDeltaFile(output: DataOutputStream, update: StoreUpdate): Unit = {
+
+ def writeUpdate(key: UnsafeRow, value: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ val valueBytes = value.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+
+ def writeRemove(key: UnsafeRow): Unit = {
+ val keyBytes = key.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(-1)
+ }
+
+ update match {
+ case ValueAdded(key, value) =>
+ writeUpdate(key, value)
+ case ValueUpdated(key, value) =>
+ writeUpdate(key, value)
+ case KeyRemoved(key) =>
+ writeRemove(key)
+ }
+ }
+
+ private def finalizeDeltaFile(output: DataOutputStream): Unit = {
+ output.writeInt(-1) // Write this magic number to signify end of file
+ output.close()
+ }
+
+ private def updateFromDeltaFile(version: Long, map: MapType): Unit = {
+ val fileToRead = deltaFile(version)
+ if (!fs.exists(fileToRead)) {
+ throw new IllegalStateException(
+ s"Error reading delta file $fileToRead of $this: $fileToRead does not exist")
+ }
+ var input: DataInputStream = null
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while(!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading delta file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ map.remove(keyRow)
+ } else {
+ val valueRowBuffer = new Array[Byte](valueSize)
+ ByteStreams.readFully(input, valueRowBuffer, 0, valueSize)
+ val valueRow = new UnsafeRow(valueSchema.fields.length)
+ valueRow.pointTo(valueRowBuffer, valueSize)
+ map.put(keyRow, valueRow)
+ }
+ }
+ }
+ } finally {
+ if (input != null) input.close()
+ }
+ logInfo(s"Read delta file for version $version of $this from $fileToRead")
+ }
+
+ private def writeSnapshotFile(version: Long, map: MapType): Unit = {
+ val fileToWrite = snapshotFile(version)
+ var output: DataOutputStream = null
+ Utils.tryWithSafeFinally {
+ output = compressStream(fs.create(fileToWrite, false))
+ val iter = map.entrySet().iterator()
+ while(iter.hasNext) {
+ val entry = iter.next()
+ val keyBytes = entry.getKey.getBytes()
+ val valueBytes = entry.getValue.getBytes()
+ output.writeInt(keyBytes.size)
+ output.write(keyBytes)
+ output.writeInt(valueBytes.size)
+ output.write(valueBytes)
+ }
+ output.writeInt(-1)
+ } {
+ if (output != null) output.close()
+ }
+ logInfo(s"Written snapshot file for version $version of $this at $fileToWrite")
+ }
+
+ private def readSnapshotFile(version: Long): Option[MapType] = {
+ val fileToRead = snapshotFile(version)
+ if (!fs.exists(fileToRead)) return None
+
+ val map = new MapType()
+ var input: DataInputStream = null
+
+ try {
+ input = decompressStream(fs.open(fileToRead))
+ var eof = false
+
+ while (!eof) {
+ val keySize = input.readInt()
+ if (keySize == -1) {
+ eof = true
+ } else if (keySize < 0) {
+ throw new Exception(
+ s"Error reading snapshot file $fileToRead of $this: key size cannot be $keySize")
+ } else {
+ val keyRowBuffer = new Array[Byte](keySize)
+ ByteStreams.readFully(input, keyRowBuffer, 0, keySize)
+
+ val keyRow = new UnsafeRow(keySchema.fields.length)
+ keyRow.pointTo(keyRowBuffer, keySize)
+
+ val valueSize = input.readInt()
+ if (valueSize < 0) {
+ throw new Exception(
--- End diff --
nit: Using `IOException` would be better
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-195175702
**[Test build #52897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52897/consoleFull)** for PR 11645 at commit [`f417bde`](https://github.com/apache/spark/commit/f417bde8bd65572e78125c7aa900221fa0bc4d7c).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` implicit class StateStoreOps[INPUT: ClassTag](dataRDD: RDD[INPUT]) `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-13809][SQL] State store for stream...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-198628443
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13809][SQL] State store for streaming a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11645#issuecomment-200133906
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53850/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org