You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/17 12:51:24 UTC
[GitHub] [spark] bozhang2820 opened a new pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
bozhang2820 opened a new pull request #33763:
URL: https://github.com/apache/spark/pull/33763
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This change creates a new type of Trigger: Trigger.AvailableNow for streaming queries. It is like Trigger.Once, which process all available data then stop the query, but with better scalability since data can be processed in multiple batches instead of one.
To achieve this, this change proposes a new interface `SupportsTriggerAvailableNow`, which is an extension of `SupportsAdmissionControl`. It has one method, `prepareForTriggerAvailableNow`, which will be called at the beginning of streaming queries with Trigger.AvailableNow, to let the source record the offset for the current latest data at the time (a.k.a the target offset for the query). The source should then behave as if there is no new data coming in after the beginning of the query, i.e., the source will not return an offset higher than the target offset when `latestOffset` is called.
This change also updates `FileStreamSource` to be an implementation of `SupportsTriggerAvailableNow`.
For other sources that does not implement `SupportsTriggerAvailableNow`, this change also provides a new class `FakeLatestOffsetSupportsTriggerAvailableNow`, which wraps the sources and makes them support Trigger.AvailableNow, by overriding their `latestOffset` method to always return the latest offset `latestOffset` is first called. It is as if there is no new data coming in from the source after the first `latestOffset` call.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Currently streaming queries with Trigger.Once will always load all of the available data in a single batch. Because of this, the amount of data the queries can process is limited, or Spark driver will run out of memory.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Users will be able to use Trigger.AvailableNow (to process all available data then stop the streaming query) with this change.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900378524
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47059/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-968159530
@HeartSaVioR @viirya @bozhang2820 is there a way to call this from Pyspark? I can't figure it out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908090908
**[Test build #142853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142853/testReport)** for PR 33763 at commit [`d13807a`](https://github.com/apache/spark/commit/d13807ad79bf8ccefdbe07ded7a04614e16a2934).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900798277
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
brkyvz commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695303455
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetMicroBatchStream.scala
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.MicroBatchStream
+
+/**
+ * This class wraps a [[MicroBatchStream]] and makes it supports Trigger.AvailableNow.
+ *
+ * See [[FakeLatestOffsetSupportsTriggerAvailableNow]] for more details.
+ */
+class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
+ extends FakeLatestOffsetSupportsTriggerAvailableNow(source) with MicroBatchStream {
+
+ override def latestOffset(): streaming.Offset = source.latestOffset()
Review comment:
should this be throwing an UnsupportedOperationException? I was confused on why you'd need to override this, and later realized that this wasn't the rate limited version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908090908
**[Test build #142853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142853/testReport)** for PR 33763 at commit [`d13807a`](https://github.com/apache/spark/commit/d13807ad79bf8ccefdbe07ded7a04614e16a2934).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r693701609
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetMicroBatchStream.scala
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.MicroBatchStream
+
+class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
+ extends FakeLatestOffsetSupportsTriggerAvailableNow(source)
+ with MicroBatchStream {
Review comment:
same indentation with `extends` according to style guide. please add a newline below if it breaks readability.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetSource.scala
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.types.StructType
+
+class FakeLatestOffsetSource(source: Source)
+ extends FakeLatestOffsetSupportsTriggerAvailableNow(source)
+ with Source {
Review comment:
Same here.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -47,10 +47,14 @@ class MicroBatchExecution(
triggerClock, plan.outputMode, plan.deleteCheckpointOnStop) {
@volatile protected var sources: Seq[SparkDataStream] = Seq.empty
+ @volatile protected var wrappedSources: MutableMap[
+ SparkDataStream, FakeLatestOffsetSupportsTriggerAvailableNow] = new
+ MutableHashMap[SparkDataStream, FakeLatestOffsetSupportsTriggerAvailableNow]()
private val triggerExecutor = trigger match {
case t: ProcessingTimeTrigger => ProcessingTimeExecutor(t, triggerClock)
- case OneTimeTrigger => OneTimeExecutor()
+ case OneTimeTrigger => OneBatchExecutor()
Review comment:
It'd be better to have consistent name rule between trigger and executor. `MultiBatchExecutor` doesn't represent the semantic from its name.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/TriggerAvailableNowSuite.scala
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import java.io.File
+
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.catalyst.plans.logical.Range
+import org.apache.spark.sql.catalyst.util.stringToFile
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.{ReadLimit, SupportsAdmissionControl}
+import org.apache.spark.sql.execution.streaming.{LongOffset, MemoryStream, Offset, SerializedOffset, Source, StreamingExecutionRelation}
+import org.apache.spark.sql.types.{LongType, StructType}
+
+class TriggerAvailableNowSuite extends FileStreamSourceTest {
+
+ import testImplicits._
+
+ abstract class TestDataFrameProvider {
+ @volatile var currentOffset = 0L
+
+ def toDF: DataFrame
+
+ def incrementAvailableOffset(numNewRows: Int): Unit
+ }
+
+ class TestSource extends TestDataFrameProvider with Source {
+ override def getOffset: Option[Offset] = {
+ if (currentOffset <= 0) None else Some(LongOffset(currentOffset))
+ }
+
+ override def getBatch(start: Option[Offset], end: Offset): DataFrame = {
+ if (currentOffset == 0) currentOffset = getOffsetValue(end)
+ val plan = Range(
+ start.map(getOffsetValue).getOrElse(0L) + 1L, getOffsetValue(end) + 1L, 1, None,
+ isStreaming = true)
+ Dataset.ofRows(spark, plan)
+ }
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ currentOffset += numNewRows
+ }
+
+ override def toDF: DataFrame =
+ Dataset.ofRows(spark, StreamingExecutionRelation(this, spark))
+
+ override def schema: StructType = new StructType().add("value", LongType)
+
+ override def stop(): Unit = {}
+
+ private def getOffsetValue(offset: Offset): Long = {
+ offset match {
+ case s: SerializedOffset => LongOffset(s).offset
+ case l: LongOffset => l.offset
+ case _ => throw new IllegalArgumentException("incorrect offset type: " + offset)
+ }
+ }
+ }
+
+ class TestSourceWithAdmissionControl extends TestSource with SupportsAdmissionControl {
+ override def getDefaultReadLimit: ReadLimit = ReadLimit.maxRows(1) // this will be overridden
+
+ override def latestOffset(startOffset: streaming.Offset, limit: ReadLimit): streaming.Offset = {
+ val currentOffset = getOffset
+ assert(currentOffset.nonEmpty,
+ "the latestOffset should be called after incrementAvailableOffset")
+ currentOffset.get
+ }
+ }
+
+ class TestMicroBatchStream extends TestDataFrameProvider {
+
+ private lazy val memoryStream = MemoryStream[Long](0, spark.sqlContext)
+
+ override def toDF: DataFrame = memoryStream.toDF()
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ for (_ <- 1 to numNewRows) {
+ currentOffset += 1
+ memoryStream.addData(currentOffset)
+ }
+ }
+ }
+
Review comment:
Let's make the test suite be exhaustive; 1) single source which doesn't support Trigger.AvailableNow 2) single source which supports Trigger.AvailableNow.
Latter can be covered by new FileStreamSource test, hence adding test for 1) would be enough.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/TriggerAvailableNowSuite.scala
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import java.io.File
+
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.catalyst.plans.logical.Range
+import org.apache.spark.sql.catalyst.util.stringToFile
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.{ReadLimit, SupportsAdmissionControl}
+import org.apache.spark.sql.execution.streaming.{LongOffset, MemoryStream, Offset, SerializedOffset, Source, StreamingExecutionRelation}
+import org.apache.spark.sql.types.{LongType, StructType}
+
+class TriggerAvailableNowSuite extends FileStreamSourceTest {
+
+ import testImplicits._
+
+ abstract class TestDataFrameProvider {
+ @volatile var currentOffset = 0L
+
+ def toDF: DataFrame
+
+ def incrementAvailableOffset(numNewRows: Int): Unit
+ }
+
+ class TestSource extends TestDataFrameProvider with Source {
+ override def getOffset: Option[Offset] = {
+ if (currentOffset <= 0) None else Some(LongOffset(currentOffset))
+ }
+
+ override def getBatch(start: Option[Offset], end: Offset): DataFrame = {
+ if (currentOffset == 0) currentOffset = getOffsetValue(end)
+ val plan = Range(
+ start.map(getOffsetValue).getOrElse(0L) + 1L, getOffsetValue(end) + 1L, 1, None,
+ isStreaming = true)
+ Dataset.ofRows(spark, plan)
+ }
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ currentOffset += numNewRows
+ }
+
+ override def toDF: DataFrame =
+ Dataset.ofRows(spark, StreamingExecutionRelation(this, spark))
+
+ override def schema: StructType = new StructType().add("value", LongType)
+
+ override def stop(): Unit = {}
+
+ private def getOffsetValue(offset: Offset): Long = {
+ offset match {
+ case s: SerializedOffset => LongOffset(s).offset
+ case l: LongOffset => l.offset
+ case _ => throw new IllegalArgumentException("incorrect offset type: " + offset)
+ }
+ }
+ }
+
+ class TestSourceWithAdmissionControl extends TestSource with SupportsAdmissionControl {
+ override def getDefaultReadLimit: ReadLimit = ReadLimit.maxRows(1) // this will be overridden
+
+ override def latestOffset(startOffset: streaming.Offset, limit: ReadLimit): streaming.Offset = {
+ val currentOffset = getOffset
+ assert(currentOffset.nonEmpty,
+ "the latestOffset should be called after incrementAvailableOffset")
+ currentOffset.get
+ }
+ }
+
+ class TestMicroBatchStream extends TestDataFrameProvider {
+
+ private lazy val memoryStream = MemoryStream[Long](0, spark.sqlContext)
+
+ override def toDF: DataFrame = memoryStream.toDF()
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ for (_ <- 1 to numNewRows) {
+ currentOffset += 1
+ memoryStream.addData(currentOffset)
+ }
+ }
+ }
+
+ Seq(
+ new TestSource,
+ new TestSourceWithAdmissionControl,
+ new TestMicroBatchStream
+ ).foreach { testSource =>
+ test(s"TriggerAvailableNow for multiple sources with ${testSource.getClass}") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ // Set up a query to read text files one at a time
+ val df1 = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ val df2 = testSource.toDF
+
+ def startQuery(): StreamingQuery = {
+ df1.union(df2).writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+
+ testSource.incrementAvailableOffset(3)
+ createFile(7)
+ createFile(8)
+ createFile(9)
+
+ val q = startQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
+ // only one batch has data in both sources, thus counted, see SPARK-24050
Review comment:
(Beyond the scope) it would be ideal if we can revisit and fix it.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
##########
@@ -48,7 +48,11 @@ class FileStreamSource(
override val schema: StructType,
partitionColumns: Seq[String],
metadataPath: String,
- options: Map[String, String]) extends SupportsAdmissionControl with Source with Logging {
+ options: Map[String, String])
+ extends SupportsAdmissionControl
Review comment:
Let's consolidate SupportsAdmissionControl and SupportsTriggerAvailableNow into SupportsTriggerAvailableNow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909045572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r698108415
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class AvailableNowDataStreamWrapper(val delegate: SparkDataStream)
+ extends SparkDataStream with SupportsTriggerAvailableNow with Logging {
+
+ private var fetchedOffset: Option[streaming.Offset] = _
+
+ override def initialOffset(): streaming.Offset = delegate.initialOffset()
+
+ override def deserializeOffset(json: String): streaming.Offset = delegate.deserializeOffset(json)
+
+ override def commit(end: streaming.Offset): Unit = delegate.commit(end)
+
+ override def stop(): Unit = delegate.stop()
+
+ override def prepareForTriggerAvailableNow(): Unit = {}
Review comment:
Thanks for the suggestion! This made me realized that we might need to have `startOffset` as an argument in `prepareForTriggerAvailableNow`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908821421
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47369/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906552528
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142797/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908421005
**[Test build #142861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142861/testReport)** for PR 33763 at commit [`0824910`](https://github.com/apache/spark/commit/08249107813f334cc32596ce1b5001a1e777c318).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-904507854
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47220/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908821421
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47369/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905050741
> Thanks for the work. I will review in next few days.
>
> I think we also need to update the document. But I don't see doc change included yet.
Thanks @viirya! Could you point me to some places where I need to update the doc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900856952
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47084/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905429915
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142755/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900964840
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142582/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908467436
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47364/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908756035
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908287610
**[Test build #142855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142855/testReport)** for PR 33763 at commit [`85cb90d`](https://github.com/apache/spark/commit/85cb90dea326cd2fd2841c630310c28272900f38).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695464067
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class AvailableNowDataStreamWrapper(source: SparkDataStream)
Review comment:
I think that is not needed because we are still using the original source names in `ProgressReporter`. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
brkyvz commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695308536
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -382,6 +413,23 @@ class MicroBatchExecution(
// Generate a map from each unique source to the next available offset.
val (nextOffsets, recentOffsets) = uniqueSources.toSeq.map {
+ case (s: FakeLatestOffsetSupportsTriggerAvailableNow, limit) =>
+ val originalSource = originalSources.getOrElse(s, throw new IllegalStateException(
Review comment:
As you actually pointed out here, I no longer see the value of having the `originalSources` map. Am I missing anything? Can we just do the unwrapping transparently inside `StreamProgress` for `FakeLatestOffsetSupportsTriggerAvailableNow`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
brkyvz commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695300282
##########
File path: sql/core/src/main/java/org/apache/spark/sql/streaming/Trigger.java
##########
@@ -20,6 +20,7 @@
import java.util.concurrent.TimeUnit;
import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.execution.streaming.AvailableNowTrigger$;
Review comment:
uber nit (also existing): isn't the import order messed up? Shouldn't scala be it's own group. the it should be
```java
import org.apache.spark.annotation.Evolving;
import org.apache.spark.sql.execution.streaming.AvailableNowTrigger$;
import org.apache.spark.sql.execution.streaming.ContinuousTrigger;
import org.apache.spark.sql.execution.streaming.OneTimeTrigger$;
import org.apache.spark.sql.execution.streaming.ProcessingTimeTrigger;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906552528
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142797/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695469004
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class AvailableNowDataStreamWrapper(source: SparkDataStream)
Review comment:
You're right. We only wrap sources in uniqueSources which won't affect ProgressReporter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905230931
**[Test build #142755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142755/testReport)** for PR 33763 at commit [`10abc21`](https://github.com/apache/spark/commit/10abc21c3f1dc085dcbd45bb08f024964bf39dca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908913552
> @viirya Would you mind revisiting this? The PR looks OK to me, but I'd like to see your approval as you're reviewing the change. Thanks in advance!
Also @brkyvz could you review again for the changes you requested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909880161
**[Test build #142896 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142896/testReport)** for PR 33763 at commit [`b6c26e2`](https://github.com/apache/spark/commit/b6c26e2e668093b82fa2218d8c4bfa6dd3148242).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909747990
**[Test build #142896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142896/testReport)** for PR 33763 at commit [`b6c26e2`](https://github.com/apache/spark/commit/b6c26e2e668093b82fa2218d8c4bfa6dd3148242).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905274726
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47255/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699107404
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
I find it a bit hard to control the order between starting the query and adding new files into the source. Do you know if there is an easy way to do so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900481747
**[Test build #142557 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142557/testReport)** for PR 33763 at commit [`5d0aee8`](https://github.com/apache/spark/commit/5d0aee8046e5fae2d326a26e78337832ef7d9684).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)`
* `class FakeLatestOffsetSource(source: Source)`
* `class FakeLatestOffsetSupportsTriggerAvailableNow(source: SparkDataStream)`
* `case class OneBatchExecutor() extends TriggerExecutor `
* `case class MultiBatchExecutor() extends TriggerExecutor `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r698897236
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by overriding
+ * its [[latestOffset]] method to always return the latest offset at the beginning of the query.
+ */
+class AvailableNowDataStreamWrapper(val delegate: SparkDataStream)
+ extends SparkDataStream with SupportsTriggerAvailableNow with Logging {
+
+ private var fetchedOffset: streaming.Offset = _
+
+ override def initialOffset(): streaming.Offset = delegate.initialOffset()
+
+ override def deserializeOffset(json: String): streaming.Offset = delegate.deserializeOffset(json)
+
+ override def commit(end: streaming.Offset): Unit = delegate.commit(end)
+
+ override def stop(): Unit = delegate.stop()
+
+ private def getInitialOffset: streaming.Offset = {
+ delegate match {
+ case _: Source => null
+ case m: MicroBatchStream => m.initialOffset
+ }
+ }
Review comment:
We can use null (for v1 sources) and iniitalOffset (for v2 streams) for the startOffset in latestOffset(startOffset, readLimit), since the readLimit is always allAvailable.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
I find it a bit hard to control the order between starting the query and adding new files into the source. Do you know if there is an easy way to do so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908909493
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908912751
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47375/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695456268
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class AvailableNowDataStreamWrapper(source: SparkDataStream)
Review comment:
I guess what @brkyvz said is to override `toString` here to delegate to the origin `source`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
brkyvz commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695309701
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetSupportsTriggerAvailableNow.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class FakeLatestOffsetSupportsTriggerAvailableNow(source: SparkDataStream)
Review comment:
can we call this `AvailableNowDataStreamWrapper(val delegate: SparkDataStream)`? Then you can special case this in `StreamProgress` and not need `originalSources` and things like that. You can even override the `toString` method to return the `delegate.toString` so that the metrics look correct as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900972864
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47101/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906259718
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47297/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-904656489
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142720/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-904507897
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47220/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905957336
**[Test build #142776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142776/testReport)** for PR 33763 at commit [`4198494`](https://github.com/apache/spark/commit/4198494efd45069bc269ab6aaa321695bc92cd13).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901336907
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142614/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900809367
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901922280
@HeartSaVioR, @brkyvz, could you review this? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900856997
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47084/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
brkyvz commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695309701
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetSupportsTriggerAvailableNow.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class FakeLatestOffsetSupportsTriggerAvailableNow(source: SparkDataStream)
Review comment:
can we call this `AvailableNowSourceWrapper(val delegate: SparkDataStream)`? Then you can special case this in `StreamProgress` and not need `originalSources` and things like that. You can even override the `toString` method to return the `delegate.toString` so that the metrics look correct as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #33763:
URL: https://github.com/apache/spark/pull/33763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906914510
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47333/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699116871
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
I'm not sure we could do it easily, as we can't let the query be "suspended" after figuring out source offsets to process. The streaming query is running concurrently with the main thread.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
`assert(q.recentProgress.count(_.numInputRows != 0) == 3)`
This verifies that the query runs three micro-batches instead of one.
Btw, the test `SPARK-36533: Trigger.AvailableNow - checkpointing` covers everything in here now. The changes on `index` determines the number of micro-batch being executed, and we also checked the output DataFrame. I think we can simply remove this test.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
`assert(q.recentProgress.count(_.numInputRows != 0) == 3)`
This verifies that the query runs three micro-batches instead of one.
Btw, the test `SPARK-36533: Trigger.AvailableNow - checkpointing` covers everything in here now. The change on `index` determines the number of micro-batch being executed, and we also checked the output DataFrame. I think we can simply remove this test.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
+ assert(q.recentProgress.count(_.numInputRows != 0) == 3)
+ checkAnswer(sql(s"SELECT * from parquet.`$targetDir`"), (1 to 3).map(_.toString).toDF)
+ } finally {
+ q.stop()
+ }
+
+ createFile(4)
+ createFile(5)
+
+ // run a second query
+ val q2 = startQuery()
+ try {
+ assert(q2.awaitTermination(streamingTimeout.toMillis))
+ assert(q2.recentProgress.count(_.numInputRows != 0) == 2)
+ checkAnswer(sql(s"SELECT * from parquet.`$targetDir`"), (1 to 5).map(_.toString).toDF)
+ } finally {
+ q2.stop()
+ }
+ }
+ }
+
+ test("SPARK-36533: Trigger.AvailableNow - checkpointing") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startTriggerOnceQuery(): StreamingQuery = {
+ df.writeStream
+ .foreachBatch((_: Dataset[Row], _: Long) => {})
+ .trigger(Trigger.Once)
+ .option("checkpointLocation", checkpoint)
+ .start()
+ }
+
+ // run a query with Trigger.Once first
+ val q = startTriggerOnceQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
+ } finally {
+ q.stop()
+ }
+
+ var index = 3 // We have processed the first 3 rows in the first query
Review comment:
nit: probably better to add the comment that it tracks the number of micro-batch execution starting from here. The code is intuitive but worth having elaboration given the importance of this variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906886028
@bozhang2820 Could you please rerun the GA workflow for the PR in your repo?
https://github.com/bozhang2820/spark/actions/runs/1169499438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906966347
Retried on the GA testing and it's passing now: https://github.com/bozhang2820/spark/actions/runs/1169499438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908726085
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908341985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699045368
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
Shouldn't we add more offsets `createFile` here after latest offset was fetched into the source? So we can verify that we only process all available at the beginning of the query?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/TriggerAvailableNowSuite.scala
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import java.io.File
+
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.catalyst.plans.logical.Range
+import org.apache.spark.sql.catalyst.util.stringToFile
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.{ReadLimit, SupportsAdmissionControl}
+import org.apache.spark.sql.execution.streaming.{LongOffset, MemoryStream, Offset, SerializedOffset, Source, StreamingExecutionRelation}
+import org.apache.spark.sql.types.{LongType, StructType}
+
+class TriggerAvailableNowSuite extends FileStreamSourceTest {
+
+ import testImplicits._
+
+ abstract class TestDataFrameProvider {
+ @volatile var currentOffset = 0L
+
+ def toDF: DataFrame
+
+ def incrementAvailableOffset(numNewRows: Int): Unit
+
+ def sourceName: String
+ }
+
+ class TestSource extends TestDataFrameProvider with Source {
+ override def getOffset: Option[Offset] = {
+ if (currentOffset <= 0) None else Some(LongOffset(currentOffset))
+ }
+
+ override def getBatch(start: Option[Offset], end: Offset): DataFrame = {
+ if (currentOffset == 0) currentOffset = getOffsetValue(end)
+ val plan = Range(
+ start.map(getOffsetValue).getOrElse(0L) + 1L, getOffsetValue(end) + 1L, 1, None,
+ isStreaming = true)
+ Dataset.ofRows(spark, plan)
+ }
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ currentOffset += numNewRows
+ }
+
+ override def toDF: DataFrame =
+ Dataset.ofRows(spark, StreamingExecutionRelation(this, spark))
+
+ override def schema: StructType = new StructType().add("value", LongType)
+
+ override def stop(): Unit = {}
+
+ private def getOffsetValue(offset: Offset): Long = {
+ offset match {
+ case s: SerializedOffset => LongOffset(s).offset
+ case l: LongOffset => l.offset
+ case _ => throw new IllegalArgumentException("incorrect offset type: " + offset)
+ }
+ }
+
+ override def sourceName: String = this.getClass.getName
+ }
+
+ class TestSourceWithAdmissionControl extends TestSource with SupportsAdmissionControl {
+ override def getDefaultReadLimit: ReadLimit = ReadLimit.maxRows(1) // this will be overridden
+
+ override def latestOffset(startOffset: streaming.Offset, limit: ReadLimit): streaming.Offset = {
+ val currentOffset = getOffset
+ assert(currentOffset.nonEmpty,
+ "the latestOffset should be called after incrementAvailableOffset")
+ currentOffset.get
+ }
+ }
+
+ class TestMicroBatchStream extends TestDataFrameProvider {
+ private lazy val memoryStream = MemoryStream[Long](0, spark.sqlContext)
+
+ override def toDF: DataFrame = memoryStream.toDF()
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ for (_ <- 1 to numNewRows) {
+ currentOffset += 1
+ memoryStream.addData(currentOffset)
+ }
+ }
+
+ // remove the trailing `$` in the class name
+ override def sourceName: String = MemoryStream.getClass.getSimpleName.dropRight(1)
+ }
+
+ Seq(
+ new TestSource,
+ new TestSourceWithAdmissionControl,
+ new TestMicroBatchStream
+ ).foreach { testSource =>
+ test(s"TriggerAvailableNow for multiple sources with ${testSource.getClass}") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ // Set up a query to read text files one at a time
+ val df1 = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ val df2 = testSource.toDF
+
+ def startQuery(): StreamingQuery = {
+ df1.union(df2).writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+
+ testSource.incrementAvailableOffset(3)
+ createFile(7)
+ createFile(8)
+ createFile(9)
+
+ val q = startQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
+ // only one batch has data in both sources, thus counted, see SPARK-24050
+ assert(q.recentProgress.count(_.numInputRows != 0) == 1)
+ q.recentProgress.foreach { p =>
+ assert(p.sources.exists(_.description.startsWith(testSource.sourceName)))
+ }
+ checkAnswer(sql(s"SELECT * from parquet.`$targetDir`"),
+ Seq(1, 2, 3, 7, 8, 9).map(_.toString).toDF())
+ } finally {
+ q.stop()
+ }
+
+ testSource.incrementAvailableOffset(3)
+ createFile(10)
+ createFile(11)
+ createFile(12)
+
+ // run a second query
+ val q2 = startQuery()
+ try {
+ assert(q2.awaitTermination(streamingTimeout.toMillis))
+ // only one batch has data in both sources, thus counted, see SPARK-24050
+ assert(q2.recentProgress.count(_.numInputRows != 0) == 1)
+ q2.recentProgress.foreach { p =>
+ assert(p.sources.exists(_.description.startsWith(testSource.sourceName)))
+ }
+ checkAnswer(sql(s"SELECT * from parquet.`$targetDir`"), (1 to 12).map(_.toString).toDF())
+ } finally {
+ q2.stop()
+ }
+ }
+ }
+ }
+
+ Seq(
+ new TestSource,
+ new TestSourceWithAdmissionControl,
+ new TestMicroBatchStream
+ ).foreach { testSource =>
+ test(s"TriggerAvailableNow for single source with ${testSource.getClass}") {
+ val tableName = "trigger_available_now_test_table"
+ withTable(tableName) {
+ val df = testSource.toDF
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("memory")
+ .queryName(tableName)
+ .trigger(Trigger.AvailableNow)
+ .start()
+ }
+
+ testSource.incrementAvailableOffset(3)
+
+ val q = startQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
Review comment:
ditto. Should we increase more offsets here after triggering the query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908458764
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47364/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901126649
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142601/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900784040
**[Test build #142578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142578/testReport)** for PR 33763 at commit [`6c017c7`](https://github.com/apache/spark/commit/6c017c70bf7957e53d6ac2810c4e687a1ddba349).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900484727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-904645189
**[Test build #142720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142720/testReport)** for PR 33763 at commit [`e0ad3eb`](https://github.com/apache/spark/commit/e0ad3eb12a059ac34152795eae893ca3cc41877f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901330797
**[Test build #142614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142614/testReport)** for PR 33763 at commit [`c2223f0`](https://github.com/apache/spark/commit/c2223f0997c8dded423c004d108c6774bf7d4855).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)`
* `class FakeLatestOffsetSource(source: Source)`
* `class FakeLatestOffsetSupportsTriggerAvailableNow(source: SparkDataStream)`
* `case class OneBatchExecutor() extends TriggerExecutor `
* `case class MultiBatchExecutor() extends TriggerExecutor `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905283387
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47255/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r696362650
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -2746,6 +2746,14 @@ Here are the different kinds of triggers that are supported.
cluster. In some case, this may lead to significant cost savings.
</td>
</tr>
+ <tr>
+ <td><b>Available-now micro-batch</b></td>
+ <td>
+ Similar to queries one-time micro-batch trigger, the query will process all the available data and then
+ stop on its own. The difference is that it will process the data in multiple micro-bathes,
Review comment:
Thanks! Updated as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908912766
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47375/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900997351
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47101/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-904451383
**[Test build #142720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142720/testReport)** for PR 33763 at commit [`e0ad3eb`](https://github.com/apache/spark/commit/e0ad3eb12a059ac34152795eae893ca3cc41877f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900277161
**[Test build #142557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142557/testReport)** for PR 33763 at commit [`5d0aee8`](https://github.com/apache/spark/commit/5d0aee8046e5fae2d326a26e78337832ef7d9684).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
brkyvz commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695303626
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetSource.scala
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.types.StructType
+
+/**
+ * This class wraps a [[Source]] and makes it supports Trigger.AvailableNow.
+ *
+ * See [[FakeLatestOffsetSupportsTriggerAvailableNow]] for more details.
+ */
+class FakeLatestOffsetSource(source: Source)
+ extends FakeLatestOffsetSupportsTriggerAvailableNow(source) with Source {
+
+ override def schema: StructType = source.schema
+
+ override def getOffset: Option[Offset] = source.getOffset
Review comment:
ditto - if this method isn't going to be called, let's have it throw an UnsupportedOperationException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906914510
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47333/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901119855
**[Test build #142614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142614/testReport)** for PR 33763 at commit [`c2223f0`](https://github.com/apache/spark/commit/c2223f0997c8dded423c004d108c6774bf7d4855).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901188286
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908671159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695428238
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -382,6 +413,23 @@ class MicroBatchExecution(
// Generate a map from each unique source to the next available offset.
val (nextOffsets, recentOffsets) = uniqueSources.toSeq.map {
+ case (s: FakeLatestOffsetSupportsTriggerAvailableNow, limit) =>
+ val originalSource = originalSources.getOrElse(s, throw new IllegalStateException(
Review comment:
For that we would have to wrap sources in the logical plan... I think having the wrapped sources only in `MicroBatchExecution.uniqueSources` is a less intrusive and safer change. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-907126280
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142831/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900277161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908896485
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47375/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r693901210
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/TriggerAvailableNowSuite.scala
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import java.io.File
+
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.catalyst.plans.logical.Range
+import org.apache.spark.sql.catalyst.util.stringToFile
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.{ReadLimit, SupportsAdmissionControl}
+import org.apache.spark.sql.execution.streaming.{LongOffset, MemoryStream, Offset, SerializedOffset, Source, StreamingExecutionRelation}
+import org.apache.spark.sql.types.{LongType, StructType}
+
+class TriggerAvailableNowSuite extends FileStreamSourceTest {
+
+ import testImplicits._
+
+ abstract class TestDataFrameProvider {
+ @volatile var currentOffset = 0L
+
+ def toDF: DataFrame
+
+ def incrementAvailableOffset(numNewRows: Int): Unit
+ }
+
+ class TestSource extends TestDataFrameProvider with Source {
+ override def getOffset: Option[Offset] = {
+ if (currentOffset <= 0) None else Some(LongOffset(currentOffset))
+ }
+
+ override def getBatch(start: Option[Offset], end: Offset): DataFrame = {
+ if (currentOffset == 0) currentOffset = getOffsetValue(end)
+ val plan = Range(
+ start.map(getOffsetValue).getOrElse(0L) + 1L, getOffsetValue(end) + 1L, 1, None,
+ isStreaming = true)
+ Dataset.ofRows(spark, plan)
+ }
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ currentOffset += numNewRows
+ }
+
+ override def toDF: DataFrame =
+ Dataset.ofRows(spark, StreamingExecutionRelation(this, spark))
+
+ override def schema: StructType = new StructType().add("value", LongType)
+
+ override def stop(): Unit = {}
+
+ private def getOffsetValue(offset: Offset): Long = {
+ offset match {
+ case s: SerializedOffset => LongOffset(s).offset
+ case l: LongOffset => l.offset
+ case _ => throw new IllegalArgumentException("incorrect offset type: " + offset)
+ }
+ }
+ }
+
+ class TestSourceWithAdmissionControl extends TestSource with SupportsAdmissionControl {
+ override def getDefaultReadLimit: ReadLimit = ReadLimit.maxRows(1) // this will be overridden
+
+ override def latestOffset(startOffset: streaming.Offset, limit: ReadLimit): streaming.Offset = {
+ val currentOffset = getOffset
+ assert(currentOffset.nonEmpty,
+ "the latestOffset should be called after incrementAvailableOffset")
+ currentOffset.get
+ }
+ }
+
+ class TestMicroBatchStream extends TestDataFrameProvider {
+
+ private lazy val memoryStream = MemoryStream[Long](0, spark.sqlContext)
+
+ override def toDF: DataFrame = memoryStream.toDF()
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ for (_ <- 1 to numNewRows) {
+ currentOffset += 1
+ memoryStream.addData(currentOffset)
+ }
+ }
+ }
+
Review comment:
Sure. Will do.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
##########
@@ -48,7 +48,11 @@ class FileStreamSource(
override val schema: StructType,
partitionColumns: Seq[String],
metadataPath: String,
- options: Map[String, String]) extends SupportsAdmissionControl with Source with Logging {
+ options: Map[String, String])
+ extends SupportsAdmissionControl
Review comment:
Thanks for catching this. Will do.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -47,10 +47,14 @@ class MicroBatchExecution(
triggerClock, plan.outputMode, plan.deleteCheckpointOnStop) {
@volatile protected var sources: Seq[SparkDataStream] = Seq.empty
+ @volatile protected var wrappedSources: MutableMap[
+ SparkDataStream, FakeLatestOffsetSupportsTriggerAvailableNow] = new
+ MutableHashMap[SparkDataStream, FakeLatestOffsetSupportsTriggerAvailableNow]()
private val triggerExecutor = trigger match {
case t: ProcessingTimeTrigger => ProcessingTimeExecutor(t, triggerClock)
- case OneTimeTrigger => OneTimeExecutor()
+ case OneTimeTrigger => OneBatchExecutor()
Review comment:
`MultiBatchExecutor` is really just running `batchRunner` multiple times. The semantic of processing all available data is implemented in `MicroBatchExecution`.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetMicroBatchStream.scala
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.MicroBatchStream
+
+class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
+ extends FakeLatestOffsetSupportsTriggerAvailableNow(source)
+ with MicroBatchStream {
Review comment:
Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909769631
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47399/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908737375
**[Test build #142865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142865/testReport)** for PR 33763 at commit [`0824910`](https://github.com/apache/spark/commit/08249107813f334cc32596ce1b5001a1e777c318).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909891592
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142896/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695432315
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -382,6 +413,23 @@ class MicroBatchExecution(
// Generate a map from each unique source to the next available offset.
val (nextOffsets, recentOffsets) = uniqueSources.toSeq.map {
+ case (s: FakeLatestOffsetSupportsTriggerAvailableNow, limit) =>
+ val originalSource = originalSources.getOrElse(s, throw new IllegalStateException(
Review comment:
Thanks @HeartSaVioR! Will follow what you suggest and remove `originalSource`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901188211
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906089816
**[Test build #142776 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142776/testReport)** for PR 33763 at commit [`4198494`](https://github.com/apache/spark/commit/4198494efd45069bc269ab6aaa321695bc92cd13).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class AvailableNowDataStreamWrapper(val delegate: SparkDataStream)`
* `class AvailableNowMicroBatchStreamWrapper(delegate: MicroBatchStream)`
* `class AvailableNowSourceWrapper(delegate: Source)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905973366
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47276/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699116871
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
I'm not sure we could do it easily, as we can't let the query be "suspended" after figuring out source offsets to process. The streaming query is running concurrently with the main thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908670081
**[Test build #142855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142855/testReport)** for PR 33763 at commit [`85cb90d`](https://github.com/apache/spark/commit/85cb90dea326cd2fd2841c630310c28272900f38).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909925730
Thanks! Merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906900479
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47333/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908801267
**[Test build #142866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142866/testReport)** for PR 33763 at commit [`c96d60d`](https://github.com/apache/spark/commit/c96d60d37500e6a8d2e508abbcdf1d75336fa56c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908874362
@viirya Would you mind revisiting this? The PR looks OK to me, but I'd like to see your approval as you're reviewing the change. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908172263
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47356/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908341932
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47358/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-968370599
Nice catch. I realized it was missing. I'm going to address this soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901188286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r693948734
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -47,10 +47,14 @@ class MicroBatchExecution(
triggerClock, plan.outputMode, plan.deleteCheckpointOnStop) {
@volatile protected var sources: Seq[SparkDataStream] = Seq.empty
+ @volatile protected var wrappedSources: MutableMap[
+ SparkDataStream, FakeLatestOffsetSupportsTriggerAvailableNow] = new
+ MutableHashMap[SparkDataStream, FakeLatestOffsetSupportsTriggerAvailableNow]()
private val triggerExecutor = trigger match {
case t: ProcessingTimeTrigger => ProcessingTimeExecutor(t, triggerClock)
- case OneTimeTrigger => OneTimeExecutor()
+ case OneTimeTrigger => OneBatchExecutor()
Review comment:
Ah OK. I agree that the executor can be separated from the trigger and be generalized by itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-902426850
cc @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900809349
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695430414
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -382,6 +413,23 @@ class MicroBatchExecution(
// Generate a map from each unique source to the next available offset.
val (nextOffsets, recentOffsets) = uniqueSources.toSeq.map {
+ case (s: FakeLatestOffsetSupportsTriggerAvailableNow, limit) =>
+ val originalSource = originalSources.getOrElse(s, throw new IllegalStateException(
Review comment:
If we use `s.source` here to extract the original source, then we can get rid of `originalSource` as well since this is the only usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900784040
**[Test build #142578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142578/testReport)** for PR 33763 at commit [`6c017c7`](https://github.com/apache/spark/commit/6c017c70bf7957e53d6ac2810c4e687a1ddba349).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908909493
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901126649
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142601/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900809367
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909045572
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142872/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906975178
Sorry I was dealing with an online issue in last few days. I may have more time tomorrow looking at this. Feel free to merge if it is urgent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906930741
@viirya @brkyvz Would you mind taking a look again? Looks like comments are addressed but would like to double check. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699727889
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
+ assert(q.recentProgress.count(_.numInputRows != 0) == 3)
+ checkAnswer(sql(s"SELECT * from parquet.`$targetDir`"), (1 to 3).map(_.toString).toDF)
+ } finally {
+ q.stop()
+ }
+
+ createFile(4)
+ createFile(5)
+
+ // run a second query
+ val q2 = startQuery()
+ try {
+ assert(q2.awaitTermination(streamingTimeout.toMillis))
+ assert(q2.recentProgress.count(_.numInputRows != 0) == 2)
+ checkAnswer(sql(s"SELECT * from parquet.`$targetDir`"), (1 to 5).map(_.toString).toDF)
+ } finally {
+ q2.stop()
+ }
+ }
+ }
+
+ test("SPARK-36533: Trigger.AvailableNow - checkpointing") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startTriggerOnceQuery(): StreamingQuery = {
+ df.writeStream
+ .foreachBatch((_: Dataset[Row], _: Long) => {})
+ .trigger(Trigger.Once)
+ .option("checkpointLocation", checkpoint)
+ .start()
+ }
+
+ // run a query with Trigger.Once first
+ val q = startTriggerOnceQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
+ } finally {
+ q.stop()
+ }
+
+ var index = 3 // We have processed the first 3 rows in the first query
Review comment:
nit: probably better to add the comment that it tracks the number of micro-batch execution starting from here. The code is intuitive but worth having elaboration given the importance of this variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900484727
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142557/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900826450
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47084/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900277161
**[Test build #142557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142557/testReport)** for PR 33763 at commit [`5d0aee8`](https://github.com/apache/spark/commit/5d0aee8046e5fae2d326a26e78337832ef7d9684).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906212168
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47297/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r698108415
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class AvailableNowDataStreamWrapper(val delegate: SparkDataStream)
+ extends SparkDataStream with SupportsTriggerAvailableNow with Logging {
+
+ private var fetchedOffset: Option[streaming.Offset] = _
+
+ override def initialOffset(): streaming.Offset = delegate.initialOffset()
+
+ override def deserializeOffset(json: String): streaming.Offset = delegate.deserializeOffset(json)
+
+ override def commit(end: streaming.Offset): Unit = delegate.commit(end)
+
+ override def stop(): Unit = delegate.stop()
+
+ override def prepareForTriggerAvailableNow(): Unit = {}
Review comment:
Thanks for the suggestion! This made me realized that we might need to have `startOffset` as an argument in `prepareForTriggerAvailableNow`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905056479
The structured streaming programming guide: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers
The source is at docs/structured-streaming-programming-guide.md.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r694601545
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
##########
@@ -48,7 +48,11 @@ class FileStreamSource(
override val schema: StructType,
partitionColumns: Seq[String],
metadataPath: String,
- options: Map[String, String]) extends SupportsAdmissionControl with Source with Logging {
+ options: Map[String, String])
+ extends SupportsAdmissionControl
Review comment:
I'm getting:
```
[error] Error while emitting FileStreamSource.scala
[error] assertion failed:
[error] cannot invokespecial org/apache/spark/sql/connector/read/streaming/SupportsAdmissionControl.getDefaultReadLimit, the interface is not a direct parent.
```
after removing `SupportsAdmissionControl`. Can I keep it and add a TODO here for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906092337
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142776/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908737375
**[Test build #142865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142865/testReport)** for PR 33763 at commit [`0824910`](https://github.com/apache/spark/commit/08249107813f334cc32596ce1b5001a1e777c318).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909773735
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47399/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699659162
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
Hmm, okay, I asked this because the test looks like to me only for verifying the query can run with correct results. Not able to check trigger.availablenow. okay for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908882404
**[Test build #142872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142872/testReport)** for PR 33763 at commit [`c96d60d`](https://github.com/apache/spark/commit/c96d60d37500e6a8d2e508abbcdf1d75336fa56c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r694681453
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
##########
@@ -48,7 +48,11 @@ class FileStreamSource(
override val schema: StructType,
partitionColumns: Seq[String],
metadataPath: String,
- options: Map[String, String]) extends SupportsAdmissionControl with Source with Logging {
+ options: Map[String, String])
+ extends SupportsAdmissionControl
Review comment:
Ah got it. It's OK to not touch if it breaks the compilation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908312697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900926349
**[Test build #142578 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142578/testReport)** for PR 33763 at commit [`6c017c7`](https://github.com/apache/spark/commit/6c017c70bf7957e53d6ac2810c4e687a1ddba349).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908801267
**[Test build #142866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142866/testReport)** for PR 33763 at commit [`c96d60d`](https://github.com/apache/spark/commit/c96d60d37500e6a8d2e508abbcdf1d75336fa56c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r694688312
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
##########
@@ -48,7 +48,11 @@ class FileStreamSource(
override val schema: StructType,
partitionColumns: Seq[String],
metadataPath: String,
- options: Map[String, String]) extends SupportsAdmissionControl with Source with Logging {
+ options: Map[String, String])
+ extends SupportsAdmissionControl // TODO: consolidate this with SupportsTriggerAvailableNow
Review comment:
Let's remove the TODO and never mind if it fails the compilation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-904451383
**[Test build #142720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142720/testReport)** for PR 33763 at commit [`e0ad3eb`](https://github.com/apache/spark/commit/e0ad3eb12a059ac34152795eae893ca3cc41877f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908872986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905283344
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47255/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900934809
**[Test build #142601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142601/testReport)** for PR 33763 at commit [`6d9253e`](https://github.com/apache/spark/commit/6d9253ea6e545605672df2101fcf2f9ab088dc20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699107404
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
I find it a bit hard to control the order between starting the query and adding new files into the source. Do you know if there is an easy way to do so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906092337
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142776/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906241114
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47297/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695428238
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -382,6 +413,23 @@ class MicroBatchExecution(
// Generate a map from each unique source to the next available offset.
val (nextOffsets, recentOffsets) = uniqueSources.toSeq.map {
+ case (s: FakeLatestOffsetSupportsTriggerAvailableNow, limit) =>
+ val originalSource = originalSources.getOrElse(s, throw new IllegalStateException(
Review comment:
For that we would have to wrap sources in the logical plan... I think having the wrapped sources only in `MicroBatchExecution.uniqueSources` is a less intrusive and safer change. What do you think?
I will also remove `originalSource` mapping and get the original source directly from the wrapped source instead.
##########
File path: sql/core/src/main/java/org/apache/spark/sql/streaming/Trigger.java
##########
@@ -20,6 +20,7 @@
import java.util.concurrent.TimeUnit;
import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.execution.streaming.AvailableNowTrigger$;
Review comment:
Will do.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetMicroBatchStream.scala
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.MicroBatchStream
+
+/**
+ * This class wraps a [[MicroBatchStream]] and makes it supports Trigger.AvailableNow.
+ *
+ * See [[FakeLatestOffsetSupportsTriggerAvailableNow]] for more details.
+ */
+class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
+ extends FakeLatestOffsetSupportsTriggerAvailableNow(source) with MicroBatchStream {
+
+ override def latestOffset(): streaming.Offset = source.latestOffset()
Review comment:
Makes sense. Will do.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetSupportsTriggerAvailableNow.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class FakeLatestOffsetSupportsTriggerAvailableNow(source: SparkDataStream)
Review comment:
I can do the rename here. For the places to wrap the sources let's have the discussion in the other thread.
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/streaming/SupportsTriggerAvailableNow.java
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read.streaming;
+
+import org.apache.spark.annotation.Evolving;
+
+/**
+ * An interface for streaming sources that supports running in Trigger.AvailableNow mode, which
+ * will process all the available data at the beginning of the query in (possibly) multiple batches.
+ *
+ * This mode will have better scalability comparing to Trigger.Once mode.
+ */
+@Evolving
Review comment:
Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908279194
**[Test build #142853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142853/testReport)** for PR 33763 at commit [`d13807a`](https://github.com/apache/spark/commit/d13807ad79bf8ccefdbe07ded7a04614e16a2934).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906885217
**[Test build #142831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142831/testReport)** for PR 33763 at commit [`874dcc2`](https://github.com/apache/spark/commit/874dcc20fdb4aaf11ac5c162f574fe959dada019).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908764593
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908650844
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142861/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-903541060
Btw, thanks for the great contribution! Nice feature indeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906890270
Don't merge yet; one of action in GA has been lost and it doesn't count the test result in the fork.
https://github.com/bozhang2820/spark/actions/runs/1169499438
@bozhang2820 reran the action and the action is running now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r693735699
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/TriggerAvailableNowSuite.scala
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import java.io.File
+
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.catalyst.plans.logical.Range
+import org.apache.spark.sql.catalyst.util.stringToFile
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.{ReadLimit, SupportsAdmissionControl}
+import org.apache.spark.sql.execution.streaming.{LongOffset, MemoryStream, Offset, SerializedOffset, Source, StreamingExecutionRelation}
+import org.apache.spark.sql.types.{LongType, StructType}
+
+class TriggerAvailableNowSuite extends FileStreamSourceTest {
+
+ import testImplicits._
+
+ abstract class TestDataFrameProvider {
+ @volatile var currentOffset = 0L
+
+ def toDF: DataFrame
+
+ def incrementAvailableOffset(numNewRows: Int): Unit
+ }
+
+ class TestSource extends TestDataFrameProvider with Source {
+ override def getOffset: Option[Offset] = {
+ if (currentOffset <= 0) None else Some(LongOffset(currentOffset))
+ }
+
+ override def getBatch(start: Option[Offset], end: Offset): DataFrame = {
+ if (currentOffset == 0) currentOffset = getOffsetValue(end)
+ val plan = Range(
+ start.map(getOffsetValue).getOrElse(0L) + 1L, getOffsetValue(end) + 1L, 1, None,
+ isStreaming = true)
+ Dataset.ofRows(spark, plan)
+ }
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ currentOffset += numNewRows
+ }
+
+ override def toDF: DataFrame =
+ Dataset.ofRows(spark, StreamingExecutionRelation(this, spark))
+
+ override def schema: StructType = new StructType().add("value", LongType)
+
+ override def stop(): Unit = {}
+
+ private def getOffsetValue(offset: Offset): Long = {
+ offset match {
+ case s: SerializedOffset => LongOffset(s).offset
+ case l: LongOffset => l.offset
+ case _ => throw new IllegalArgumentException("incorrect offset type: " + offset)
+ }
+ }
+ }
+
+ class TestSourceWithAdmissionControl extends TestSource with SupportsAdmissionControl {
+ override def getDefaultReadLimit: ReadLimit = ReadLimit.maxRows(1) // this will be overridden
+
+ override def latestOffset(startOffset: streaming.Offset, limit: ReadLimit): streaming.Offset = {
+ val currentOffset = getOffset
+ assert(currentOffset.nonEmpty,
+ "the latestOffset should be called after incrementAvailableOffset")
+ currentOffset.get
+ }
+ }
+
+ class TestMicroBatchStream extends TestDataFrameProvider {
+
+ private lazy val memoryStream = MemoryStream[Long](0, spark.sqlContext)
+
+ override def toDF: DataFrame = memoryStream.toDF()
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ for (_ <- 1 to numNewRows) {
+ currentOffset += 1
+ memoryStream.addData(currentOffset)
+ }
+ }
+ }
+
+ Seq(
+ new TestSource,
+ new TestSourceWithAdmissionControl,
+ new TestMicroBatchStream
+ ).foreach { testSource =>
+ test(s"TriggerAvailableNow for multiple sources with ${testSource.getClass}") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ // Set up a query to read text files one at a time
+ val df1 = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ val df2 = testSource.toDF
+
+ def startQuery(): StreamingQuery = {
+ df1.union(df2).writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+
+ testSource.incrementAvailableOffset(3)
+ createFile(7)
+ createFile(8)
+ createFile(9)
+
+ val q = startQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
+ // only one batch has data in both sources, thus counted, see SPARK-24050
Review comment:
(Beyond the scope of the PR) it would be ideal if we can revisit and fix it.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/TriggerAvailableNowSuite.scala
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import java.io.File
+
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.catalyst.plans.logical.Range
+import org.apache.spark.sql.catalyst.util.stringToFile
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.{ReadLimit, SupportsAdmissionControl}
+import org.apache.spark.sql.execution.streaming.{LongOffset, MemoryStream, Offset, SerializedOffset, Source, StreamingExecutionRelation}
+import org.apache.spark.sql.types.{LongType, StructType}
+
+class TriggerAvailableNowSuite extends FileStreamSourceTest {
+
+ import testImplicits._
+
+ abstract class TestDataFrameProvider {
+ @volatile var currentOffset = 0L
+
+ def toDF: DataFrame
+
+ def incrementAvailableOffset(numNewRows: Int): Unit
+ }
+
+ class TestSource extends TestDataFrameProvider with Source {
+ override def getOffset: Option[Offset] = {
+ if (currentOffset <= 0) None else Some(LongOffset(currentOffset))
+ }
+
+ override def getBatch(start: Option[Offset], end: Offset): DataFrame = {
+ if (currentOffset == 0) currentOffset = getOffsetValue(end)
+ val plan = Range(
+ start.map(getOffsetValue).getOrElse(0L) + 1L, getOffsetValue(end) + 1L, 1, None,
+ isStreaming = true)
+ Dataset.ofRows(spark, plan)
+ }
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ currentOffset += numNewRows
+ }
+
+ override def toDF: DataFrame =
+ Dataset.ofRows(spark, StreamingExecutionRelation(this, spark))
+
+ override def schema: StructType = new StructType().add("value", LongType)
+
+ override def stop(): Unit = {}
+
+ private def getOffsetValue(offset: Offset): Long = {
+ offset match {
+ case s: SerializedOffset => LongOffset(s).offset
+ case l: LongOffset => l.offset
+ case _ => throw new IllegalArgumentException("incorrect offset type: " + offset)
+ }
+ }
+ }
+
+ class TestSourceWithAdmissionControl extends TestSource with SupportsAdmissionControl {
+ override def getDefaultReadLimit: ReadLimit = ReadLimit.maxRows(1) // this will be overridden
+
+ override def latestOffset(startOffset: streaming.Offset, limit: ReadLimit): streaming.Offset = {
+ val currentOffset = getOffset
+ assert(currentOffset.nonEmpty,
+ "the latestOffset should be called after incrementAvailableOffset")
+ currentOffset.get
+ }
+ }
+
+ class TestMicroBatchStream extends TestDataFrameProvider {
+
+ private lazy val memoryStream = MemoryStream[Long](0, spark.sqlContext)
+
+ override def toDF: DataFrame = memoryStream.toDF()
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ for (_ <- 1 to numNewRows) {
+ currentOffset += 1
+ memoryStream.addData(currentOffset)
+ }
+ }
+ }
+
+ Seq(
+ new TestSource,
+ new TestSourceWithAdmissionControl,
+ new TestMicroBatchStream
+ ).foreach { testSource =>
+ test(s"TriggerAvailableNow for multiple sources with ${testSource.getClass}") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ // Set up a query to read text files one at a time
+ val df1 = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ val df2 = testSource.toDF
+
+ def startQuery(): StreamingQuery = {
+ df1.union(df2).writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+
+ testSource.incrementAvailableOffset(3)
+ createFile(7)
+ createFile(8)
+ createFile(9)
+
+ val q = startQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
+ // only one batch has data in both sources, thus counted, see SPARK-24050
Review comment:
(Beyond the scope of the PR) it would be ideal if we can revisit and fix it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908836630
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142865/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900856997
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47084/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908460315
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47364/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699116871
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
I'm not sure we could do it easily, as we can't let the query be "suspended" after figuring out source offsets to process. The streaming query is running concurrently with the main thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908726085
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699726063
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
`assert(q.recentProgress.count(_.numInputRows != 0) == 3)`
This verifies that the query runs three micro-batches instead of one.
Btw, the test `SPARK-36533: Trigger.AvailableNow - checkpointing` covers everything in here now. The changes on `index` determines the number of micro-batch being executed, and we also checked the output DataFrame. I think we can simply remove this test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699046646
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/TriggerAvailableNowSuite.scala
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import java.io.File
+
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.catalyst.plans.logical.Range
+import org.apache.spark.sql.catalyst.util.stringToFile
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.{ReadLimit, SupportsAdmissionControl}
+import org.apache.spark.sql.execution.streaming.{LongOffset, MemoryStream, Offset, SerializedOffset, Source, StreamingExecutionRelation}
+import org.apache.spark.sql.types.{LongType, StructType}
+
+class TriggerAvailableNowSuite extends FileStreamSourceTest {
+
+ import testImplicits._
+
+ abstract class TestDataFrameProvider {
+ @volatile var currentOffset = 0L
+
+ def toDF: DataFrame
+
+ def incrementAvailableOffset(numNewRows: Int): Unit
+
+ def sourceName: String
+ }
+
+ class TestSource extends TestDataFrameProvider with Source {
+ override def getOffset: Option[Offset] = {
+ if (currentOffset <= 0) None else Some(LongOffset(currentOffset))
+ }
+
+ override def getBatch(start: Option[Offset], end: Offset): DataFrame = {
+ if (currentOffset == 0) currentOffset = getOffsetValue(end)
+ val plan = Range(
+ start.map(getOffsetValue).getOrElse(0L) + 1L, getOffsetValue(end) + 1L, 1, None,
+ isStreaming = true)
+ Dataset.ofRows(spark, plan)
+ }
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ currentOffset += numNewRows
+ }
+
+ override def toDF: DataFrame =
+ Dataset.ofRows(spark, StreamingExecutionRelation(this, spark))
+
+ override def schema: StructType = new StructType().add("value", LongType)
+
+ override def stop(): Unit = {}
+
+ private def getOffsetValue(offset: Offset): Long = {
+ offset match {
+ case s: SerializedOffset => LongOffset(s).offset
+ case l: LongOffset => l.offset
+ case _ => throw new IllegalArgumentException("incorrect offset type: " + offset)
+ }
+ }
+
+ override def sourceName: String = this.getClass.getName
+ }
+
+ class TestSourceWithAdmissionControl extends TestSource with SupportsAdmissionControl {
+ override def getDefaultReadLimit: ReadLimit = ReadLimit.maxRows(1) // this will be overridden
+
+ override def latestOffset(startOffset: streaming.Offset, limit: ReadLimit): streaming.Offset = {
+ val currentOffset = getOffset
+ assert(currentOffset.nonEmpty,
+ "the latestOffset should be called after incrementAvailableOffset")
+ currentOffset.get
+ }
+ }
+
+ class TestMicroBatchStream extends TestDataFrameProvider {
+ private lazy val memoryStream = MemoryStream[Long](0, spark.sqlContext)
+
+ override def toDF: DataFrame = memoryStream.toDF()
+
+ override def incrementAvailableOffset(numNewRows: Int): Unit = {
+ for (_ <- 1 to numNewRows) {
+ currentOffset += 1
+ memoryStream.addData(currentOffset)
+ }
+ }
+
+ // remove the trailing `$` in the class name
+ override def sourceName: String = MemoryStream.getClass.getSimpleName.dropRight(1)
+ }
+
+ Seq(
+ new TestSource,
+ new TestSourceWithAdmissionControl,
+ new TestMicroBatchStream
+ ).foreach { testSource =>
+ test(s"TriggerAvailableNow for multiple sources with ${testSource.getClass}") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ // Set up a query to read text files one at a time
+ val df1 = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ val df2 = testSource.toDF
+
+ def startQuery(): StreamingQuery = {
+ df1.union(df2).writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+
+ testSource.incrementAvailableOffset(3)
+ createFile(7)
+ createFile(8)
+ createFile(9)
+
+ val q = startQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
+ // only one batch has data in both sources, thus counted, see SPARK-24050
+ assert(q.recentProgress.count(_.numInputRows != 0) == 1)
+ q.recentProgress.foreach { p =>
+ assert(p.sources.exists(_.description.startsWith(testSource.sourceName)))
+ }
+ checkAnswer(sql(s"SELECT * from parquet.`$targetDir`"),
+ Seq(1, 2, 3, 7, 8, 9).map(_.toString).toDF())
+ } finally {
+ q.stop()
+ }
+
+ testSource.incrementAvailableOffset(3)
+ createFile(10)
+ createFile(11)
+ createFile(12)
+
+ // run a second query
+ val q2 = startQuery()
+ try {
+ assert(q2.awaitTermination(streamingTimeout.toMillis))
+ // only one batch has data in both sources, thus counted, see SPARK-24050
+ assert(q2.recentProgress.count(_.numInputRows != 0) == 1)
+ q2.recentProgress.foreach { p =>
+ assert(p.sources.exists(_.description.startsWith(testSource.sourceName)))
+ }
+ checkAnswer(sql(s"SELECT * from parquet.`$targetDir`"), (1 to 12).map(_.toString).toDF())
+ } finally {
+ q2.stop()
+ }
+ }
+ }
+ }
+
+ Seq(
+ new TestSource,
+ new TestSourceWithAdmissionControl,
+ new TestMicroBatchStream
+ ).foreach { testSource =>
+ test(s"TriggerAvailableNow for single source with ${testSource.getClass}") {
+ val tableName = "trigger_available_now_test_table"
+ withTable(tableName) {
+ val df = testSource.toDF
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("memory")
+ .queryName(tableName)
+ .trigger(Trigger.AvailableNow)
+ .start()
+ }
+
+ testSource.incrementAvailableOffset(3)
+
+ val q = startQuery()
+
+ try {
+ assert(q.awaitTermination(streamingTimeout.toMillis))
Review comment:
ditto. Should we increase more offsets here after triggering the query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908119698
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47356/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908287610
**[Test build #142855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142855/testReport)** for PR 33763 at commit [`85cb90d`](https://github.com/apache/spark/commit/85cb90dea326cd2fd2841c630310c28272900f38).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901002807
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47101/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900954389
**[Test build #142582 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142582/testReport)** for PR 33763 at commit [`c2223f0`](https://github.com/apache/spark/commit/c2223f0997c8dded423c004d108c6774bf7d4855).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)`
* `class FakeLatestOffsetSource(source: Source)`
* `class FakeLatestOffsetSupportsTriggerAvailableNow(source: SparkDataStream)`
* `case class OneBatchExecutor() extends TriggerExecutor `
* `case class MultiBatchExecutor() extends TriggerExecutor `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900378524
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47059/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906002133
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47276/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-904656489
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142720/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901119855
**[Test build #142614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142614/testReport)** for PR 33763 at commit [`c2223f0`](https://github.com/apache/spark/commit/c2223f0997c8dded423c004d108c6774bf7d4855).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905231021
> This looks awesome! Left some comments. It would also be great to add tests around the progress reporting (not overwriting the Source description with that of the wrapper class, etc)
Thanks! Added assertions for the source descriptions in `TriggerAvailableNowSuite`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908908558
**[Test build #142866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142866/testReport)** for PR 33763 at commit [`c96d60d`](https://github.com/apache/spark/commit/c96d60d37500e6a8d2e508abbcdf1d75336fa56c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908645785
**[Test build #142861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142861/testReport)** for PR 33763 at commit [`0824910`](https://github.com/apache/spark/commit/08249107813f334cc32596ce1b5001a1e777c318).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909747990
**[Test build #142896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142896/testReport)** for PR 33763 at commit [`b6c26e2`](https://github.com/apache/spark/commit/b6c26e2e668093b82fa2218d8c4bfa6dd3148242).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699726063
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
`assert(q.recentProgress.count(_.numInputRows != 0) == 3)`
This verifies that the query runs three micro-batches instead of one.
Btw, the test `SPARK-36533: Trigger.AvailableNow - checkpointing` covers everything in here now. The change on `index` determines the number of micro-batch being executed, and we also checked the output DataFrame. I think we can simply remove this test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699659162
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
Hmm, okay, I asked this because the test looks like to me only for verifying the query can run with correct results. Not able to check trigger.availablenow. okay for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900328800
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47059/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908671159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906885217
**[Test build #142831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142831/testReport)** for PR 33763 at commit [`874dcc2`](https://github.com/apache/spark/commit/874dcc20fdb4aaf11ac5c162f574fe959dada019).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908467436
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47364/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908421005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908341985
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47358/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908882404
**[Test build #142872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142872/testReport)** for PR 33763 at commit [`c96d60d`](https://github.com/apache/spark/commit/c96d60d37500e6a8d2e508abbcdf1d75336fa56c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
brkyvz commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695300282
##########
File path: sql/core/src/main/java/org/apache/spark/sql/streaming/Trigger.java
##########
@@ -20,6 +20,7 @@
import java.util.concurrent.TimeUnit;
import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.execution.streaming.AvailableNowTrigger$;
Review comment:
uber nit (also existing): isn't the import order messed up? Shouldn't scala be it's own group. the it should be
```java
import org.apache.spark.annotation.Evolving;
import org.apache.spark.sql.execution.streaming.AvailableNowTrigger$;
import org.apache.spark.sql.execution.streaming.ContinuousTrigger;import
import org.apache.spark.sql.execution.streaming.OneTimeTrigger$;
import org.apache.spark.sql.execution.streaming.ProcessingTimeTrigger;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909777317
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47399/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r694627350
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -382,6 +413,23 @@ class MicroBatchExecution(
// Generate a map from each unique source to the next available offset.
val (nextOffsets, recentOffsets) = uniqueSources.toSeq.map {
+ case (s: FakeLatestOffsetSupportsTriggerAvailableNow, limit) =>
+ val originalSource = originalSources.getOrElse(s, throw new IllegalStateException(
Review comment:
We can also combine this with the following pattern, but that would need to change the type of `originalSource` to be more generic (`SupportsAdmissionControl` instead of `FakeLatestOffsetSupportsTriggerAvailableNow`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r699045368
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
##########
@@ -1304,6 +1304,140 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
}
}
+ test("SPARK-36533: Trigger.AvailableNow") {
+ withTempDirs { (src, target) =>
+ val checkpoint = new File(target, "chk").getCanonicalPath
+ val targetDir = new File(target, "data").getCanonicalPath
+ var lastFileModTime: Option[Long] = None
+
+ /** Create a text file with a single data item */
+ def createFile(data: Int): File = {
+ val file = stringToFile(new File(src, s"$data.txt"), data.toString)
+ if (lastFileModTime.nonEmpty) file.setLastModified(lastFileModTime.get + 1000)
+ lastFileModTime = Some(file.lastModified)
+ file
+ }
+
+ createFile(1)
+ createFile(2)
+ createFile(3)
+
+ // Set up a query to read text files one at a time
+ val df = spark
+ .readStream
+ .option("maxFilesPerTrigger", 1)
+ .text(src.getCanonicalPath)
+
+ def startQuery(): StreamingQuery = {
+ df.writeStream
+ .format("parquet")
+ .trigger(Trigger.AvailableNow)
+ .option("checkpointLocation", checkpoint)
+ .start(targetDir)
+ }
+ val q = startQuery()
+
+ try {
Review comment:
Shouldn't we add more offsets `createFile` here after latest offset was fetched into the source? So we can verify that we only process all available at the beginning of the query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908872986
retest this, please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r698016404
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
##########
@@ -105,6 +109,8 @@ class FileStreamSource(
// Visible for testing and debugging in production.
val seenFiles = new SeenFilesMap(maxFileAgeMs, fileNameOnly)
+ var allFilesForTriggerAvailableNow: Seq[(String, Long)] = null
Review comment:
Thanks for catching this! Will change to private.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -382,6 +404,22 @@ class MicroBatchExecution(
// Generate a map from each unique source to the next available offset.
val (nextOffsets, recentOffsets) = uniqueSources.toSeq.map {
+ case (s: AvailableNowDataStreamWrapper, limit) =>
+ val originalSource = s.delegate
+ updateStatusMessage(s"Getting offsets from $s")
+ reportTimeTaken("latestOffset") {
+ val startOffsetOpt = availableOffsets.get(originalSource)
+ val startOffset = s match {
Review comment:
It should not matter since the result will always be the same. However since we get `startOffsetOpt` by `originalSource`, I think it makes sense to change to match against that.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -121,18 +122,36 @@ class MicroBatchExecution(
// v2 source
case r: StreamingDataSourceV2Relation => r.stream
}
- uniqueSources = sources.distinct.map {
- case source: SupportsAdmissionControl =>
- val limit = source.getDefaultReadLimit
- if (trigger == OneTimeTrigger && limit != ReadLimit.allAvailable()) {
- logWarning(s"The read limit $limit for $source is ignored when Trigger.Once() is used.")
- source -> ReadLimit.allAvailable()
- } else {
- source -> limit
- }
- case other =>
- other -> ReadLimit.allAvailable()
- }.toMap
+ uniqueSources = triggerExecutor match {
+ case _: SingleBatchExecutor =>
+ sources.distinct.map {
+ case s: SupportsAdmissionControl =>
+ val limit = s.getDefaultReadLimit
+ if (limit != ReadLimit.allAvailable()) {
+ logWarning(
+ s"The read limit $limit for $s is ignored when Trigger.Once is used.")
+ }
+ s -> ReadLimit.allAvailable()
+ case s =>
+ s -> ReadLimit.allAvailable()
+ }.toMap
+
+ case _: MultiBatchExecutor =>
+ sources.distinct.map {
+ case s: SupportsTriggerAvailableNow => s
+ case s: Source => new AvailableNowSourceWrapper(s)
+ case s: MicroBatchStream => new AvailableNowMicroBatchStreamWrapper(s)
+ }.map { s =>
+ s.prepareForTriggerAvailableNow()
+ s -> s.getDefaultReadLimit
Review comment:
I think it's fine. Users might put read limit on some sources but not the others when using `Trigger.AvailableNow` and we do not need to warn in those cases.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
##########
@@ -126,7 +132,12 @@ class FileStreamSource(
unreadFiles
} else {
// All the new files found - ignore aged files and files that we have seen.
Review comment:
Will update the comment here to describe the logic for Trigger.AvailableNow instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908283948
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142853/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905957336
**[Test build #142776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142776/testReport)** for PR 33763 at commit [`4198494`](https://github.com/apache/spark/commit/4198494efd45069bc269ab6aaa321695bc92cd13).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908650844
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142861/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909891592
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142896/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908764593
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r697823961
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class AvailableNowDataStreamWrapper(val delegate: SparkDataStream)
+ extends SparkDataStream with SupportsTriggerAvailableNow with Logging {
+
+ private var fetchedOffset: Option[streaming.Offset] = _
+
+ override def initialOffset(): streaming.Offset = delegate.initialOffset()
+
+ override def deserializeOffset(json: String): streaming.Offset = delegate.deserializeOffset(json)
+
+ override def commit(end: streaming.Offset): Unit = delegate.commit(end)
+
+ override def stop(): Unit = delegate.stop()
+
+ override def prepareForTriggerAvailableNow(): Unit = {}
Review comment:
As this implements `SupportsTriggerAvailableNow`, but seems `prepareForTriggerAvailableNow` is useless? Why don't we record latest offset here?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
##########
@@ -105,6 +109,8 @@ class FileStreamSource(
// Visible for testing and debugging in production.
val seenFiles = new SeenFilesMap(maxFileAgeMs, fileNameOnly)
+ var allFilesForTriggerAvailableNow: Seq[(String, Long)] = null
Review comment:
`private`?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -382,6 +404,22 @@ class MicroBatchExecution(
// Generate a map from each unique source to the next available offset.
val (nextOffsets, recentOffsets) = uniqueSources.toSeq.map {
+ case (s: AvailableNowDataStreamWrapper, limit) =>
+ val originalSource = s.delegate
+ updateStatusMessage(s"Getting offsets from $s")
+ reportTimeTaken("latestOffset") {
+ val startOffsetOpt = availableOffsets.get(originalSource)
+ val startOffset = s match {
Review comment:
here shouldn't we match against `originalSource`?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
##########
@@ -126,7 +132,12 @@ class FileStreamSource(
unreadFiles
} else {
// All the new files found - ignore aged files and files that we have seen.
Review comment:
Should this comment follow `fetchAllFiles`?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -121,18 +122,36 @@ class MicroBatchExecution(
// v2 source
case r: StreamingDataSourceV2Relation => r.stream
}
- uniqueSources = sources.distinct.map {
- case source: SupportsAdmissionControl =>
- val limit = source.getDefaultReadLimit
- if (trigger == OneTimeTrigger && limit != ReadLimit.allAvailable()) {
- logWarning(s"The read limit $limit for $source is ignored when Trigger.Once() is used.")
- source -> ReadLimit.allAvailable()
- } else {
- source -> limit
- }
- case other =>
- other -> ReadLimit.allAvailable()
- }.toMap
+ uniqueSources = triggerExecutor match {
+ case _: SingleBatchExecutor =>
+ sources.distinct.map {
+ case s: SupportsAdmissionControl =>
+ val limit = s.getDefaultReadLimit
+ if (limit != ReadLimit.allAvailable()) {
+ logWarning(
+ s"The read limit $limit for $s is ignored when Trigger.Once is used.")
+ }
+ s -> ReadLimit.allAvailable()
+ case s =>
+ s -> ReadLimit.allAvailable()
+ }.toMap
+
+ case _: MultiBatchExecutor =>
+ sources.distinct.map {
+ case s: SupportsTriggerAvailableNow => s
+ case s: Source => new AvailableNowSourceWrapper(s)
+ case s: MicroBatchStream => new AvailableNowMicroBatchStreamWrapper(s)
+ }.map { s =>
+ s.prepareForTriggerAvailableNow()
+ s -> s.getDefaultReadLimit
Review comment:
Should we log warning if it is `getDefaultReadLimit` is `ReadLimit.allAvailable()` for `AvailableNowTrigger`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900803050
**[Test build #142582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142582/testReport)** for PR 33763 at commit [`c2223f0`](https://github.com/apache/spark/commit/c2223f0997c8dded423c004d108c6774bf7d4855).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906259718
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47297/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908912766
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47375/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906884961
retest this, please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901157898
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r694642021
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/streaming/SupportsTriggerAvailableNow.java
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read.streaming;
+
+import org.apache.spark.annotation.Evolving;
+
+/**
+ * An interface for streaming sources that supports running in Trigger.AvailableNow mode, which
+ * will process all the available data at the beginning of the query in (possibly) multiple batches.
+ *
+ * This mode will have better scalability comparing to Trigger.Once mode.
+ */
+@Evolving
Review comment:
Could you add `@since` annotation too? E.g. `SupportsAdmissionControl`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r696244549
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -2746,6 +2746,14 @@ Here are the different kinds of triggers that are supported.
cluster. In some case, this may lead to significant cost savings.
</td>
</tr>
+ <tr>
+ <td><b>Available-now micro-batch</b></td>
+ <td>
+ Similar to queries one-time micro-batch trigger, the query will process all the available data and then
+ stop on its own. The difference is that it will process the data in multiple micro-bathes,
Review comment:
And worth to mention that splitting batches is not done automatically/magically. It follows the source option on splitting batches.
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -2746,6 +2746,14 @@ Here are the different kinds of triggers that are supported.
cluster. In some case, this may lead to significant cost savings.
</td>
</tr>
+ <tr>
+ <td><b>Available-now micro-batch</b></td>
+ <td>
+ Similar to queries one-time micro-batch trigger, the query will process all the available data and then
+ stop on its own. The difference is that it will process the data in multiple micro-bathes,
Review comment:
nit: micro-batches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908341985
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47358/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r698243792
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class AvailableNowDataStreamWrapper(val delegate: SparkDataStream)
+ extends SparkDataStream with SupportsTriggerAvailableNow with Logging {
+
+ private var fetchedOffset: Option[streaming.Offset] = _
+
+ override def initialOffset(): streaming.Offset = delegate.initialOffset()
+
+ override def deserializeOffset(json: String): streaming.Offset = delegate.deserializeOffset(json)
+
+ override def commit(end: streaming.Offset): Unit = delegate.commit(end)
+
+ override def stop(): Unit = delegate.stop()
+
+ override def prepareForTriggerAvailableNow(): Unit = {}
Review comment:
Turned out `startOffset` is not needed in `prepareForTriggerAvailableNow`. See the latest code update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
brkyvz commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695300790
##########
File path: sql/core/src/main/java/org/apache/spark/sql/streaming/Trigger.java
##########
@@ -89,15 +90,23 @@ public static Trigger ProcessingTime(String interval) {
}
/**
- * A trigger that process only one batch of data in a streaming query then terminates
- * the query.
+ * A trigger that processes all available data in a single batch then terminates the query.
*
* @since 2.2.0
*/
public static Trigger Once() {
return OneTimeTrigger$.MODULE$;
}
+ /**
+ * A trigger that processes all available data in multiple batches then terminates the query.
Review comment:
`... all available data at the start of the query in one or multiple batches...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
brkyvz commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695304926
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -47,10 +47,14 @@ class MicroBatchExecution(
triggerClock, plan.outputMode, plan.deleteCheckpointOnStop) {
@volatile protected var sources: Seq[SparkDataStream] = Seq.empty
+ @volatile protected var wrappedSources: MutableMap[
+ SparkDataStream, FakeLatestOffsetSupportsTriggerAvailableNow] = new
+ MutableHashMap[SparkDataStream, FakeLatestOffsetSupportsTriggerAvailableNow]()
private val triggerExecutor = trigger match {
case t: ProcessingTimeTrigger => ProcessingTimeExecutor(t, triggerClock)
- case OneTimeTrigger => OneTimeExecutor()
+ case OneTimeTrigger => OneBatchExecutor()
Review comment:
uber nit: `SingleBatchExecutor`, while is longer better fits with `MultiBatchExecutor`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905428595
**[Test build #142755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142755/testReport)** for PR 33763 at commit [`10abc21`](https://github.com/apache/spark/commit/10abc21c3f1dc085dcbd45bb08f024964bf39dca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class AvailableNowDataStreamWrapper(source: SparkDataStream)`
* `class AvailableNowMicroBatchStreamWrapper(source: MicroBatchStream)`
* `class AvailableNowSourceWrapper(source: Source)`
* `case class SingleBatchExecutor() extends TriggerExecutor `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908341985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909035935
**[Test build #142872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142872/testReport)** for PR 33763 at commit [`c96d60d`](https://github.com/apache/spark/commit/c96d60d37500e6a8d2e508abbcdf1d75336fa56c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908882404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909045572
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142872/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r695464067
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by
+ * overriding its [[latestOffset]] method to always return the latest offset when the method is
+ * first called. It is as if there is no new data coming in from the source after the first
+ * [[latestOffset]] call.
+ */
+class AvailableNowDataStreamWrapper(source: SparkDataStream)
Review comment:
I think that is not needed because we are still using the original source names in `ProgressReporter`. Let me know if you think otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900803050
**[Test build #142582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142582/testReport)** for PR 33763 at commit [`c2223f0`](https://github.com/apache/spark/commit/c2223f0997c8dded423c004d108c6774bf7d4855).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909927323
Thanks @bozhang2820 for the great contribution and thanks all for reviewing! I just merged this into master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908421005
**[Test build #142861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142861/testReport)** for PR 33763 at commit [`0824910`](https://github.com/apache/spark/commit/08249107813f334cc32596ce1b5001a1e777c318).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r694621288
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetMicroBatchStream.scala
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.MicroBatchStream
+
+class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
Review comment:
Done.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetMicroBatchStream.scala
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.MicroBatchStream
+
+class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
+ extends FakeLatestOffsetSupportsTriggerAvailableNow(source)
+ with MicroBatchStream {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r693208341
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetMicroBatchStream.scala
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.MicroBatchStream
+
+class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
Review comment:
Would be great if we have some documentation for this utility class to explain why this fake stream wrapper is needed.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetMicroBatchStream.scala
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory}
+import org.apache.spark.sql.connector.read.streaming
+import org.apache.spark.sql.connector.read.streaming.MicroBatchStream
+
+class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
+ extends FakeLatestOffsetSupportsTriggerAvailableNow(source)
+ with MicroBatchStream {
Review comment:
seems it's not the right code style, we can put the `with` in the same line with extends.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FakeLatestOffsetSource.scala
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.types.StructType
+
+class FakeLatestOffsetSource(source: Source)
Review comment:
ditto for the doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908160444
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47356/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901002807
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47101/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-907126280
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142831/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900934809
**[Test build #142601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142601/testReport)** for PR 33763 at commit [`6d9253e`](https://github.com/apache/spark/commit/6d9253ea6e545605672df2101fcf2f9ab088dc20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900481747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-904481757
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47220/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906011090
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47276/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906914489
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47333/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900964840
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142582/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-904507897
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47220/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908913552
> @viirya Would you mind revisiting this? The PR looks OK to me, but I'd like to see your approval as you're reviewing the change. Thanks in advance!
Also @brkyvz could you review again for the changes you requested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906180753
**[Test build #142797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142797/testReport)** for PR 33763 at commit [`874dcc2`](https://github.com/apache/spark/commit/874dcc20fdb4aaf11ac5c162f574fe959dada019).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-907195514
Not urgent. Please take your time. Thanks for keeping an eye on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908821399
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47369/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905230931
**[Test build #142755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142755/testReport)** for PR 33763 at commit [`10abc21`](https://github.com/apache/spark/commit/10abc21c3f1dc085dcbd45bb08f024964bf39dca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-901125109
**[Test build #142601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142601/testReport)** for PR 33763 at commit [`6d9253e`](https://github.com/apache/spark/commit/6d9253ea6e545605672df2101fcf2f9ab088dc20).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905429915
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142755/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900378413
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47059/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-907125975
**[Test build #142831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142831/testReport)** for PR 33763 at commit [`874dcc2`](https://github.com/apache/spark/commit/874dcc20fdb4aaf11ac5c162f574fe959dada019).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r697826519
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -243,6 +262,9 @@ class MicroBatchExecution(
if (isCurrentBatchConstructed) {
currentBatchId += 1
isCurrentBatchConstructed = false
+ } else if (triggerExecutor.isInstanceOf[MultiBatchExecutor]) {
+ logInfo("Finished processing all data, terminating this Trigger.AvailableNow query")
Review comment:
all data -> all available data for the trigger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900484727
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142557/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-900484727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908836630
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142865/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909035935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908829704
**[Test build #142865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142865/testReport)** for PR 33763 at commit [`0824910`](https://github.com/apache/spark/commit/08249107813f334cc32596ce1b5001a1e777c318).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #33763:
URL: https://github.com/apache/spark/pull/33763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909925730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908172263
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47356/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-905283387
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47255/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908818086
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47369/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bozhang2820 commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
bozhang2820 commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r698897236
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
+import org.apache.spark.sql.connector.read.streaming
+
+/**
+ * This class wraps a [[SparkDataStream]] and makes it support Trigger.AvailableNow, by overriding
+ * its [[latestOffset]] method to always return the latest offset at the beginning of the query.
+ */
+class AvailableNowDataStreamWrapper(val delegate: SparkDataStream)
+ extends SparkDataStream with SupportsTriggerAvailableNow with Logging {
+
+ private var fetchedOffset: streaming.Offset = _
+
+ override def initialOffset(): streaming.Offset = delegate.initialOffset()
+
+ override def deserializeOffset(json: String): streaming.Offset = delegate.deserializeOffset(json)
+
+ override def commit(end: streaming.Offset): Unit = delegate.commit(end)
+
+ override def stop(): Unit = delegate.stop()
+
+ private def getInitialOffset: streaming.Offset = {
+ delegate match {
+ case _: Source => null
+ case m: MicroBatchStream => m.initialOffset
+ }
+ }
Review comment:
We can use null (for v1 sources) and iniitalOffset (for v2 streams) for the startOffset in latestOffset(startOffset, readLimit), since the readLimit is always allAvailable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908312697
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47358/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-908283948
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142853/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-906011090
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47276/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909777317
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47399/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33763:
URL: https://github.com/apache/spark/pull/33763#discussion_r694645350
##########
File path: sql/core/src/main/java/org/apache/spark/sql/streaming/Trigger.java
##########
@@ -89,15 +90,23 @@ public static Trigger ProcessingTime(String interval) {
}
/**
- * A trigger that process only one batch of data in a streaming query then terminates
- * the query.
+ * A trigger that processes all available data in a single batch then terminates the query.
Review comment:
Maybe add "For better scalability, `AvailableNow` can be used alternatively to process it in multiple batches.".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33763: [SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33763:
URL: https://github.com/apache/spark/pull/33763#issuecomment-909045572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org