You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "hadoopkandy (via GitHub)" <gi...@apache.org> on 2024/01/10 09:39:24 UTC
[PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
hadoopkandy opened a new pull request, #5960:
URL: https://github.com/apache/kyuubi/pull/5960
Flink engine should return the result when fetch timeout but already fetched some data
# :mag: Description
## Issue References 🔗
<!-- Append the issue number after #. If there is no issue for you to link create one or -->
<!-- If there are no issues to link, please provide details here. -->
This pull request fixes #
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes :bookmark:
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklist 📝
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "hadoopkandy (via GitHub)" <gi...@apache.org>.
hadoopkandy commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-1893000200
> behavior
As I understand ，Streaming query does't have that switch. Considering such case, if there are many rows data in Kafka, we use query from kafka with limit 1 , should we return one row data or timeout ? For jdbc client,I think we shoud return one row. Considering of such situation，when timeout is reached, hasNext should be false as the streaming query is not finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "link3280 (via GitHub)" <gi...@apache.org>.
link3280 commented on code in PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#discussion_r1447256405
##########
externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala:
##########
@@ -1282,6 +1282,26 @@ abstract class FlinkOperationSuite extends HiveJDBCTestHelper with WithFlinkTest
assert(exception.getMessage === "Futures timed out after [60000 milliseconds]")
}
+ test("test result fetch timeout but already fetched some data") {
+ withSessionConf()(Map(
+ ENGINE_FLINK_MAX_ROWS.key -> "10",
+ ENGINE_FLINK_FETCH_TIMEOUT.key -> "PT5S"))(Map.empty) {
+ withJdbcStatement() { statement =>
+ statement.execute("create table tbl_src_a (a bigint) with (" +
Review Comment:
Why do we need 2 tables in the test?
##########
externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala:
##########
@@ -1282,6 +1282,26 @@ abstract class FlinkOperationSuite extends HiveJDBCTestHelper with WithFlinkTest
assert(exception.getMessage === "Futures timed out after [60000 milliseconds]")
}
+ test("test result fetch timeout but already fetched some data") {
+ withSessionConf()(Map(
+ ENGINE_FLINK_MAX_ROWS.key -> "10",
+ ENGINE_FLINK_FETCH_TIMEOUT.key -> "PT5S"))(Map.empty) {
+ withJdbcStatement() { statement =>
+ statement.execute("create table tbl_src_a (a bigint) with (" +
+ "'connector' = 'datagen', 'rows-per-second' = '1','number-of-rows' = '1')")
+ statement.execute("create table tbl_src_b (a bigint) with (" +
+ "'connector' = 'datagen', 'rows-per-second' = '0')")
+ val resultSet = statement.executeQuery(s"select a from tbl_src_a " +
+ s"union all select a from tbl_src_b")
+ var rows = 0
+ while (resultSet.next()) {
+ rows += 1
+ }
+ assert(rows === 1)
Review Comment:
Before the PR, the rows would be 0, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "hadoopkandy (via GitHub)" <gi...@apache.org>.
hadoopkandy commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-1893000696
> I'm not very familiar with the streaming process, does Flink/Beam have such a switch to control the behavior of timeout reached but query not completed?
As I understand ，Streaming query does't have that switch. Considering such case, if there are many rows data in Kafka, we use query from kafka with limit 1 , should we return one row data or timeout ? For jdbc client,I think we shoud return one row. Considering of such situation，when timeout is reached, hasNext should be false as the streaming query is not finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "hadoopkandy (via GitHub)" <gi...@apache.org>.
hadoopkandy commented on code in PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#discussion_r1447456576
##########
externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala:
##########
@@ -1282,6 +1282,26 @@ abstract class FlinkOperationSuite extends HiveJDBCTestHelper with WithFlinkTest
assert(exception.getMessage === "Futures timed out after [60000 milliseconds]")
}
+ test("test result fetch timeout but already fetched some data") {
+ withSessionConf()(Map(
+ ENGINE_FLINK_MAX_ROWS.key -> "10",
+ ENGINE_FLINK_FETCH_TIMEOUT.key -> "PT5S"))(Map.empty) {
+ withJdbcStatement() { statement =>
+ statement.execute("create table tbl_src_a (a bigint) with (" +
Review Comment:
one is bounded data ,it only have one rows, the other is unbounded data, the union query is just like we query from kafka,but it have only one rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "link3280 (via GitHub)" <gi...@apache.org>.
link3280 commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-1888739495
Thanks for updating! The PR looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-1895635401
the cases are different, if limit 1 is applied, once 1 record is reached, the query is finished, no matter streaming or batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "hadoopkandy (via GitHub)" <gi...@apache.org>.
hadoopkandy commented on code in PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#discussion_r1447460610
##########
externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala:
##########
@@ -1282,6 +1282,26 @@ abstract class FlinkOperationSuite extends HiveJDBCTestHelper with WithFlinkTest
assert(exception.getMessage === "Futures timed out after [60000 milliseconds]")
}
+ test("test result fetch timeout but already fetched some data") {
+ withSessionConf()(Map(
+ ENGINE_FLINK_MAX_ROWS.key -> "10",
+ ENGINE_FLINK_FETCH_TIMEOUT.key -> "PT5S"))(Map.empty) {
+ withJdbcStatement() { statement =>
+ statement.execute("create table tbl_src_a (a bigint) with (" +
+ "'connector' = 'datagen', 'rows-per-second' = '1','number-of-rows' = '1')")
+ statement.execute("create table tbl_src_b (a bigint) with (" +
+ "'connector' = 'datagen', 'rows-per-second' = '0')")
+ val resultSet = statement.executeQuery(s"select a from tbl_src_a " +
+ s"union all select a from tbl_src_b")
+ var rows = 0
+ while (resultSet.next()) {
+ rows += 1
+ }
+ assert(rows === 1)
Review Comment:
Futures timed out after [5000 milliseconds]
##########
externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala:
##########
@@ -1282,6 +1282,26 @@ abstract class FlinkOperationSuite extends HiveJDBCTestHelper with WithFlinkTest
assert(exception.getMessage === "Futures timed out after [60000 milliseconds]")
}
+ test("test result fetch timeout but already fetched some data") {
+ withSessionConf()(Map(
+ ENGINE_FLINK_MAX_ROWS.key -> "10",
+ ENGINE_FLINK_FETCH_TIMEOUT.key -> "PT5S"))(Map.empty) {
+ withJdbcStatement() { statement =>
+ statement.execute("create table tbl_src_a (a bigint) with (" +
+ "'connector' = 'datagen', 'rows-per-second' = '1','number-of-rows' = '1')")
+ statement.execute("create table tbl_src_b (a bigint) with (" +
+ "'connector' = 'datagen', 'rows-per-second' = '0')")
+ val resultSet = statement.executeQuery(s"select a from tbl_src_a " +
+ s"union all select a from tbl_src_b")
+ var rows = 0
+ while (resultSet.next()) {
+ rows += 1
+ }
+ assert(rows === 1)
Review Comment:
Futures timed out after [5000 milliseconds]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-1891270476
when timeout is reached, `hasNext` should be `true` as the streaming query is not finished. `beeline` supports `--incremental=true` to print the fetched records immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "link3280 (via GitHub)" <gi...@apache.org>.
link3280 commented on code in PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#discussion_r1447274098
##########
externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/result/IncrementalResultFetchIterator.scala:
##########
@@ -125,7 +125,16 @@ class IncrementalResultFetchIterator(
Thread.sleep(FETCH_INTERVAL_MS)
}
})
- Await.result(future, resultFetchTimeout)
+ try {
+ Await.result(future, resultFetchTimeout)
+ } catch {
+ case e: TimeoutException =>
Review Comment:
We need a `case _ ` to cover other exceptions. Simply rethrowing them would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "link3280 (via GitHub)" <gi...@apache.org>.
link3280 commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-1902033382
> I'm not very familiar with the streaming process, does Flink/Beam have such a switch to control the behavior of timeout reached but query not completed?
Flink SQL Gateway exits the fetch loop and sets `hasMoreResults` to `true` once there's a fetch that returns no data. That's, like we discussed before, senmatically incorrect. See https://github.com/apache/flink/blob/fd673a2f46206ff65978f05fcb96b525696aead2/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/result/ResultFetcher.java#L265 .
I think we could add a option to make fetch timeout as EOS, while keep the current behavior as default. WDTY? @pan3793 @hadoopkandy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-1886141017
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5960?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`698db5b`)](https://app.codecov.io/gh/apache/kyuubi/commit/698db5b848daae4e373929cf8b151b4023ddea64?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.29% compared to head [(`f4d0a09`)](https://app.codecov.io/gh/apache/kyuubi/pull/5960?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.23%.
> Report is 1 commits behind head on master.
> :exclamation: Current head f4d0a09 differs from pull request most recent head 6cc0846. Consider uploading reports for the commit 6cc0846 to get more accurate results
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #5960 +/- ##
============================================
- Coverage 61.29% 61.23% -0.06%
Complexity 23 23
============================================
Files 622 622
Lines 36882 36882
Branches 5014 5014
============================================
- Hits 22606 22586 -20
- Misses 11850 11864 +14
- Partials 2426 2432 +6
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/5960?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "link3280 (via GitHub)" <gi...@apache.org>.
link3280 commented on code in PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#discussion_r1447274098
##########
externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/result/IncrementalResultFetchIterator.scala:
##########
@@ -125,7 +125,16 @@ class IncrementalResultFetchIterator(
Thread.sleep(FETCH_INTERVAL_MS)
}
})
- Await.result(future, resultFetchTimeout)
+ try {
+ Await.result(future, resultFetchTimeout)
+ } catch {
+ case e: TimeoutException =>
Review Comment:
We need a `case _ ` to cover other exceptions. Simply rethrow them would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-2798252769
Thanks for the PR! This PR is being closed due to inactivity. This isn't a judgement on the merit of the PR in any way. If this is still an issue with the latest version of Kyuubi, please reopen it and ask a committer to remove the Stale tag!
Thank you for using Kyuubi!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "hadoopkandy (via GitHub)" <gi...@apache.org>.
hadoopkandy commented on code in PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#discussion_r1447460610
##########
externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala:
##########
@@ -1282,6 +1282,26 @@ abstract class FlinkOperationSuite extends HiveJDBCTestHelper with WithFlinkTest
assert(exception.getMessage === "Futures timed out after [60000 milliseconds]")
}
+ test("test result fetch timeout but already fetched some data") {
+ withSessionConf()(Map(
+ ENGINE_FLINK_MAX_ROWS.key -> "10",
+ ENGINE_FLINK_FETCH_TIMEOUT.key -> "PT5S"))(Map.empty) {
+ withJdbcStatement() { statement =>
+ statement.execute("create table tbl_src_a (a bigint) with (" +
+ "'connector' = 'datagen', 'rows-per-second' = '1','number-of-rows' = '1')")
+ statement.execute("create table tbl_src_b (a bigint) with (" +
+ "'connector' = 'datagen', 'rows-per-second' = '0')")
+ val resultSet = statement.executeQuery(s"select a from tbl_src_a " +
+ s"union all select a from tbl_src_b")
+ var rows = 0
+ while (resultSet.next()) {
+ rows += 1
+ }
+ assert(rows === 1)
Review Comment:
Futures timed out after [5000 milliseconds]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-1891277882
I'm not very familiar with the streaming process, does Flink/Beam have such a switch to control the behavior of timeout reached but query not completed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "link3280 (via GitHub)" <gi...@apache.org>.
link3280 commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-1891271650
> when timeout is reached, `hasNext` should be `true` as the streaming query is not finished. `beeline` supports `--incremental=true` to print the fetched records immediately.
How about adding a `exit-on-fetch-timeout` option to Flink engine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "Gooch0922 (via GitHub)" <gi...@apache.org>.
Gooch0922 commented on PR #5960:
URL: https://github.com/apache/kyuubi/pull/5960#issuecomment-2997011588
For REST APIs and streaming queries, when calling the {operationHandle}/rowset interface, users can specify an optional parameter (similar to KyuubiConf.OPERATION_INCREMENTAL_COLLECT) to decide whether to return cached data. The specific implementation approach is to modify the IncrementalResultFetchIterator so that instead of blocking to fetch results from Flink, a single-threaded scheduler periodically retrieves data from Flink's Gateway API. This allows the system to determine if there is sufficient data available. If not, it can directly return the data that is already cached. @pan3793 cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5960: [KYUUBI #5959][Bug] Flink engine fetch the unbounded data timeout
URL: https://github.com/apache/kyuubi/pull/5960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org