You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "turboFei (via GitHub)" <gi...@apache.org> on 2024/12/01 19:49:41 UTC
[PR] Batch pending time [kyuubi]
turboFei opened a new pull request, #6829:
URL: https://github.com/apache/kyuubi/pull/6829
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/master/contributing/code/index.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug, and what versions are affected.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If a generative AI tooling has been used in the process of authoring this patch, please include
phrase 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #6829: Add metrics for batch pending max elapse time
URL: https://github.com/apache/kyuubi/pull/6829
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866690901
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala:
##########
@@ -257,14 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata] = {
+ desc: Boolean = false,
+ orderByKeyId: Boolean = true): Seq[Metadata] = {
val queryBuilder = new StringBuilder
val params = ListBuffer[Any]()
queryBuilder.append("SELECT ")
queryBuilder.append(METADATA_COLUMNS)
queryBuilder.append(s" FROM $METADATA_TABLE")
queryBuilder.append(s" ${assembleWhereClause(filter, params)}")
- queryBuilder.append(" ORDER BY key_id ")
+ queryBuilder.append(" ORDER BY ").append(if (orderByKeyId) "key_id " else "create_time ")
Review Comment:
It is not guaranteed.
<img width="921" alt="image" src="https://github.com/user-attachments/assets/e5f09198-c2e2-410a-b17a-a66794c00611">
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala:
##########
@@ -257,14 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata] = {
+ desc: Boolean = false,
+ orderByKeyId: Boolean = true): Seq[Metadata] = {
val queryBuilder = new StringBuilder
val params = ListBuffer[Any]()
queryBuilder.append("SELECT ")
queryBuilder.append(METADATA_COLUMNS)
queryBuilder.append(s" FROM $METADATA_TABLE")
queryBuilder.append(s" ${assembleWhereClause(filter, params)}")
- queryBuilder.append(" ORDER BY key_id ")
+ queryBuilder.append(" ORDER BY ").append(if (orderByKeyId) "key_id " else "create_time ")
Review Comment:
Looks good:
<img width="1099" alt="image" src="https://github.com/user-attachments/assets/0d671de3-dae2-41f5-b5a8-f5a1df14959b">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#issuecomment-2512317662
thanks for the comments, will address it today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#issuecomment-2519850008
Thanks, merged to master/1.10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866692335
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala:
##########
@@ -257,14 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata] = {
+ desc: Boolean = false,
+ orderByKeyId: Boolean = true): Seq[Metadata] = {
val queryBuilder = new StringBuilder
val params = ListBuffer[Any]()
queryBuilder.append("SELECT ")
queryBuilder.append(METADATA_COLUMNS)
queryBuilder.append(s" FROM $METADATA_TABLE")
queryBuilder.append(s" ${assembleWhereClause(filter, params)}")
- queryBuilder.append(" ORDER BY key_id ")
+ queryBuilder.append(" ORDER BY ").append(if (orderByKeyId) "key_id " else "create_time ")
Review Comment:
But for small dataset, it looks good. I think we can have a try.
<img width="1166" alt="image" src="https://github.com/user-attachments/assets/ecdd76c6-b171-4b3e-9889-24e64a49e0cf">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866702053
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala:
##########
@@ -257,14 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata] = {
+ desc: Boolean = false,
+ orderByKeyId: Boolean = true): Seq[Metadata] = {
val queryBuilder = new StringBuilder
val params = ListBuffer[Any]()
queryBuilder.append("SELECT ")
queryBuilder.append(METADATA_COLUMNS)
queryBuilder.append(s" FROM $METADATA_TABLE")
queryBuilder.append(s" ${assembleWhereClause(filter, params)}")
- queryBuilder.append(" ORDER BY key_id ")
+ queryBuilder.append(" ORDER BY ").append(if (orderByKeyId) "key_id " else "create_time ")
Review Comment:
And the UT can not pass without `order by`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1867128022
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:
##########
@@ -2023,6 +2023,16 @@ object KyuubiConf {
.intConf
.createWithDefault(65536)
+ val METADATA_SEARCH_WINDOW: OptionalConfigEntry[Long] =
+ buildConf("kyuubi.metadata.search.window")
+ .doc("The time window to restrict user queries to metadata within a specific period. " +
+ "For example, if the window is set to P7D, only metadata from the past 7 days can be " +
+ "queried. If not set, it allows searching all metadata information in the metadata store.")
Review Comment:
the doc should declare the affected user-faced interfaces, for example, it only affects xxx API, and with suggestions like, you may want to set this to XXX when XXX to achieve XXX goals, also should include the side-effects if any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866106661
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:
##########
@@ -2023,6 +2023,14 @@ object KyuubiConf {
.intConf
.createWithDefault(65536)
+ val METADATA_SEARCH_WINDOW: OptionalConfigEntry[Long] =
+ buildConf("kyuubi.metadata.search.window")
+ .doc("The time window to search the metadata from metadata store.")
Review Comment:
from a user perspective, I have no idea what I should do with this conf after reading the docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866713021
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:
##########
@@ -2023,6 +2023,14 @@ object KyuubiConf {
.intConf
.createWithDefault(65536)
+ val METADATA_SEARCH_WINDOW: OptionalConfigEntry[Long] =
+ buildConf("kyuubi.metadata.search.window")
+ .doc("The time window to search the metadata from metadata store.")
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1867128022
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:
##########
@@ -2023,6 +2023,16 @@ object KyuubiConf {
.intConf
.createWithDefault(65536)
+ val METADATA_SEARCH_WINDOW: OptionalConfigEntry[Long] =
+ buildConf("kyuubi.metadata.search.window")
+ .doc("The time window to restrict user queries to metadata within a specific period. " +
+ "For example, if the window is set to P7D, only metadata from the past 7 days can be " +
+ "queried. If not set, it allows searching all metadata information in the metadata store.")
Review Comment:
the doc should declare the affected user-faced interfaces, for example, it only affects xxx API, and with suggestions like, you may want to set this to XXX when XXX to achieve XXX goals, also should mention the side-effects if any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866655856
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala:
##########
@@ -420,15 +421,24 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
s"The valid batch state can be one of the following: ${VALID_BATCH_STATES.mkString(",")}")
}
+ val createTimeFilter =
+ math.max(createTime, metadataSearchWindow.map(System.currentTimeMillis() - _).getOrElse(0L))
val filter = MetadataFilter(
sessionType = SessionType.BATCH,
engineType = batchType,
username = batchUser,
state = batchState,
requestName = batchName,
- createTime = createTime,
+ createTime = createTimeFilter,
endTime = endTime)
Review Comment:
Will address it in another PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866692335
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala:
##########
@@ -257,14 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata] = {
+ desc: Boolean = false,
+ orderByKeyId: Boolean = true): Seq[Metadata] = {
val queryBuilder = new StringBuilder
val params = ListBuffer[Any]()
queryBuilder.append("SELECT ")
queryBuilder.append(METADATA_COLUMNS)
queryBuilder.append(s" FROM $METADATA_TABLE")
queryBuilder.append(s" ${assembleWhereClause(filter, params)}")
- queryBuilder.append(" ORDER BY key_id ")
+ queryBuilder.append(" ORDER BY ").append(if (orderByKeyId) "key_id " else "create_time ")
Review Comment:
But for small dataset, it looks good. I think we can have a try.
<img width="1166" alt="image" src="https://github.com/user-attachments/assets/ecdd76c6-b171-4b3e-9889-24e64a49e0cf">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866623849
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala:
##########
@@ -257,14 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata] = {
+ desc: Boolean = false,
+ orderByKeyId: Boolean = true): Seq[Metadata] = {
val queryBuilder = new StringBuilder
val params = ListBuffer[Any]()
queryBuilder.append("SELECT ")
queryBuilder.append(METADATA_COLUMNS)
queryBuilder.append(s" FROM $METADATA_TABLE")
queryBuilder.append(s" ${assembleWhereClause(filter, params)}")
- queryBuilder.append(" ORDER BY key_id ")
+ queryBuilder.append(" ORDER BY ").append(if (orderByKeyId) "key_id " else "create_time ")
Review Comment:
Looks good:
<img width="1099" alt="image" src="https://github.com/user-attachments/assets/0d671de3-dae2-41f5-b5a8-f5a1df14959b">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1868151674
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:
##########
@@ -2023,6 +2023,16 @@ object KyuubiConf {
.intConf
.createWithDefault(65536)
+ val METADATA_SEARCH_WINDOW: OptionalConfigEntry[Long] =
+ buildConf("kyuubi.metadata.search.window")
+ .doc("The time window to restrict user queries to metadata within a specific period. " +
+ "For example, if the window is set to P7D, only metadata from the past 7 days can be " +
+ "queried. If not set, it allows searching all metadata information in the metadata store.")
Review Comment:
thanks, addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866092096
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala:
##########
@@ -420,15 +421,24 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
s"The valid batch state can be one of the following: ${VALID_BATCH_STATES.mkString(",")}")
}
+ val createTimeFilter =
+ math.max(createTime, metadataSearchWindow.map(System.currentTimeMillis() - _).getOrElse(0L))
val filter = MetadataFilter(
sessionType = SessionType.BATCH,
engineType = batchType,
username = batchUser,
state = batchState,
requestName = batchName,
- createTime = createTime,
+ createTime = createTimeFilter,
endTime = endTime)
Review Comment:
not related to this PR, but `createTime` and `endTime` are confusing, should be `minCreateTime` and `maxEndTime`, and say if they are inclusive or exclusive in the comments, with an explanation for specific values
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala:
##########
@@ -257,14 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata] = {
+ desc: Boolean = false,
+ orderByKeyId: Boolean = true): Seq[Metadata] = {
val queryBuilder = new StringBuilder
val params = ListBuffer[Any]()
queryBuilder.append("SELECT ")
queryBuilder.append(METADATA_COLUMNS)
queryBuilder.append(s" FROM $METADATA_TABLE")
queryBuilder.append(s" ${assembleWhereClause(filter, params)}")
- queryBuilder.append(" ORDER BY key_id ")
+ queryBuilder.append(" ORDER BY ").append(if (orderByKeyId) "key_id " else "create_time ")
Review Comment:
have you tried without `ORDER BY` clause? if the result ordering is stable (for example, ordered by pk), it might be the fastest approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866233445
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala:
##########
@@ -257,14 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata] = {
+ desc: Boolean = false,
+ orderByKeyId: Boolean = true): Seq[Metadata] = {
Review Comment:
the parameters are evolving to Spark's `SortOrder`
```
case class SortOrder(
child: Expression,
direction: SortDirection,
nullOrdering: NullOrdering,
sameOrderExpressions: Seq[Expression])
```
how about using some SQL terminologies to redesign the method's signature?
```
override def getMetadataList(
filter: MetadataFilter,
offset: Int,
limit: Int,
orderBy: Option[String],
direction: String = "ASC"): Seq[Metadata]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#issuecomment-2510266498
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/6829?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `43 lines` in your changes missing coverage. Please review.
> Project coverage is 0.00%. Comparing base [(`2ab2789`)](https://app.codecov.io/gh/apache/kyuubi/commit/2ab27892a4f651725ed2dd3df292a0cd805276a6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`1e93dd2`)](https://app.codecov.io/gh/apache/kyuubi/commit/1e93dd27635bc1c999d2b99325b9b8877bc7ca33?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/kyuubi/pull/6829?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&filepath=kyuubi-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fconfig%2FKyuubiConf.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | 0.00% | [12 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/kyuubi/server/KyuubiRestFrontendService.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fserver%2FKyuubiRestFrontendService.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvS3l1dWJpUmVzdEZyb250ZW5kU2VydmljZS5zY2FsYQ==) | 0.00% | [11 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/kyuubi/server/metadata/MetadataManager.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fserver%2Fmetadata%2FMetadataManager.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvTWV0YWRhdGFNYW5hZ2VyLnNjYWxh) | 0.00% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../apache/kyuubi/server/api/v1/BatchesResource.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fserver%2Fapi%2Fv1%2FBatchesResource.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvYXBpL3YxL0JhdGNoZXNSZXNvdXJjZS5zY2FsYQ==) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fserver%2Fmetadata%2Fjdbc%2FJDBCMetadataStore.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KREJDTWV0YWRhdGFTdG9yZS5zY2FsYQ==) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/kyuubi/session/KyuubiSessionManager.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fsession%2FKyuubiSessionManager.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXNzaW9uL0t5dXViaVNlc3Npb25NYW5hZ2VyLnNjYWxh) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../apache/kyuubi/server/metadata/MetadataStore.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fserver%2Fmetadata%2FMetadataStore.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvTWV0YWRhdGFTdG9yZS5zY2FsYQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...a/org/apache/kyuubi/metrics/MetricsConstants.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&filepath=kyuubi-metrics%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fmetrics%2FMetricsConstants.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLW1ldHJpY3Mvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvbWV0cmljcy9NZXRyaWNzQ29uc3RhbnRzLnNjYWxh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6829?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #6829 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 687 687
Lines 42439 42473 +34
Branches 5791 5795 +4
======================================
- Misses 42439 42473 +34
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/6829?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1867128022
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:
##########
@@ -2023,6 +2023,16 @@ object KyuubiConf {
.intConf
.createWithDefault(65536)
+ val METADATA_SEARCH_WINDOW: OptionalConfigEntry[Long] =
+ buildConf("kyuubi.metadata.search.window")
+ .doc("The time window to restrict user queries to metadata within a specific period. " +
+ "For example, if the window is set to P7D, only metadata from the past 7 days can be " +
+ "queried. If not set, it allows searching all metadata information in the metadata store.")
Review Comment:
the doc should declare the affected user-faced interfaces, for example, it only affects xxx API, and with suggestions like, you may want to set this to XXX when XXX, also should include the side-effects if any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Add metrics for batch pending max elapse time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6829:
URL: https://github.com/apache/kyuubi/pull/6829#discussion_r1866165469
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala:
##########
@@ -420,15 +421,24 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
s"The valid batch state can be one of the following: ${VALID_BATCH_STATES.mkString(",")}")
}
+ val createTimeFilter =
+ math.max(createTime, metadataSearchWindow.map(System.currentTimeMillis() - _).getOrElse(0L))
val filter = MetadataFilter(
sessionType = SessionType.BATCH,
engineType = batchType,
username = batchUser,
state = batchState,
requestName = batchName,
- createTime = createTime,
+ createTime = createTimeFilter,
endTime = endTime)
Review Comment:
update: the predication name might be more intuitive if we use `greatThanXXX` `lessThanOrEqualsXXX`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org