You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/08/24 03:19:20 UTC

[GitHub] [incubator-kyuubi] yanghua opened a new pull request #980: [KYUUBI 979] Introduce AbstractFrontendService and KyuubiFrontendService

yanghua opened a new pull request #980:
URL: https://github.com/apache/incubator-kyuubi/pull/980


   <!--
   Thanks for sending a pull request!
   
   Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
     2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
   -->
   
   ### _Why are the changes needed?_
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you add a feature, you can talk about the use case of it.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### _How was this patch tested?_
   - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
   
   - [ ] Add screenshots for manual tests if appropriate
   
   - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #980: [KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #980:
URL: https://github.com/apache/incubator-kyuubi/pull/980#issuecomment-904297418


   # [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/980?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#980](https://codecov.io/gh/apache/incubator-kyuubi/pull/980?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e67e37a) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/f1ae664910d6cbcf327e126f8bc96f4b8dabb13b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f1ae664) will **decrease** coverage by `0.03%`.
   > The diff coverage is `76.19%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-kyuubi/pull/980/graphs/tree.svg?width=650&height=150&src=pr&token=925D4tb9AH&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-kyuubi/pull/980?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master     #980      +/-   ##
   ============================================
   - Coverage     80.77%   80.74%   -0.04%     
     Complexity       11       11              
   ============================================
     Files           146      148       +2     
     Lines          5389     5406      +17     
     Branches        636      638       +2     
   ============================================
   + Hits           4353     4365      +12     
   - Misses          660      663       +3     
   - Partials        376      378       +2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/980?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...n/scala/org/apache/kyuubi/service/Serverable.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2aWNlL1NlcnZlcmFibGUuc2NhbGE=) | `50.00% <ø> (+5.55%)` | :arrow_up: |
   | [...rg/apache/kyuubi/engine/spark/SparkSQLEngine.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2VuZ2luZS9zcGFyay9TcGFya1NRTEVuZ2luZS5zY2FsYQ==) | `70.58% <50.00%> (-0.63%)` | :arrow_down: |
   | [...pache/kyuubi/service/AbstractFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2aWNlL0Fic3RyYWN0RnJvbnRlbmRTZXJ2aWNlLnNjYWxh) | `66.66% <66.66%> (ø)` | |
   | [...g/apache/kyuubi/server/KyuubiFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvS3l1dWJpRnJvbnRlbmRTZXJ2aWNlLnNjYWxh) | `76.92% <76.92%> (ø)` | |
   | [.../apache/kyuubi/service/ThriftFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2aWNlL1RocmlmdEZyb250ZW5kU2VydmljZS5zY2FsYQ==) | `97.00% <100.00%> (ø)` | |
   | [.../scala/org/apache/kyuubi/server/KyuubiServer.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvS3l1dWJpU2VydmVyLnNjYWxh) | `51.28% <100.00%> (+1.28%)` | :arrow_up: |
   | [...ain/scala/org/apache/kyuubi/engine/EngineRef.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvRW5naW5lUmVmLnNjYWxh) | `76.11% <0.00%> (-1.50%)` | :arrow_down: |
   | [...org/apache/kyuubi/ha/client/ServiceDiscovery.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWhhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2hhL2NsaWVudC9TZXJ2aWNlRGlzY292ZXJ5LnNjYWxh) | `45.52% <0.00%> (-0.82%)` | :arrow_down: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/980?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/980?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f1ae664...e67e37a](https://codecov.io/gh/apache/incubator-kyuubi/pull/980?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yanghua commented on pull request #980: [KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService

Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #980:
URL: https://github.com/apache/incubator-kyuubi/pull/980#issuecomment-904341983


   @yaooqinn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #980: [KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #980:
URL: https://github.com/apache/incubator-kyuubi/pull/980#discussion_r694598080



##########
File path: kyuubi-common/src/main/scala/org/apache/kyuubi/service/AbstractFrontendService.scala
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.service
+
+/**
+ * A basic abstraction for frontend services.
+ */
+abstract class AbstractFrontendService(name: String, be: BackendService)
+  extends CompositeService(name) {
+
+  def connectionUrl(server: Boolean = false): String

Review comment:
       shouldn't it be a map?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yanghua commented on a change in pull request #980: [KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService

Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #980:
URL: https://github.com/apache/incubator-kyuubi/pull/980#discussion_r694744542



##########
File path: externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
##########
@@ -33,15 +33,17 @@ import org.apache.kyuubi.engine.spark.SparkSQLEngine.countDownLatch
 import org.apache.kyuubi.engine.spark.events.{EngineEvent, EventLoggingService}
 import org.apache.kyuubi.ha.HighAvailabilityConf._
 import org.apache.kyuubi.ha.client.{EngineServiceDiscovery, RetryPolicies, ServiceDiscovery}
-import org.apache.kyuubi.service.{Serverable, Service, ServiceState}
+import org.apache.kyuubi.service.{Serverable, Service, ServiceState, ThriftFrontendService}
 import org.apache.kyuubi.util.SignalRegister
 
 case class SparkSQLEngine(spark: SparkSession) extends Serverable("SparkSQLEngine") {
 
   lazy val engineStatus: EngineEvent = EngineEvent(this)
 
+  private val OOMHook = new Runnable { override def run(): Unit = stop() }
   private val eventLogging = new EventLoggingService(this)
   override val backendService = new SparkSQLBackendService(spark)
+  override val frontendService = new ThriftFrontendService(backendService, OOMHook)

Review comment:
       Will open a separated pr to remove the `server` flag.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yanghua commented on a change in pull request #980: [KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService

Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #980:
URL: https://github.com/apache/incubator-kyuubi/pull/980#discussion_r694619833



##########
File path: externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
##########
@@ -33,15 +33,17 @@ import org.apache.kyuubi.engine.spark.SparkSQLEngine.countDownLatch
 import org.apache.kyuubi.engine.spark.events.{EngineEvent, EventLoggingService}
 import org.apache.kyuubi.ha.HighAvailabilityConf._
 import org.apache.kyuubi.ha.client.{EngineServiceDiscovery, RetryPolicies, ServiceDiscovery}
-import org.apache.kyuubi.service.{Serverable, Service, ServiceState}
+import org.apache.kyuubi.service.{Serverable, Service, ServiceState, ThriftFrontendService}
 import org.apache.kyuubi.util.SignalRegister
 
 case class SparkSQLEngine(spark: SparkSession) extends Serverable("SparkSQLEngine") {
 
   lazy val engineStatus: EngineEvent = EngineEvent(this)
 
+  private val OOMHook = new Runnable { override def run(): Unit = stop() }
   private val eventLogging = new EventLoggingService(this)
   override val backendService = new SparkSQLBackendService(spark)
+  override val frontendService = new ThriftFrontendService(backendService, OOMHook)

Review comment:
       sounds good.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yanghua commented on a change in pull request #980: [KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService

Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #980:
URL: https://github.com/apache/incubator-kyuubi/pull/980#discussion_r694619073



##########
File path: kyuubi-common/src/main/scala/org/apache/kyuubi/service/AbstractFrontendService.scala
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.service
+
+/**
+ * A basic abstraction for frontend services.
+ */
+abstract class AbstractFrontendService(name: String, be: BackendService)
+  extends CompositeService(name) {
+
+  def connectionUrl(server: Boolean = false): String

Review comment:
       It would be a map, of cause. Currently, we have not introduced the `RestFrontendService` and relevant config options.
   This PR only introduced a framework for the abstraction, e.g. we only let `KyuubiFrontendService` delegate thrift front service.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yaooqinn closed pull request #980: [KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService

Posted by GitBox <gi...@apache.org>.
yaooqinn closed pull request #980:
URL: https://github.com/apache/incubator-kyuubi/pull/980


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #980: [KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #980:
URL: https://github.com/apache/incubator-kyuubi/pull/980#discussion_r694604433



##########
File path: externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
##########
@@ -33,15 +33,17 @@ import org.apache.kyuubi.engine.spark.SparkSQLEngine.countDownLatch
 import org.apache.kyuubi.engine.spark.events.{EngineEvent, EventLoggingService}
 import org.apache.kyuubi.ha.HighAvailabilityConf._
 import org.apache.kyuubi.ha.client.{EngineServiceDiscovery, RetryPolicies, ServiceDiscovery}
-import org.apache.kyuubi.service.{Serverable, Service, ServiceState}
+import org.apache.kyuubi.service.{Serverable, Service, ServiceState, ThriftFrontendService}
 import org.apache.kyuubi.util.SignalRegister
 
 case class SparkSQLEngine(spark: SparkSession) extends Serverable("SparkSQLEngine") {
 
   lazy val engineStatus: EngineEvent = EngineEvent(this)
 
+  private val OOMHook = new Runnable { override def run(): Unit = stop() }
   private val eventLogging = new EventLoggingService(this)
   override val backendService = new SparkSQLBackendService(spark)
+  override val frontendService = new ThriftFrontendService(backendService, OOMHook)

Review comment:
       we can create a `SparkSQLFrontendService` like `SparkSQLBackendService`, and we can remove the `server` flag which is a bit hacky




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org