You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/07/29 05:53:34 UTC

[GitHub] [incubator-kyuubi] zhang1002 opened a new pull request #873: Dump statementInfo into local file that the format is parquet

zhang1002 opened a new pull request #873:
URL: https://github.com/apache/incubator-kyuubi/pull/873


   ### _Why are the changes needed?_
   We need to dump data from mem into file regularly.
   This PR is just for local file, the file's format is parquet.
   
   Notice:
   1. There will store lots of small files if we don't use data-lake
   2. Need to discuss how to set partition for this schema
   3. If the schema is changed, how to query the old data
   
   
   ### _How was this patch tested?_
   - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
   
   - [ ] Add screenshots for manual tests if appropriate
   
   - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #873: Dump statementInfo into local file that the format is parquet

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #873:
URL: https://github.com/apache/incubator-kyuubi/pull/873#issuecomment-889672529


   # [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/873?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#873](https://codecov.io/gh/apache/incubator-kyuubi/pull/873?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0afaa57) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/6c3b722174e0d046ee7701b27249fcf3f4267a92?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6c3b722) will **not change** coverage.
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 0afaa57 differs from pull request most recent head bee3653. Consider uploading reports for the commit bee3653 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-kyuubi/pull/873/graphs/tree.svg?width=650&height=150&src=pr&token=925D4tb9AH&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-kyuubi/pull/873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff            @@
   ##             master     #873   +/-   ##
   =========================================
     Coverage     78.77%   78.77%           
     Complexity       10       10           
   =========================================
     Files           137      137           
     Lines          5187     5187           
     Branches        644      644           
   =========================================
     Hits           4086     4086           
     Misses          747      747           
     Partials        354      354           
   ```
   
   
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/873?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/873?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6c3b722...bee3653](https://codecov.io/gh/apache/incubator-kyuubi/pull/873?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #873: Dump statementInfo into local file that the format is parquet

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #873:
URL: https://github.com/apache/incubator-kyuubi/pull/873#discussion_r679707835



##########
File path: externals/kyuubi-spark-monitor/src/main/scala/org/apache/kyuubi/engine/spark/monitor/KyuubiStatementMonitor.scala
##########
@@ -69,22 +101,40 @@ object KyuubiStatementMonitor extends Logging{
    */
   // TODO: Lack size type threshold and time type threshold
   def putStatementInfoIntoQueue(kyuubiStatementInfo: KyuubiStatementInfo): Unit = {
-    if (kyuubiStatementQueue.size() >= maxSize) {
-      removeAndDumpStatementInfoFromQueue()
+    if (dumpEnable) {
+      if (kyuubiStatementQueue.size() >= maxSize) {
+        dumpStatementInfoFromQueue()
+      }
+      if (!kyuubiStatementQueue.contains(kyuubiStatementInfo)) {
+        val isSuccess = kyuubiStatementQueue.add(kyuubiStatementInfo)
+        info(s"Add kyuubiStatementInfo into queue is [$isSuccess], " +
+          s"statementId is [${kyuubiStatementInfo.statementId}]")
+      }
     }
-    val isSuccess = kyuubiStatementQueue.add(kyuubiStatementInfo)
-    info(s"Add kyuubiStatementInfo into queue is [$isSuccess], " +
-      s"statementId is [${kyuubiStatementInfo.statementId}]")
   }
 
   /**
-   * This method is used for removing kyuubiStatementInfo from blockingQueue(statementQueue)
-   * and dumpping them to a file by threshold.
+   * This method is used for dumpping kyuubiStatementInfo to a file by threshold.
    */
-  // TODO: Need ensure those items have finished. If not, we should put them into this queue again.
-  private def removeAndDumpStatementInfoFromQueue(): Unit = {
-    // TODO: Just for test
-    kyuubiStatementQueue.clear()
+  private def dumpStatementInfoFromQueue(): Unit = kyuubiStatementQueue.synchronized {
+    var statementInfoList: List[KyuubiStatementInfo] = List()
+    val size = kyuubiStatementQueue.size()
+    (0 to size-1).foreach { _ =>
+      statementInfoList = kyuubiStatementQueue.poll() :: statementInfoList
+    }
+    var schema = classToStructTyeMap.get(classOf[KyuubiStatementInfo])
+    if (schema == null) {
+      schema = ScalaReflection.schemaFor[KyuubiStatementInfo].dataType.asInstanceOf[StructType]
+      classToStructTyeMap.put(classOf[KyuubiStatementInfo], schema)
+    }
+    val rowList = new java.util.ArrayList[Row]()
+    for (kyuubiStatementInfo <- statementInfoList) {
+      rowList.add(Row.fromTuple(kyuubiStatementInfo))
+    }
+    sparkSession.createDataFrame(rowList, schema)
+      .write.mode(SaveMode.Append)
+      .format("parquet")
+      .save(dumpLocalDir + "/statement_info.parquet")

Review comment:
       this param is a path not a file name

##########
File path: externals/kyuubi-spark-monitor/src/main/scala/org/apache/kyuubi/engine/spark/monitor/KyuubiStatementMonitor.scala
##########
@@ -69,22 +101,40 @@ object KyuubiStatementMonitor extends Logging{
    */
   // TODO: Lack size type threshold and time type threshold
   def putStatementInfoIntoQueue(kyuubiStatementInfo: KyuubiStatementInfo): Unit = {
-    if (kyuubiStatementQueue.size() >= maxSize) {
-      removeAndDumpStatementInfoFromQueue()
+    if (dumpEnable) {
+      if (kyuubiStatementQueue.size() >= maxSize) {
+        dumpStatementInfoFromQueue()
+      }
+      if (!kyuubiStatementQueue.contains(kyuubiStatementInfo)) {
+        val isSuccess = kyuubiStatementQueue.add(kyuubiStatementInfo)
+        info(s"Add kyuubiStatementInfo into queue is [$isSuccess], " +
+          s"statementId is [${kyuubiStatementInfo.statementId}]")
+      }
     }
-    val isSuccess = kyuubiStatementQueue.add(kyuubiStatementInfo)
-    info(s"Add kyuubiStatementInfo into queue is [$isSuccess], " +
-      s"statementId is [${kyuubiStatementInfo.statementId}]")
   }
 
   /**
-   * This method is used for removing kyuubiStatementInfo from blockingQueue(statementQueue)
-   * and dumpping them to a file by threshold.
+   * This method is used for dumpping kyuubiStatementInfo to a file by threshold.
    */
-  // TODO: Need ensure those items have finished. If not, we should put them into this queue again.
-  private def removeAndDumpStatementInfoFromQueue(): Unit = {
-    // TODO: Just for test
-    kyuubiStatementQueue.clear()
+  private def dumpStatementInfoFromQueue(): Unit = kyuubiStatementQueue.synchronized {
+    var statementInfoList: List[KyuubiStatementInfo] = List()
+    val size = kyuubiStatementQueue.size()
+    (0 to size-1).foreach { _ =>
+      statementInfoList = kyuubiStatementQueue.poll() :: statementInfoList
+    }
+    var schema = classToStructTyeMap.get(classOf[KyuubiStatementInfo])
+    if (schema == null) {
+      schema = ScalaReflection.schemaFor[KyuubiStatementInfo].dataType.asInstanceOf[StructType]
+      classToStructTyeMap.put(classOf[KyuubiStatementInfo], schema)

Review comment:
       can we move this code as static code block ?

##########
File path: externals/kyuubi-spark-monitor/src/main/scala/org/apache/kyuubi/engine/spark/monitor/KyuubiStatementMonitor.scala
##########
@@ -17,27 +17,59 @@
 
 package org.apache.kyuubi.engine.spark.monitor
 
-import java.util.concurrent.{ArrayBlockingQueue, ConcurrentHashMap}
+import java.util.concurrent.{ArrayBlockingQueue, ConcurrentHashMap, ScheduledThreadPoolExecutor, TimeUnit}
 
 import org.apache.spark.scheduler.SparkListenerJobEnd
+import org.apache.spark.sql.{Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.ScalaReflection
+import org.apache.spark.sql.types.StructType
 
-import org.apache.kyuubi.Logging
+import org.apache.kyuubi.{Logging, Utils}
+import org.apache.kyuubi.config.KyuubiConf
 import org.apache.kyuubi.engine.spark.monitor.entity.{KyuubiJobInfo, KyuubiStatementInfo}
 
 // TODO: Thread Safe need to consider
 object KyuubiStatementMonitor extends Logging{
 
-  // TODO: Just for test. We will remove them in the future
-  private val maxCapacity: Int = 10
-  private val maxSize: Int = 7
+  private val classToStructTyeMap = new java.util.HashMap[Class[_], StructType]()
+
+  import org.apache.kyuubi.config.KyuubiConf._
+
+  private val sparkSession = SparkSession.active
+  private val kyuubiConf: KyuubiConf = KyuubiConf()
+  private val maxCapacity = kyuubiConf.get(SPARK_MONITOR_STATEMENTINFO_QUEUE_CAPACITY)
+  private val timeThreshold = kyuubiConf.get(SPARK_MONITOR_TIME_THRESHOLD)  // seconds
+  private val dumpEnable = kyuubiConf.get(SPARK_MONITOR_DUMP_LOCAL_ENABLE)
+  private val dumpLocalDir = kyuubiConf.get(SPARK_MONITOR_DUMP_LOCAL_DIR) match {
+    case null =>
+      if (Utils.isTesting) {
+        "file://" + System.getProperty("user.dir") + "/target/dump_data"
+      } else {
+        "file://" + System.getProperty("user.dir") + "/dump_data"
+      }
+    case dir: String =>
+      "file://" + dir
+  }
+
+  /**
+   * The max size that trigger to dump data is 0.7*maxCapacity
+   */
+  private val maxSize: Int = scala.math.floor(0.7*maxCapacity).intValue()
+
+  if (dumpEnable) {
+    val dumpDataByTimeThresholdThreadPool = new ScheduledThreadPoolExecutor(1)
+    dumpDataByTimeThresholdThreadPool.scheduleWithFixedDelay(new Runnable {
+      override def run(): Unit = {
+        dumpStatementInfoFromQueue()
+      }
+    }, 60, timeThreshold, TimeUnit.SECONDS)

Review comment:
       `}, timeThreshold, timeThreshold, TimeUnit.SECONDS)`

##########
File path: externals/kyuubi-spark-monitor/src/main/scala/org/apache/kyuubi/engine/spark/monitor/KyuubiStatementMonitor.scala
##########
@@ -17,27 +17,59 @@
 
 package org.apache.kyuubi.engine.spark.monitor
 
-import java.util.concurrent.{ArrayBlockingQueue, ConcurrentHashMap}
+import java.util.concurrent.{ArrayBlockingQueue, ConcurrentHashMap, ScheduledThreadPoolExecutor, TimeUnit}
 
 import org.apache.spark.scheduler.SparkListenerJobEnd
+import org.apache.spark.sql.{Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.ScalaReflection
+import org.apache.spark.sql.types.StructType
 
-import org.apache.kyuubi.Logging
+import org.apache.kyuubi.{Logging, Utils}
+import org.apache.kyuubi.config.KyuubiConf
 import org.apache.kyuubi.engine.spark.monitor.entity.{KyuubiJobInfo, KyuubiStatementInfo}
 
 // TODO: Thread Safe need to consider
 object KyuubiStatementMonitor extends Logging{
 
-  // TODO: Just for test. We will remove them in the future
-  private val maxCapacity: Int = 10
-  private val maxSize: Int = 7
+  private val classToStructTyeMap = new java.util.HashMap[Class[_], StructType]()
+
+  import org.apache.kyuubi.config.KyuubiConf._
+
+  private val sparkSession = SparkSession.active
+  private val kyuubiConf: KyuubiConf = KyuubiConf()
+  private val maxCapacity = kyuubiConf.get(SPARK_MONITOR_STATEMENTINFO_QUEUE_CAPACITY)
+  private val timeThreshold = kyuubiConf.get(SPARK_MONITOR_TIME_THRESHOLD)  // seconds
+  private val dumpEnable = kyuubiConf.get(SPARK_MONITOR_DUMP_LOCAL_ENABLE)
+  private val dumpLocalDir = kyuubiConf.get(SPARK_MONITOR_DUMP_LOCAL_DIR) match {
+    case null =>
+      if (Utils.isTesting) {
+        "file://" + System.getProperty("user.dir") + "/target/dump_data"
+      } else {
+        "file://" + System.getProperty("user.dir") + "/dump_data"
+      }
+    case dir: String =>
+      "file://" + dir
+  }
+
+  /**
+   * The max size that trigger to dump data is 0.7*maxCapacity
+   */
+  private val maxSize: Int = scala.math.floor(0.7*maxCapacity).intValue()

Review comment:
       `maxStatementSizeInMemory`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #873: Dump statementInfo into local file that the format is parquet

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #873:
URL: https://github.com/apache/incubator-kyuubi/pull/873#discussion_r678848536



##########
File path: externals/kyuubi-spark-monitor/src/main/scala/org/apache/kyuubi/engine/spark/monitor/KyuubiStatementMonitor.scala
##########
@@ -69,22 +98,38 @@ object KyuubiStatementMonitor extends Logging{
    */
   // TODO: Lack size type threshold and time type threshold
   def putStatementInfoIntoQueue(kyuubiStatementInfo: KyuubiStatementInfo): Unit = {
-    if (kyuubiStatementQueue.size() >= maxSize) {
-      removeAndDumpStatementInfoFromQueue()
+    if (dumpEnable && kyuubiStatementQueue.size() >= maxSize) {
+      dumpStatementInfoFromQueue()
+      val isSuccess = kyuubiStatementQueue.add(kyuubiStatementInfo)
+      info(s"Add kyuubiStatementInfo into queue is [$isSuccess], " +
+        s"statementId is [${kyuubiStatementInfo.statementId}]")
     }
-    val isSuccess = kyuubiStatementQueue.add(kyuubiStatementInfo)
-    info(s"Add kyuubiStatementInfo into queue is [$isSuccess], " +
-      s"statementId is [${kyuubiStatementInfo.statementId}]")
   }
 
   /**
-   * This method is used for removing kyuubiStatementInfo from blockingQueue(statementQueue)
-   * and dumpping them to a file by threshold.
+   * This method is used for dumpping kyuubiStatementInfo to a file by threshold.
    */
-  // TODO: Need ensure those items have finished. If not, we should put them into this queue again.
-  private def removeAndDumpStatementInfoFromQueue(): Unit = {
-    // TODO: Just for test
-    kyuubiStatementQueue.clear()
+  private def dumpStatementInfoFromQueue(): Unit = kyuubiStatementQueue.synchronized {
+    var statementInfoList = new util.ArrayList[KyuubiStatementInfo]()
+    val size = kyuubiStatementQueue.size()
+    (0 to size-1).foreach { _ =>
+      statementInfoList.add(kyuubiStatementQueue.poll())
+    }
+    if (DataFrameUtil.getStructType(classOf[KyuubiStatementInfo]).nonEmpty) {

Review comment:
       org.apache.spark.sql.Encoders.product[xxx].schema?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-kyuubi] zhang1002 closed pull request #873: Dump statementInfo into local file that the format is parquet

Posted by GitBox <gi...@apache.org>.
zhang1002 closed pull request #873:
URL: https://github.com/apache/incubator-kyuubi/pull/873


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org