[GitHub] [kyuubi] turboFei commented on a diff in pull request #4271: Clarify incremental collect mode ignore `kyuubi.operation.result.max.rows`

["turboFei (via GitHub)" <gi...@apache.org>]

turboFei commented on code in PR #4271:
URL: https://github.com/apache/kyuubi/pull/4271#discussion_r1100910027


##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala:
##########
@@ -70,43 +70,40 @@ class ExecuteStatement(
       addOperationListener()
       result = spark.sql(statement)
 
-      iter =
-        if (incrementalCollect) {
-          info("Execute in incremental collect mode")
+      val resultMaxRows = spark.conf.getOption(OPERATION_RESULT_MAX_ROWS.key).map(_.toInt)
+        .getOrElse(session.sessionManager.getConf.get(OPERATION_RESULT_MAX_ROWS))
+      iter = if (incrementalCollect) {
+        if (resultMaxRows > 0) {
+          logger.warn(s"Ignore ${OPERATION_RESULT_MAX_ROWS.key} on incremental collect mode.")
+        }
+        val internalIterator = if (arrowEnabled) {
+          SparkDatasetHelper.toArrowBatchRdd(convertComplexType(result)).toLocalIterator
+        } else {
+          result.toLocalIterator().asScala
+        }
+        new IterableFetchIterator[Any](new Iterable[Any] {
+          override def iterator: Iterator[Any] = internalIterator

Review Comment:
   It changes the behavior.
   
   before:
   ```
   override def iterator: Iterator[Row] = result.toLocalIterator().asScala
   ```
   
   now:
   
   ```
   val internalIterator = result.toLocalIterator().asScala
   override def iterator: Iterator[Any] = internalIterator
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org