You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/06/07 01:55:28 UTC
[GitHub] [incubator-kyuubi] pan3793 opened a new issue, #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make splitPerTask configurable
pan3793 opened a new issue, #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues) and found no similar issues.
### Describe the subtask
Currently `rowCountPerTask` is hardcoded as 1 million, which impacts the parallelism and final impacts the performance.
```
private val rowCountPerTask: Int = 1000000
private val rowCount: Long = new Scaling(scale).getRowCount(table)
private val parallelism: Int =
if (table.isSmall) 1
else math.max(
SparkSession.active.sparkContext.defaultParallelism,
(rowCount / rowCountPerTask.toDouble).ceil.toInt)
```
We'd better make it configurable.
### Parent issue
https://github.com/apache/incubator-kyuubi/issues/2538
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make rowCountPerTask configurable
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543#issuecomment-1132427303
In practice, I notice that different table costs different time to generate one row, the `rowCountPerTask` maybe not a good idea.
Instead, I think we can introduce a concept `cu`(compute unit, and I'm open to the name, naming suggestion is welcome), 1 cu means the task should cost 1 second.
To achieve this, we need a benchmark for every table and calculate the generate rate(how many rows can be generated per second).
@wForget @cxzl25 WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make rowCountPerTask configurable
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543#issuecomment-1134206964
> Thanks @wForget, the benchmark results shows the huge performance difference over the tables, could you please create a benchmark test just like `ZorderCoreBenchmark`? Another thing is that the overhead of converting the raw record to `InternalRow` is non-negligible, the benchmark should cover that.
Great idea, I will try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make inputPartitionSize configurable
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543#issuecomment-1151021927
@wForget are you working on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make splitPerTask configurable
Posted by GitBox <gi...@apache.org>.
pan3793 closed issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make splitPerTask configurable
URL: https://github.com/apache/incubator-kyuubi/issues/2543
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make inputPartitionSize configurable
Posted by GitBox <gi...@apache.org>.
pan3793 closed issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make inputPartitionSize configurable
URL: https://github.com/apache/incubator-kyuubi/issues/2543
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make inputPartitionSize configurable
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543#issuecomment-1153217753
> are you working on this?
Yes, I was busy with other things these days, I will send a pr asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make rowCountPerTask configurable
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543#issuecomment-1134179467
Thanks @wForget, the benchmark results shows the huge performance difference over the tables, could you please create a benchmark test just like `ZorderCoreBenchmark`?
Another thing is that the overhead of converting the raw record to `InternalRow` is non-negligible, the benchmark should cover that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make rowCountPerTask configurable
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543#issuecomment-1133920370
Is it ok that I try to get the generate time like this?
```scala
test("test generate time") {
val scale = 100000
val session = Session.getDefaultSession.withScale(scale)
def generateTime(table: Table, rowCount: Long): Long = {
val iterator = new Results(table, 1, rowCount, session).iterator()
val start = System.currentTimeMillis()
while (iterator.hasNext) {
iterator.next()
}
System.currentTimeMillis() - start
}
Table.getBaseTables.asScala.filter { table =>
TPCDSStatisticsUtils.numRows(table, scale) > 1000000
}.map { table =>
val arr = Array(100000, 1000000, 10000000, 100000000)
.filter(_ < TPCDSStatisticsUtils.numRows(table, scale))
var time = 0L
var rowCount = 0
var i = 0
while (i < arr.size && time <= 10000) {
rowCount = arr(i)
time = generateTime(table, arr(i))
i += 1
}
val sizeInBytes = rowCount * TPCDSStatisticsUtils.TABLE_AVG_ROW_BYTES(table)
// scalastyle:off println
println(s"table: ${table.getName}, rowCount: $rowCount," +
s" time: $time, sizeInBytes: $sizeInBytes")
// scalastyle:on println
(table, rowCount, time)
}
}
```
results:
```
// first time
table: catalog_returns, rowCount: 1000000, time: 11237, sizeInBytes: 166000000
table: catalog_sales, rowCount: 1000000, time: 99161, sizeInBytes: 226000000
table: customer, rowCount: 10000000, time: 65169, sizeInBytes: 1320000000
table: customer_address, rowCount: 10000000, time: 41374, sizeInBytes: 1100000000
table: customer_demographics, rowCount: 1000000, time: 386, sizeInBytes: 42000000
table: inventory, rowCount: 100000000, time: 31076, sizeInBytes: 1600000000
table: store_returns, rowCount: 1000000, time: 13781, sizeInBytes: 134000000
table: store_sales, rowCount: 100000, time: 10703, sizeInBytes: 16400000
table: web_returns, rowCount: 1000000, time: 16766, sizeInBytes: 162000000
table: web_sales, rowCount: 100000, time: 14117, sizeInBytes: 22600000
// second time
table: catalog_returns, rowCount: 1000000, time: 11142, sizeInBytes: 166000000
table: catalog_sales, rowCount: 1000000, time: 98132, sizeInBytes: 226000000
table: customer, rowCount: 10000000, time: 67072, sizeInBytes: 1320000000
table: customer_address, rowCount: 10000000, time: 40335, sizeInBytes: 1100000000
table: customer_demographics, rowCount: 1000000, time: 401, sizeInBytes: 42000000
table: inventory, rowCount: 100000000, time: 31808, sizeInBytes: 1600000000
table: store_returns, rowCount: 1000000, time: 13498, sizeInBytes: 134000000
table: store_sales, rowCount: 100000, time: 10492, sizeInBytes: 16400000
table: web_returns, rowCount: 1000000, time: 15443, sizeInBytes: 162000000
table: web_sales, rowCount: 100000, time: 13794, sizeInBytes: 22600000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make inputPartitionSize configurable
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543#issuecomment-1153220277
Thanks for replying
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make rowCountPerTask configurable
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543#issuecomment-1115748081
Could you please assign this to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make rowCountPerTask configurable
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543#issuecomment-1115751778
Thanks, assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] cxzl25 commented on issue #2543: [Subtask] Kyuubi Spark TPC-DS Connector - Make rowCountPerTask configurable
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on issue #2543:
URL: https://github.com/apache/incubator-kyuubi/issues/2543#issuecomment-1132612540
> In practice, I notice that different table costs different time to generate one row, the `rowCountPerTask` maybe not a good idea. Instead, I think we can introduce a concept `cu`(compute unit, and I'm open to the name, naming suggestion is welcome), 1 cu means the task should cost 1 second. To achieve this, we need a benchmark for every table to calculate the generation rate(how many rows can be generated per second). @wForget @cxzl25 WDYT?
+1
Make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org