You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sannaroby (via GitHub)" <gi...@apache.org> on 2024/04/15 10:13:21 UTC
[I] flink autoscaler: how set write-parallelism ? [iceberg]
sannaroby opened a new issue, #10147:
URL: https://github.com/apache/iceberg/issues/10147
### Query engine
Flink
### Question
Hello, I'm using iceberg-flink-1.18-1.5.0.
I've configured the [flink-operator autoscaler feature](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.8/docs/custom-resource/autoscaler/). I noted that the iceberg-stream-writer operator doesn't change the "write-parallelism" when the autoscaler change the operator parallelism.
As you can see in the image below, the parallelism is equals to 32 but is working just one subtask:
There is a way to force the "write-parallelism" to the parallelism sets by the autoscaler ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]
Posted by "sannaroby (via GitHub)" <gi...@apache.org>.
sannaroby commented on issue #10147:
URL: https://github.com/apache/iceberg/issues/10147#issuecomment-2058495395
Hi @pvary, thanks for your reply.
We're using the HASH distribution mode and this is an extract from our flink job:
```
SingleOutputStreamOperator<Row> mainFunction = env.addSource(kinesisSource)
.keyBy(inputMessage-> inputMessage.getKey())
.window(TumblingProcessingTimeWindows.of(Time.seconds(10)))
.process(new InputMessageToRowProcessFunction(), rowTypeInfo);
FlinkSink.forRow(mainFunction, loadedTableSchema)
.tableLoader(tableLoader)
.table(icebergTable)
.equalityFieldColumns(List.of("messageKey", "messageTimestamp"))
.distributionMode(DistributionMode.HASH)
.append();
```
The autoscaler didn't change the upstream operator parallelism (the "InputMessageToRowProcessFunction"), because it judged it to be unnecessary.
What do you mean for "rebalance step" ?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]
Posted by "jad-grepr (via GitHub)" <gi...@apache.org>.
jad-grepr commented on issue #10147:
URL: https://github.com/apache/iceberg/issues/10147#issuecomment-2445506849
Hi just wanted to check if you figured this one out @sannaroby
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #10147:
URL: https://github.com/apache/iceberg/issues/10147#issuecomment-2061388525
Could it be, that the table is partitioned and all of the new data is targeting a single partition?
If you start the sink with higher writer parallelism, how does the data distribution look like?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10147:
URL: https://github.com/apache/iceberg/issues/10147#issuecomment-2870385478
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10147: flink autoscaler: how set write-parallelism ?
URL: https://github.com/apache/iceberg/issues/10147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #10147:
URL: https://github.com/apache/iceberg/issues/10147#issuecomment-2057528514
@sannaroby: can you share the Sink code?
What distribution mode do you use? Maybe we need a rebalance step before the writer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10147:
URL: https://github.com/apache/iceberg/issues/10147#issuecomment-2833718304
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org