You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "lirui-apache (via GitHub)" <gi...@apache.org> on 2023/08/01 07:02:13 UTC
[GitHub] [iceberg] lirui-apache opened a new issue, #8196: NamedReference::bind performance issue
lirui-apache opened a new issue, #8196:
URL: https://github.com/apache/iceberg/issues/8196
### Apache Iceberg version
0.13.1
### Query engine
Trino
### Please describe the bug 🐞
`NamedReference::bind` works by creating a `Schema` from the `StructType` parameter. This usually means repeated creation of the Schema instance, which involves re-computing fields like `idToName`, `idToAccessor`, etc. So I wonder whether we can have an interface to directly work with a Schema parameter?
The following code takes about 1400ms on my laptop. If I bind against the Schema instance, run time is reduced to 40ms.
```java
public void test() throws Exception {
List<Types.NestedField> fields = Lists.newArrayList();
for (int i = 1; i <= 100; i++) {
fields.add(Types.NestedField.optional(i, String.format("x%d", i), Types.IntegerType.get()));
}
Schema schema = new Schema(fields);
Expression filter = Expressions.alwaysTrue();
for (int i = 1; i <= 3; i++) {
filter = Expressions.and(filter, Expressions.equal(String.format("x%d", i), i));
}
long start = System.currentTimeMillis();
for (int i = 0; i < 10000; i++) {
Binder.bind(schema.asStruct(), filter, false);
}
System.out.println(System.currentTimeMillis() - start);
}
```
Some background: we implemented indexes that can be used to skip splits on Trino workers. For these indexes to work, the coordinator needs to distribute the filter expressions to workers. Therefore we need to access `FileScanTask::residual` on coordinator during split generation, which involves the evaluation and binding stuff. And we noticed it can become a hot spot method, or even the bottleneck of the whole query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache commented on issue #8196: NamedReference::bind performance issue
Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on issue #8196:
URL: https://github.com/apache/iceberg/issues/8196#issuecomment-1674179189
Yeah we are also binding against Schema internally. So far everything works fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] NamedReference::bind performance issue [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8196:
URL: https://github.com/apache/iceberg/issues/8196#issuecomment-2372611237
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] NamedReference::bind performance issue [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8196:
URL: https://github.com/apache/iceberg/issues/8196#issuecomment-2339369293
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #8196: NamedReference::bind performance issue
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #8196:
URL: https://github.com/apache/iceberg/issues/8196#issuecomment-1673653101
Sounds good, nice find
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on issue #8196: NamedReference::bind performance issue
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on issue #8196:
URL: https://github.com/apache/iceberg/issues/8196#issuecomment-1672542711
Yeah, this is indeed a performance problem that we have noticed as well, especially for tables with huge amounts of columns. We have implemented the binding to the schema directly internally. I could submit a PR recently.
cc @rdblue @aokolnychyi @szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] NamedReference::bind performance issue [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8196: NamedReference::bind performance issue
URL: https://github.com/apache/iceberg/issues/8196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org