You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/06/15 20:37:58 UTC
[GitHub] [iceberg] edgarRd opened a new pull request #1115: Refactor TableScan optionals into an immutable context object
edgarRd opened a new pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115
This PR is a refactor of `TableScan` implementations to use a `TableScanContext` object for the optional arguments used.
As shown in https://github.com/apache/iceberg/pull/1094#discussion_r435532764, when new arguments are added to a `TableScan` the number of changes needed is really large only to pass down the arguments through the refinement calls.
This change will make such changes more straightforward minimizing the number of changes required to add an argument, which would only be required now in the context object and in the specific `TableScan` that would use it, leading to more maintainable code.
PTAL @aokolnychyi @rdblue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441031968
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -167,59 +154,52 @@ public TableScan asOfTime(long timestampMillis) {
@Override
public TableScan option(String property, String value) {
ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();
- builder.putAll(options);
+ builder.putAll(context.options());
builder.put(property, value);
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, builder.build());
+ ops, table, schema, context.options(builder.build()));
}
@Override
public TableScan project(Schema projectedSchema) {
return newRefinedScan(
- ops, table, snapshotId, projectedSchema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, options);
+ ops, table, projectedSchema, context.copy());
Review comment:
What is your rationale for not making `projectedSchema` part of `context`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441158866
##########
File path: core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java
##########
@@ -19,55 +19,43 @@
package org.apache.iceberg;
-import java.util.Collection;
import java.util.List;
import java.util.Set;
-import org.apache.iceberg.expressions.Expression;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.FluentIterable;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.relocated.com.google.common.collect.Sets;
import org.apache.iceberg.util.SnapshotUtil;
import org.apache.iceberg.util.ThreadPools;
class IncrementalDataTableScan extends DataTableScan {
- private long fromSnapshotId;
- private long toSnapshotId;
-
- IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats,
- Collection<String> selectedColumns, ImmutableMap<String, String> options,
- long fromSnapshotId, long toSnapshotId) {
- super(ops, table, null, schema, rowFilter, ignoreResiduals, caseSensitive, colStats, selectedColumns, options);
- validateSnapshotIds(table, fromSnapshotId, toSnapshotId);
- this.fromSnapshotId = fromSnapshotId;
- this.toSnapshotId = toSnapshotId;
+
+ IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, TableScanContext context) {
+ super(ops, table, schema, context.useSnapshotId(null));
Review comment:
This change was done following the previous code that did pass `null` in the `super` constructor. `TableScanContext` does start as null, but there's no guarantee that it's still `null` in this part since it's only receiving a context as argument, not creating one from scratch, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r440488123
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ private TableScanContext(Long snapshotId, Expression rowFilter, boolean ignoreResiduals,
+ boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
+ ImmutableMap<String, String> options, Long fromSnapshotId, Long toSnapshotId) {
+ this.snapshotId = snapshotId;
+ this.rowFilter = rowFilter;
+ this.ignoreResiduals = ignoreResiduals;
+ this.caseSensitive = caseSensitive;
+ this.colStats = colStats;
+ this.selectedColumns = selectedColumns;
+ this.options = options;
+ this.fromSnapshotId = fromSnapshotId;
+ this.toSnapshotId = toSnapshotId;
+ }
+
+ Long snapshotId() {
+ return snapshotId;
+ }
+
+ Expression rowFilter() {
+ return rowFilter;
+ }
+
+ boolean ignoreResiduals() {
+ return ignoreResiduals;
+ }
+
+ boolean caseSensitive() {
+ return caseSensitive;
+ }
+
+ boolean colStats() {
+ return colStats;
+ }
+
+ Collection<String> selectedColumns() {
+ return selectedColumns;
+ }
+
+ ImmutableMap<String, String> options() {
+ return options;
+ }
+
+ Long fromSnapshotId() {
+ return fromSnapshotId;
+ }
+
+ Long toSnapshotId() {
+ return toSnapshotId;
+ }
+
+ TableScanContext copy() {
+ return TableScanContext.builder(this).build();
+ }
+
+ static Builder builder() {
+ return new Builder();
+ }
+
+ static Builder builder(TableScanContext other) {
+ return new Builder(other);
+ }
+
+ static class Builder {
Review comment:
Yeah, that's another option. I can change it to use that pattern.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441116950
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -273,7 +269,7 @@ public String toString() {
return MoreObjects.toStringHelper(this)
.add("table", table)
.add("projection", schema().asStruct())
- .add("filter", context.rowFilter())
+ .add("filter", context.filterRows())
Review comment:
The `rowFilter` getter doesn't need to change. It is still getting the row filter. Just the setter should be a verb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441034330
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -243,6 +224,7 @@ public TableScan ignoreResiduals() {
@Override
public CloseableIterable<CombinedScanTask> planTasks() {
+ ImmutableMap<String, String> options = context.options();
Review comment:
We avoid leaking specific classes through the API. Can you update this to use `Map` instead? The getter method `options` should also only return `Map`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#issuecomment-645073566
Looks good! Thanks @edgarRd!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441035067
##########
File path: core/src/main/java/org/apache/iceberg/DataFilesTable.java
##########
@@ -76,24 +74,17 @@ public Schema schema() {
this.fileSchema = fileSchema;
}
- private FilesTableScan(
- TableOperations ops, Table table, Long snapshotId, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats,
- Collection<String> selectedColumns, Schema fileSchema, ImmutableMap<String, String> options) {
- super(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, options);
+ private FilesTableScan(TableOperations ops, Table table, Schema schema, Schema fileSchema,
+ TableScanContext context) {
+ super(ops, table, schema, context);
this.fileSchema = fileSchema;
}
@Override
- protected TableScan newRefinedScan(
- TableOperations ops, Table table, Long snapshotId, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
- ImmutableMap<String, String> options) {
+ protected TableScan newRefinedScan(TableOperations ops, Table table, Schema schema,
+ TableScanContext context) {
Review comment:
Nit: Does this need a newline?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441038706
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ TableScanContext() {
+ this(null, Expressions.alwaysTrue(), false, true, false, null, ImmutableMap.of(), null, null);
+ }
+
+ private TableScanContext(Long snapshotId, Expression rowFilter, boolean ignoreResiduals,
+ boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
+ ImmutableMap<String, String> options, Long fromSnapshotId, Long toSnapshotId) {
+ this.snapshotId = snapshotId;
+ this.rowFilter = rowFilter;
+ this.ignoreResiduals = ignoreResiduals;
+ this.caseSensitive = caseSensitive;
+ this.colStats = colStats;
+ this.selectedColumns = selectedColumns;
+ this.options = options;
+ this.fromSnapshotId = fromSnapshotId;
+ this.toSnapshotId = toSnapshotId;
+ }
+
+ Long snapshotId() {
+ return snapshotId;
+ }
+
+ TableScanContext snapshotId(Long id) {
+ return new TableScanContext(id, rowFilter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ Expression rowFilter() {
+ return rowFilter;
+ }
+
+ TableScanContext rowFilter(Expression filter) {
+ return new TableScanContext(snapshotId, filter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ boolean ignoreResiduals() {
+ return ignoreResiduals;
+ }
+
+ TableScanContext ignoreResiduals(boolean shouldIgnoreResiduals) {
+ return new TableScanContext(snapshotId, rowFilter, shouldIgnoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ boolean caseSensitive() {
+ return caseSensitive;
+ }
+
+ TableScanContext caseSensitive(boolean isCaseSensitive) {
+ return new TableScanContext(snapshotId, rowFilter, ignoreResiduals,
+ isCaseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ boolean colStats() {
+ return colStats;
+ }
+
+ TableScanContext colStats(boolean shouldUseColumnStats) {
Review comment:
This controls whether or not column stats are returned in the `DataFile` instances in the resulting splits, not whether column stats should be used in planning. Columns stats are always used in planning. To avoid confusion, can you rename this to `shouldReturnColumnStats`? A similar update for the getter would be useful as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441034773
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -304,16 +286,18 @@ public String toString() {
* @return the Schema to project
*/
private Schema lazyColumnProjection() {
+ Collection<String> selectedColumns = context.selectedColumns();
Review comment:
This use of `selectedColumns` works because it is a getter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#issuecomment-644920019
Thanks for updating this, @edgarRd! It looks good, but I found a few minor issues with style and maintainability to fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441031494
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -167,59 +154,52 @@ public TableScan asOfTime(long timestampMillis) {
@Override
public TableScan option(String property, String value) {
ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();
- builder.putAll(options);
+ builder.putAll(context.options());
builder.put(property, value);
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, builder.build());
+ ops, table, schema, context.options(builder.build()));
Review comment:
I think it would be better to add `withOption(String property, String value)` to context and move the map builder there, instead of replacing all options. That keeps this simple and delegates work to the class with the most context (tell, don't ask).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441037327
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ TableScanContext() {
+ this(null, Expressions.alwaysTrue(), false, true, false, null, ImmutableMap.of(), null, null);
+ }
+
+ private TableScanContext(Long snapshotId, Expression rowFilter, boolean ignoreResiduals,
+ boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
+ ImmutableMap<String, String> options, Long fromSnapshotId, Long toSnapshotId) {
+ this.snapshotId = snapshotId;
+ this.rowFilter = rowFilter;
+ this.ignoreResiduals = ignoreResiduals;
+ this.caseSensitive = caseSensitive;
+ this.colStats = colStats;
+ this.selectedColumns = selectedColumns;
+ this.options = options;
+ this.fromSnapshotId = fromSnapshotId;
+ this.toSnapshotId = toSnapshotId;
+ }
+
+ Long snapshotId() {
+ return snapshotId;
+ }
+
+ TableScanContext snapshotId(Long id) {
+ return new TableScanContext(id, rowFilter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ Expression rowFilter() {
+ return rowFilter;
+ }
+
+ TableScanContext rowFilter(Expression filter) {
Review comment:
To make this a verb, how about `filterRows`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441088935
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -304,16 +286,18 @@ public String toString() {
* @return the Schema to project
*/
private Schema lazyColumnProjection() {
+ Collection<String> selectedColumns = context.selectedColumns();
Review comment:
I wonder if it would be valuable to publish (add in repo) a naming style guide to standardize this and save time. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441116447
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -153,47 +153,43 @@ public TableScan asOfTime(long timestampMillis) {
@Override
public TableScan option(String property, String value) {
- ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();
- builder.putAll(context.options());
- builder.put(property, value);
-
return newRefinedScan(
- ops, table, schema, context.options(builder.build()));
+ ops, table, schema, context.withOption(property, value));
}
@Override
public TableScan project(Schema projectedSchema) {
return newRefinedScan(
- ops, table, projectedSchema, context.copy());
+ ops, table, projectedSchema, context);
}
@Override
public TableScan caseSensitive(boolean scanCaseSensitive) {
return newRefinedScan(
- ops, table, schema, context.caseSensitive(scanCaseSensitive));
+ ops, table, schema, context.isCaseSensitive(scanCaseSensitive));
Review comment:
Using the prefix `is` normal signals a boolean getter, not a setter. What about `setCaseSensitive` or just `caseSensitive`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441037141
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ TableScanContext() {
+ this(null, Expressions.alwaysTrue(), false, true, false, null, ImmutableMap.of(), null, null);
+ }
+
+ private TableScanContext(Long snapshotId, Expression rowFilter, boolean ignoreResiduals,
+ boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
+ ImmutableMap<String, String> options, Long fromSnapshotId, Long toSnapshotId) {
+ this.snapshotId = snapshotId;
+ this.rowFilter = rowFilter;
+ this.ignoreResiduals = ignoreResiduals;
+ this.caseSensitive = caseSensitive;
+ this.colStats = colStats;
+ this.selectedColumns = selectedColumns;
+ this.options = options;
+ this.fromSnapshotId = fromSnapshotId;
+ this.toSnapshotId = toSnapshotId;
+ }
+
+ Long snapshotId() {
+ return snapshotId;
+ }
+
+ TableScanContext snapshotId(Long id) {
Review comment:
How about `scanSnapshotId`?
Also, why is `id` a boxed `Long`? If this is called, it should be non-null, right?
We always prefer primitive types so we can guarantee they are non-null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441030228
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -135,19 +123,18 @@ public TableScan appendsAfter(long fromSnapshotId) {
@Override
public TableScan useSnapshot(long scanSnapshotId) {
- Preconditions.checkArgument(this.snapshotId == null,
- "Cannot override snapshot, already set to id=%s", snapshotId);
+ Preconditions.checkState(context.snapshotId() == null,
Review comment:
I think this should continue to be an `IllegalArgumentException` because it is validating the argument passed into `useSnapshot`. The problem is not with the context, it is with how the user is trying to modify that context.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441037141
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ TableScanContext() {
+ this(null, Expressions.alwaysTrue(), false, true, false, null, ImmutableMap.of(), null, null);
+ }
+
+ private TableScanContext(Long snapshotId, Expression rowFilter, boolean ignoreResiduals,
+ boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
+ ImmutableMap<String, String> options, Long fromSnapshotId, Long toSnapshotId) {
+ this.snapshotId = snapshotId;
+ this.rowFilter = rowFilter;
+ this.ignoreResiduals = ignoreResiduals;
+ this.caseSensitive = caseSensitive;
+ this.colStats = colStats;
+ this.selectedColumns = selectedColumns;
+ this.options = options;
+ this.fromSnapshotId = fromSnapshotId;
+ this.toSnapshotId = toSnapshotId;
+ }
+
+ Long snapshotId() {
+ return snapshotId;
+ }
+
+ TableScanContext snapshotId(Long id) {
Review comment:
How about `scanSnapshotId`?
Also, why is `id` a boxed `Long`? If this is called, it should be non-null, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441176413
##########
File path: core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java
##########
@@ -19,55 +19,43 @@
package org.apache.iceberg;
-import java.util.Collection;
import java.util.List;
import java.util.Set;
-import org.apache.iceberg.expressions.Expression;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.FluentIterable;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.relocated.com.google.common.collect.Sets;
import org.apache.iceberg.util.SnapshotUtil;
import org.apache.iceberg.util.ThreadPools;
class IncrementalDataTableScan extends DataTableScan {
- private long fromSnapshotId;
- private long toSnapshotId;
-
- IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats,
- Collection<String> selectedColumns, ImmutableMap<String, String> options,
- long fromSnapshotId, long toSnapshotId) {
- super(ops, table, null, schema, rowFilter, ignoreResiduals, caseSensitive, colStats, selectedColumns, options);
- validateSnapshotIds(table, fromSnapshotId, toSnapshotId);
- this.fromSnapshotId = fromSnapshotId;
- this.toSnapshotId = toSnapshotId;
+
+ IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, TableScanContext context) {
+ super(ops, table, schema, context.useSnapshotId(null));
Review comment:
The guarantee that the snapshot ID is currently null is the check done in the refinement methods that create an incremental scan. I think those validations should be done in the context because the context should be responsible for knowing what is a valid scan. If a table doesn't support incremental scanning, then it can have additional validations, but anything common should be in the context.
That would allow you to not set the snapshot to null here. We may also want to get rid of `IncrementalDataTableScan`, though probably not in this PR. I think the purpose of it was to minimize changes to `BaseTableScan`. But this refactor makes that much less necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441068118
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -167,59 +154,52 @@ public TableScan asOfTime(long timestampMillis) {
@Override
public TableScan option(String property, String value) {
ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();
- builder.putAll(options);
+ builder.putAll(context.options());
builder.put(property, value);
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, builder.build());
+ ops, table, schema, context.options(builder.build()));
}
@Override
public TableScan project(Schema projectedSchema) {
return newRefinedScan(
- ops, table, snapshotId, projectedSchema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, options);
+ ops, table, projectedSchema, context.copy());
Review comment:
Schema does not have a `default` value like the elements in the context, hence it would always need to be specified when creating the context.
In this refinement call, `projectedSchema` is not an additional context field but takes over schema in `TableScan`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441029110
##########
File path: core/src/main/java/org/apache/iceberg/AllEntriesTable.java
##########
@@ -81,23 +79,14 @@ public Schema schema() {
super(ops, table, schema);
}
- private Scan(
- TableOperations ops, Table table, Long snapshotId, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats,
- Collection<String> selectedColumns, ImmutableMap<String, String> options) {
- super(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, options);
+ private Scan(TableOperations ops, Table table, Schema schema, TableScanContext context) {
+ super(ops, table, schema, context);
}
@Override
- protected TableScan newRefinedScan(
- TableOperations ops, Table table, Long snapshotId, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
- ImmutableMap<String, String> options) {
- return new Scan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, options);
+ protected TableScan newRefinedScan(TableOperations ops, Table table, Schema schema,
+ TableScanContext context) {
+ return new Scan(ops, table, schema, context.copy());
Review comment:
If the context is immutable, we don't need to call `copy`, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441036700
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ TableScanContext() {
+ this(null, Expressions.alwaysTrue(), false, true, false, null, ImmutableMap.of(), null, null);
Review comment:
I think it would be more clear and maintainable if this didn't use `this(...)`. When new options are added, this list will need to change and it is easy to get off by one when passing in nulls. If instead this set each field default individually, it would be easier to read and avoid update errors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441119353
##########
File path: core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java
##########
@@ -19,55 +19,43 @@
package org.apache.iceberg;
-import java.util.Collection;
import java.util.List;
import java.util.Set;
-import org.apache.iceberg.expressions.Expression;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.FluentIterable;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.relocated.com.google.common.collect.Sets;
import org.apache.iceberg.util.SnapshotUtil;
import org.apache.iceberg.util.ThreadPools;
class IncrementalDataTableScan extends DataTableScan {
- private long fromSnapshotId;
- private long toSnapshotId;
-
- IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats,
- Collection<String> selectedColumns, ImmutableMap<String, String> options,
- long fromSnapshotId, long toSnapshotId) {
- super(ops, table, null, schema, rowFilter, ignoreResiduals, caseSensitive, colStats, selectedColumns, options);
- validateSnapshotIds(table, fromSnapshotId, toSnapshotId);
- this.fromSnapshotId = fromSnapshotId;
- this.toSnapshotId = toSnapshotId;
+
+ IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, TableScanContext context) {
+ super(ops, table, schema, context.useSnapshotId(null));
+ validateSnapshotIds(table, context.fromSnapshotId(), context.toSnapshotId());
Review comment:
Shouldn't validations be done in the context? I don't see why the context would allow a change that is invalid and needs to be enforced in different scans.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441033703
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -167,59 +154,52 @@ public TableScan asOfTime(long timestampMillis) {
@Override
public TableScan option(String property, String value) {
ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();
- builder.putAll(options);
+ builder.putAll(context.options());
builder.put(property, value);
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, builder.build());
+ ops, table, schema, context.options(builder.build()));
}
@Override
public TableScan project(Schema projectedSchema) {
return newRefinedScan(
- ops, table, snapshotId, projectedSchema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, options);
+ ops, table, projectedSchema, context.copy());
}
@Override
public TableScan caseSensitive(boolean scanCaseSensitive) {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- scanCaseSensitive, colStats, selectedColumns, options);
+ ops, table, schema, context.caseSensitive(scanCaseSensitive));
}
@Override
public TableScan includeColumnStats() {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, true, selectedColumns, options);
+ ops, table, schema, context.colStats(true));
}
@Override
public TableScan select(Collection<String> columns) {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, columns, options);
+ ops, table, schema, context.selectedColumns(columns));
}
@Override
public TableScan filter(Expression expr) {
- return newRefinedScan(
- ops, table, snapshotId, schema, Expressions.and(rowFilter, expr),
- ignoreResiduals, caseSensitive, colStats, selectedColumns, options);
+ return newRefinedScan(ops, table, schema,
+ context.rowFilter(Expressions.and(context.rowFilter(), expr)));
}
@Override
public Expression filter() {
- return rowFilter;
+ return context.rowFilter();
}
@Override
public TableScan ignoreResiduals() {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, true,
- caseSensitive, colStats, selectedColumns, options);
+ ops, table, schema, context.ignoreResiduals(true));
Review comment:
Since `ignoreResiduals` is a verb, it would be more consistent to make all of the methods verbs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441096450
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ TableScanContext() {
+ this(null, Expressions.alwaysTrue(), false, true, false, null, ImmutableMap.of(), null, null);
+ }
+
+ private TableScanContext(Long snapshotId, Expression rowFilter, boolean ignoreResiduals,
+ boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
+ ImmutableMap<String, String> options, Long fromSnapshotId, Long toSnapshotId) {
+ this.snapshotId = snapshotId;
+ this.rowFilter = rowFilter;
+ this.ignoreResiduals = ignoreResiduals;
+ this.caseSensitive = caseSensitive;
+ this.colStats = colStats;
+ this.selectedColumns = selectedColumns;
+ this.options = options;
+ this.fromSnapshotId = fromSnapshotId;
+ this.toSnapshotId = toSnapshotId;
+ }
+
+ Long snapshotId() {
+ return snapshotId;
+ }
+
+ TableScanContext snapshotId(Long id) {
Review comment:
> If this is called, it should be non-null, right?
No, it is set to `null` in the constructor of `IncrementalDataTableScan`: https://github.com/apache/iceberg/blob/e0180f783f62f4cd0f30b13a62ef5fcd1cc6f3b9/core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java#L44
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441119013
##########
File path: core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java
##########
@@ -19,55 +19,43 @@
package org.apache.iceberg;
-import java.util.Collection;
import java.util.List;
import java.util.Set;
-import org.apache.iceberg.expressions.Expression;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.FluentIterable;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.relocated.com.google.common.collect.Sets;
import org.apache.iceberg.util.SnapshotUtil;
import org.apache.iceberg.util.ThreadPools;
class IncrementalDataTableScan extends DataTableScan {
- private long fromSnapshotId;
- private long toSnapshotId;
-
- IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats,
- Collection<String> selectedColumns, ImmutableMap<String, String> options,
- long fromSnapshotId, long toSnapshotId) {
- super(ops, table, null, schema, rowFilter, ignoreResiduals, caseSensitive, colStats, selectedColumns, options);
- validateSnapshotIds(table, fromSnapshotId, toSnapshotId);
- this.fromSnapshotId = fromSnapshotId;
- this.toSnapshotId = toSnapshotId;
+
+ IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, TableScanContext context) {
+ super(ops, table, schema, context.useSnapshotId(null));
Review comment:
Doesn't the snapshot ID start off as null? It seems like we don't need to set it here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441096450
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ TableScanContext() {
+ this(null, Expressions.alwaysTrue(), false, true, false, null, ImmutableMap.of(), null, null);
+ }
+
+ private TableScanContext(Long snapshotId, Expression rowFilter, boolean ignoreResiduals,
+ boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
+ ImmutableMap<String, String> options, Long fromSnapshotId, Long toSnapshotId) {
+ this.snapshotId = snapshotId;
+ this.rowFilter = rowFilter;
+ this.ignoreResiduals = ignoreResiduals;
+ this.caseSensitive = caseSensitive;
+ this.colStats = colStats;
+ this.selectedColumns = selectedColumns;
+ this.options = options;
+ this.fromSnapshotId = fromSnapshotId;
+ this.toSnapshotId = toSnapshotId;
+ }
+
+ Long snapshotId() {
+ return snapshotId;
+ }
+
+ TableScanContext snapshotId(Long id) {
Review comment:
> If this is called, it should be non-null, right?
No, it is set to null in the constructor of `IncrementalDataTableScan`: https://github.com/apache/iceberg/blob/e0180f783f62f4cd0f30b13a62ef5fcd1cc6f3b9/core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java#L44
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441035135
##########
File path: core/src/main/java/org/apache/iceberg/DataFilesTable.java
##########
@@ -76,24 +74,17 @@ public Schema schema() {
this.fileSchema = fileSchema;
}
- private FilesTableScan(
- TableOperations ops, Table table, Long snapshotId, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats,
- Collection<String> selectedColumns, Schema fileSchema, ImmutableMap<String, String> options) {
- super(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, options);
+ private FilesTableScan(TableOperations ops, Table table, Schema schema, Schema fileSchema,
+ TableScanContext context) {
+ super(ops, table, schema, context);
this.fileSchema = fileSchema;
}
@Override
- protected TableScan newRefinedScan(
- TableOperations ops, Table table, Long snapshotId, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
- ImmutableMap<String, String> options) {
+ protected TableScan newRefinedScan(TableOperations ops, Table table, Schema schema,
+ TableScanContext context) {
return new FilesTableScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, fileSchema, options);
+ ops, table, schema, fileSchema, context.copy());
Review comment:
Same here: can we remove this newline?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441162770
##########
File path: core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java
##########
@@ -19,55 +19,43 @@
package org.apache.iceberg;
-import java.util.Collection;
import java.util.List;
import java.util.Set;
-import org.apache.iceberg.expressions.Expression;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.FluentIterable;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.relocated.com.google.common.collect.Sets;
import org.apache.iceberg.util.SnapshotUtil;
import org.apache.iceberg.util.ThreadPools;
class IncrementalDataTableScan extends DataTableScan {
- private long fromSnapshotId;
- private long toSnapshotId;
-
- IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats,
- Collection<String> selectedColumns, ImmutableMap<String, String> options,
- long fromSnapshotId, long toSnapshotId) {
- super(ops, table, null, schema, rowFilter, ignoreResiduals, caseSensitive, colStats, selectedColumns, options);
- validateSnapshotIds(table, fromSnapshotId, toSnapshotId);
- this.fromSnapshotId = fromSnapshotId;
- this.toSnapshotId = toSnapshotId;
+
+ IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, TableScanContext context) {
+ super(ops, table, schema, context.useSnapshotId(null));
+ validateSnapshotIds(table, context.fromSnapshotId(), context.toSnapshotId());
Review comment:
I'm not sure about pushing logic to the `TableScanContext` object, specially validation logic that requires another state for context. I'd agree with validations only for the values of the `snapshotId`s or from within `TableScanContext`, e.g. being different, or non-null, snapshotId not being set if to/from snaphosts are set. However, any other logic dependent of another context like `table` could be specific to where this is being used, for which I'd rather keep `TableScanContext` as simple as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441087401
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -243,6 +224,7 @@ public TableScan ignoreResiduals() {
@Override
public CloseableIterable<CombinedScanTask> planTasks() {
+ ImmutableMap<String, String> options = context.options();
Review comment:
I guess this also applies to the already existing [BaseTableScan#options()](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/BaseTableScan.java#L103).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441038826
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ TableScanContext() {
+ this(null, Expressions.alwaysTrue(), false, true, false, null, ImmutableMap.of(), null, null);
+ }
+
+ private TableScanContext(Long snapshotId, Expression rowFilter, boolean ignoreResiduals,
+ boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
+ ImmutableMap<String, String> options, Long fromSnapshotId, Long toSnapshotId) {
+ this.snapshotId = snapshotId;
+ this.rowFilter = rowFilter;
+ this.ignoreResiduals = ignoreResiduals;
+ this.caseSensitive = caseSensitive;
+ this.colStats = colStats;
+ this.selectedColumns = selectedColumns;
+ this.options = options;
+ this.fromSnapshotId = fromSnapshotId;
+ this.toSnapshotId = toSnapshotId;
+ }
+
+ Long snapshotId() {
+ return snapshotId;
+ }
+
+ TableScanContext snapshotId(Long id) {
+ return new TableScanContext(id, rowFilter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ Expression rowFilter() {
+ return rowFilter;
+ }
+
+ TableScanContext rowFilter(Expression filter) {
+ return new TableScanContext(snapshotId, filter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ boolean ignoreResiduals() {
+ return ignoreResiduals;
+ }
+
+ TableScanContext ignoreResiduals(boolean shouldIgnoreResiduals) {
+ return new TableScanContext(snapshotId, rowFilter, shouldIgnoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ boolean caseSensitive() {
+ return caseSensitive;
+ }
+
+ TableScanContext caseSensitive(boolean isCaseSensitive) {
+ return new TableScanContext(snapshotId, rowFilter, ignoreResiduals,
+ isCaseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ boolean colStats() {
+ return colStats;
+ }
+
+ TableScanContext colStats(boolean shouldUseColumnStats) {
+ return new TableScanContext(snapshotId, rowFilter, ignoreResiduals,
+ caseSensitive, shouldUseColumnStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ Collection<String> selectedColumns() {
+ return selectedColumns;
+ }
+
+ TableScanContext selectedColumns(Collection<String> columns) {
Review comment:
As I noted above, how about `selectColumns`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441033322
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -167,59 +154,52 @@ public TableScan asOfTime(long timestampMillis) {
@Override
public TableScan option(String property, String value) {
ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();
- builder.putAll(options);
+ builder.putAll(context.options());
builder.put(property, value);
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, builder.build());
+ ops, table, schema, context.options(builder.build()));
}
@Override
public TableScan project(Schema projectedSchema) {
return newRefinedScan(
- ops, table, snapshotId, projectedSchema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, options);
+ ops, table, projectedSchema, context.copy());
}
@Override
public TableScan caseSensitive(boolean scanCaseSensitive) {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- scanCaseSensitive, colStats, selectedColumns, options);
+ ops, table, schema, context.caseSensitive(scanCaseSensitive));
}
@Override
public TableScan includeColumnStats() {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, true, selectedColumns, options);
+ ops, table, schema, context.colStats(true));
}
@Override
public TableScan select(Collection<String> columns) {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, columns, options);
+ ops, table, schema, context.selectedColumns(columns));
Review comment:
I think it is worth renaming some of these methods; `selectedColumns` is a noun, not a verb. I would expect a method to be phrased using a verb, like `selectColumns(columns)`. Otherwise, this is just dropping `set` from a setter method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441039177
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ TableScanContext() {
+ this(null, Expressions.alwaysTrue(), false, true, false, null, ImmutableMap.of(), null, null);
+ }
+
+ private TableScanContext(Long snapshotId, Expression rowFilter, boolean ignoreResiduals,
+ boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
+ ImmutableMap<String, String> options, Long fromSnapshotId, Long toSnapshotId) {
+ this.snapshotId = snapshotId;
+ this.rowFilter = rowFilter;
+ this.ignoreResiduals = ignoreResiduals;
+ this.caseSensitive = caseSensitive;
+ this.colStats = colStats;
+ this.selectedColumns = selectedColumns;
+ this.options = options;
+ this.fromSnapshotId = fromSnapshotId;
+ this.toSnapshotId = toSnapshotId;
+ }
+
+ Long snapshotId() {
+ return snapshotId;
+ }
+
+ TableScanContext snapshotId(Long id) {
+ return new TableScanContext(id, rowFilter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ Expression rowFilter() {
+ return rowFilter;
+ }
+
+ TableScanContext rowFilter(Expression filter) {
+ return new TableScanContext(snapshotId, filter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ boolean ignoreResiduals() {
+ return ignoreResiduals;
+ }
+
+ TableScanContext ignoreResiduals(boolean shouldIgnoreResiduals) {
+ return new TableScanContext(snapshotId, rowFilter, shouldIgnoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ boolean caseSensitive() {
+ return caseSensitive;
+ }
+
+ TableScanContext caseSensitive(boolean isCaseSensitive) {
+ return new TableScanContext(snapshotId, rowFilter, ignoreResiduals,
+ isCaseSensitive, colStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ boolean colStats() {
+ return colStats;
+ }
+
+ TableScanContext colStats(boolean shouldUseColumnStats) {
+ return new TableScanContext(snapshotId, rowFilter, ignoreResiduals,
+ caseSensitive, shouldUseColumnStats, selectedColumns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ Collection<String> selectedColumns() {
+ return selectedColumns;
+ }
+
+ TableScanContext selectedColumns(Collection<String> columns) {
+ return new TableScanContext(snapshotId, rowFilter, ignoreResiduals,
+ caseSensitive, colStats, columns, options, fromSnapshotId, toSnapshotId);
+ }
+
+ ImmutableMap<String, String> options() {
+ return options;
+ }
+
+ TableScanContext options(Map<String, String> extraOptions) {
+ return new TableScanContext(snapshotId, rowFilter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, ImmutableMap.copyOf(extraOptions),
+ fromSnapshotId, toSnapshotId);
+ }
+
+ Long fromSnapshotId() {
+ return fromSnapshotId;
+ }
+
+ TableScanContext fromSnapshotId(Long id) {
+ return new TableScanContext(snapshotId, rowFilter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, id, toSnapshotId);
+ }
+
+ Long toSnapshotId() {
+ return toSnapshotId;
+ }
+
+ TableScanContext toSnapshotId(Long id) {
+ return new TableScanContext(snapshotId, rowFilter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, options, fromSnapshotId, id);
+ }
+
+ TableScanContext copy() {
Review comment:
I'm not sure this is needed if the context is immutable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r440480833
##########
File path: core/src/main/java/org/apache/iceberg/TableScanContext.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.util.Collection;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * Context object with optional arguments for a TableScan.
+ */
+final class TableScanContext {
+ private final Long snapshotId;
+ private final Expression rowFilter;
+ private final boolean ignoreResiduals;
+ private final boolean caseSensitive;
+ private final boolean colStats;
+ private final Collection<String> selectedColumns;
+ private final ImmutableMap<String, String> options;
+ private final Long fromSnapshotId;
+ private final Long toSnapshotId;
+
+ private TableScanContext(Long snapshotId, Expression rowFilter, boolean ignoreResiduals,
+ boolean caseSensitive, boolean colStats, Collection<String> selectedColumns,
+ ImmutableMap<String, String> options, Long fromSnapshotId, Long toSnapshotId) {
+ this.snapshotId = snapshotId;
+ this.rowFilter = rowFilter;
+ this.ignoreResiduals = ignoreResiduals;
+ this.caseSensitive = caseSensitive;
+ this.colStats = colStats;
+ this.selectedColumns = selectedColumns;
+ this.options = options;
+ this.fromSnapshotId = fromSnapshotId;
+ this.toSnapshotId = toSnapshotId;
+ }
+
+ Long snapshotId() {
+ return snapshotId;
+ }
+
+ Expression rowFilter() {
+ return rowFilter;
+ }
+
+ boolean ignoreResiduals() {
+ return ignoreResiduals;
+ }
+
+ boolean caseSensitive() {
+ return caseSensitive;
+ }
+
+ boolean colStats() {
+ return colStats;
+ }
+
+ Collection<String> selectedColumns() {
+ return selectedColumns;
+ }
+
+ ImmutableMap<String, String> options() {
+ return options;
+ }
+
+ Long fromSnapshotId() {
+ return fromSnapshotId;
+ }
+
+ Long toSnapshotId() {
+ return toSnapshotId;
+ }
+
+ TableScanContext copy() {
+ return TableScanContext.builder(this).build();
+ }
+
+ static Builder builder() {
+ return new Builder();
+ }
+
+ static Builder builder(TableScanContext other) {
+ return new Builder(other);
+ }
+
+ static class Builder {
Review comment:
I think a builder makes this refactor more awkward because the context is always copied twice. Once in the builder, then once when `copy` is called on the builder's result. It would be simpler to keep the refinement pattern, where every method that alters the context returns a new context. Like this:
```java
TableScanContext caseSensitive() {
if (this.caseSensitive) {
return this;
} else {
return new TableScanContext(..., true /* case sensitive */, ...);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441177115
##########
File path: core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java
##########
@@ -19,55 +19,43 @@
package org.apache.iceberg;
-import java.util.Collection;
import java.util.List;
import java.util.Set;
-import org.apache.iceberg.expressions.Expression;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.FluentIterable;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.relocated.com.google.common.collect.Sets;
import org.apache.iceberg.util.SnapshotUtil;
import org.apache.iceberg.util.ThreadPools;
class IncrementalDataTableScan extends DataTableScan {
- private long fromSnapshotId;
- private long toSnapshotId;
-
- IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, Expression rowFilter,
- boolean ignoreResiduals, boolean caseSensitive, boolean colStats,
- Collection<String> selectedColumns, ImmutableMap<String, String> options,
- long fromSnapshotId, long toSnapshotId) {
- super(ops, table, null, schema, rowFilter, ignoreResiduals, caseSensitive, colStats, selectedColumns, options);
- validateSnapshotIds(table, fromSnapshotId, toSnapshotId);
- this.fromSnapshotId = fromSnapshotId;
- this.toSnapshotId = toSnapshotId;
+
+ IncrementalDataTableScan(TableOperations ops, Table table, Schema schema, TableScanContext context) {
+ super(ops, table, schema, context.useSnapshotId(null));
+ validateSnapshotIds(table, context.fromSnapshotId(), context.toSnapshotId());
Review comment:
I agree with you. If the validation is checking that to and from snapshot IDs are valid, then that makes sense. But for validations like snapshot ID should be null if from snapshot ID is set can be done in the context.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1115: Refactor TableScan optional arguments into an immutable context object
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1115:
URL: https://github.com/apache/iceberg/pull/1115#discussion_r441109204
##########
File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java
##########
@@ -167,59 +154,52 @@ public TableScan asOfTime(long timestampMillis) {
@Override
public TableScan option(String property, String value) {
ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();
- builder.putAll(options);
+ builder.putAll(context.options());
builder.put(property, value);
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, builder.build());
+ ops, table, schema, context.options(builder.build()));
}
@Override
public TableScan project(Schema projectedSchema) {
return newRefinedScan(
- ops, table, snapshotId, projectedSchema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, selectedColumns, options);
+ ops, table, projectedSchema, context.copy());
}
@Override
public TableScan caseSensitive(boolean scanCaseSensitive) {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- scanCaseSensitive, colStats, selectedColumns, options);
+ ops, table, schema, context.caseSensitive(scanCaseSensitive));
}
@Override
public TableScan includeColumnStats() {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, true, selectedColumns, options);
+ ops, table, schema, context.colStats(true));
}
@Override
public TableScan select(Collection<String> columns) {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
- caseSensitive, colStats, columns, options);
+ ops, table, schema, context.selectedColumns(columns));
}
@Override
public TableScan filter(Expression expr) {
- return newRefinedScan(
- ops, table, snapshotId, schema, Expressions.and(rowFilter, expr),
- ignoreResiduals, caseSensitive, colStats, selectedColumns, options);
+ return newRefinedScan(ops, table, schema,
+ context.rowFilter(Expressions.and(context.rowFilter(), expr)));
}
@Override
public Expression filter() {
- return rowFilter;
+ return context.rowFilter();
}
@Override
public TableScan ignoreResiduals() {
return newRefinedScan(
- ops, table, snapshotId, schema, rowFilter, true,
- caseSensitive, colStats, selectedColumns, options);
+ ops, table, schema, context.ignoreResiduals(true));
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org