You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by arina-ielchiieva <gi...@git.apache.org> on 2017/07/21 14:56:32 UTC
[GitHub] drill pull request #882: DRILL-4735: ConvertCountToDirectScan rule enhanceme...
GitHub user arina-ielchiieva opened a pull request:
https://github.com/apache/drill/pull/882
DRILL-4735: ConvertCountToDirectScan rule enhancements
1. ConvertCountToDirectScan rule will be applicable for 2 or more COUNT aggregates.
To achieve this DynamicPojoRecordReader was added which accepts any number of columns,
on the contrary with PojoRecordReader which depends on class fields.
AbstractPojoRecordReader class was added to factor out common logic for these two readers.
2. ConvertCountToDirectScan will distinguish between missing, directory and implicit columns.
For missing columns count will be set 0, for implicit to the total records count
since implicit columns are based on files and there is no data without a file.
If directory column will be encountered, rule won't be applied.
CountsCollector class was introduced to encapsulate counts collection logic.
3. MetadataDirectGroupScan class was introduced to indicate to the user when metadata was used
during calculation and for which files it was applied.
Details in Jira [DRILL-4735](https://issues.apache.org/jira/browse/DRILL-4735).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/arina-ielchiieva/drill DRILL-4735
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/882.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #882
----
commit 7d75460132f41f2f075d9d279bf71e9e43b060b8
Author: Arina Ielchiieva <ar...@gmail.com>
Date: 2017-07-20T16:26:44Z
DRILL-4735: ConvertCountToDirectScan rule enhancements
1. ConvertCountToDirectScan rule will be applicable for 2 or more COUNT aggregates.
To achieve this DynamicPojoRecordReader was added which accepts any number of columns,
on the contrary with PojoRecordReader which depends on class fields.
AbstractPojoRecordReader class was added to factor out common logic for these two readers.
2. ConvertCountToDirectScan will distinguish between missing, directory and implicit columns.
For missing columns count will be set 0, for implicit to the total records count
since implicit columns are based on files and there is no data without a file.
If directory column will be encountered, rule won't be applied.
CountsCollector class was introduced to encapsulate counts collection logic.
3. MetadataDirectGroupScan class was introduced to indicate to the user when metadata was used
during calculation and for which files it was applied.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #882: DRILL-4735: ConvertCountToDirectScan rule enhanceme...
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva closed the pull request at:
https://github.com/apache/drill/pull/882
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #882: DRILL-4735: ConvertCountToDirectScan rule enhanceme...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/882#discussion_r131447047
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java ---
@@ -85,109 +91,231 @@ protected ConvertCountToDirectScan(RelOptRuleOperand rule, String id) {
@Override
public void onMatch(RelOptRuleCall call) {
final DrillAggregateRel agg = (DrillAggregateRel) call.rel(0);
- final DrillScanRel scan = (DrillScanRel) call.rel(call.rels.length -1);
- final DrillProjectRel proj = call.rels.length == 3 ? (DrillProjectRel) call.rel(1) : null;
+ final DrillScanRel scan = (DrillScanRel) call.rel(call.rels.length - 1);
+ final DrillProjectRel project = call.rels.length == 3 ? (DrillProjectRel) call.rel(1) : null;
final GroupScan oldGrpScan = scan.getGroupScan();
final PlannerSettings settings = PrelUtil.getPlannerSettings(call.getPlanner());
- // Only apply the rule when :
+ // Only apply the rule when:
// 1) scan knows the exact row count in getSize() call,
// 2) No GroupBY key,
- // 3) only one agg function (Check if it's count(*) below).
- // 4) No distinct agg call.
+ // 3) No distinct agg call.
if (!(oldGrpScan.getScanStats(settings).getGroupScanProperty().hasExactRowCount()
&& agg.getGroupCount() == 0
- && agg.getAggCallList().size() == 1
&& !agg.containsDistinctCall())) {
return;
}
- AggregateCall aggCall = agg.getAggCallList().get(0);
-
- if (aggCall.getAggregation().getName().equals("COUNT") ) {
-
- long cnt = 0;
- // count(*) == > empty arg ==> rowCount
- // count(Not-null-input) ==> rowCount
- if (aggCall.getArgList().isEmpty() ||
- (aggCall.getArgList().size() == 1 &&
- ! agg.getInput().getRowType().getFieldList().get(aggCall.getArgList().get(0).intValue()).getType().isNullable())) {
- cnt = (long) oldGrpScan.getScanStats(settings).getRecordCount();
- } else if (aggCall.getArgList().size() == 1) {
- // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
- int index = aggCall.getArgList().get(0);
-
- if (proj != null) {
- // project in the middle of Agg and Scan : Only when input of AggCall is a RexInputRef in Project, we find the index of Scan's field.
- // For instance,
- // Agg - count($0)
- // \
- // Proj - Exp={$1}
- // \
- // Scan (col1, col2).
- // return count of "col2" in Scan's metadata, if found.
-
- if (proj.getProjects().get(index) instanceof RexInputRef) {
- index = ((RexInputRef) proj.getProjects().get(index)).getIndex();
- } else {
- return; // do not apply for all other cases.
- }
- }
+ final CountsCollector countsCollector = new CountsCollector(settings);
+ // if counts were not collected, rule won't be applied
+ if (!countsCollector.collect(agg, scan, project)) {
+ return;
+ }
- String columnName = scan.getRowType().getFieldNames().get(index).toLowerCase();
+ final RelDataType scanRowType = constructDataType(agg);
- cnt = oldGrpScan.getColumnValueCount(SchemaPath.getSimplePath(columnName));
- if (cnt == GroupScan.NO_COLUMN_STATS) {
- // if column stats are not available don't apply this rule
- return;
- }
- } else {
- return; // do nothing.
- }
+ final DynamicPojoRecordReader<Long> reader = new DynamicPojoRecordReader<>(
+ buildSchema(scanRowType.getFieldNames()),
+ Collections.singletonList(countsCollector.getCounts()));
- RelDataType scanRowType = getCountDirectScanRowType(agg.getCluster().getTypeFactory());
+ final ScanStats scanStats = new ScanStats(ScanStats.GroupScanProperty.EXACT_ROW_COUNT, 1, 1, scanRowType.getFieldCount());
+ final GroupScan directScan = new MetadataDirectGroupScan(reader, oldGrpScan.getFiles(), scanStats);
- final ScanPrel newScan = ScanPrel.create(scan,
- scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(DrillDistributionTrait.SINGLETON), getCountDirectScan(cnt),
- scanRowType);
+ final ScanPrel newScan = ScanPrel.create(scan,
+ scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(DrillDistributionTrait.SINGLETON), directScan,
+ scanRowType);
- List<RexNode> exprs = Lists.newArrayList();
- exprs.add(RexInputRef.of(0, scanRowType));
+ final ProjectPrel newProject = new ProjectPrel(agg.getCluster(), agg.getTraitSet().plus(Prel.DRILL_PHYSICAL)
+ .plus(DrillDistributionTrait.SINGLETON), newScan, prepareFieldExpressions(scanRowType), agg.getRowType());
- final ProjectPrel newProj = new ProjectPrel(agg.getCluster(), agg.getTraitSet().plus(Prel.DRILL_PHYSICAL)
- .plus(DrillDistributionTrait.SINGLETON), newScan, exprs, agg.getRowType());
+ call.transformTo(newProject);
+ }
- call.transformTo(newProj);
+ /**
+ * For each aggregate call creates field with "count$" prefix and bigint type.
+ * Constructs record type for created fields.
+ *
+ * @param aggregateRel aggregate relation expression
+ * @return record type
+ */
+ private RelDataType constructDataType(DrillAggregateRel aggregateRel) {
+ List<RelDataTypeField> fields = new ArrayList<>();
+ for (int i = 0; i < aggregateRel.getAggCallList().size(); i++) {
+ RelDataTypeField field = new RelDataTypeFieldImpl("count$" + i, i, aggregateRel.getCluster().getTypeFactory().createSqlType(SqlTypeName.BIGINT));
+ fields.add(field);
}
-
+ return new RelRecordType(fields);
}
/**
- * Class to represent the count aggregate result.
+ * Builds schema based on given field names.
+ * Type for each schema is set to long.class.
+ *
+ * @param fieldNames field names
+ * @return schema
*/
- public static class CountQueryResult {
- public long count;
+ private LinkedHashMap<String, Class<?>> buildSchema(List<String> fieldNames) {
+ LinkedHashMap<String, Class<?>> schema = new LinkedHashMap<>();
+ for (String fieldName: fieldNames) {
+ schema.put(fieldName, long.class);
+ }
+ return schema;
+ }
- public CountQueryResult(long cnt) {
- this.count = cnt;
+ /**
+ * For each field creates row expression.
+ *
+ * @param rowType row type
+ * @return list of row expressions
+ */
+ private List<RexNode> prepareFieldExpressions(RelDataType rowType) {
+ List<RexNode> expressions = new ArrayList<>();
+ for (int i = 0; i < rowType.getFieldCount(); i++) {
+ expressions.add(RexInputRef.of(i, rowType));
}
+ return expressions;
}
- private RelDataType getCountDirectScanRowType(RelDataTypeFactory typeFactory) {
- List<RelDataTypeField> fields = Lists.newArrayList();
- fields.add(new RelDataTypeFieldImpl("count", 0, typeFactory.createSqlType(SqlTypeName.BIGINT)));
+ /**
+ * Helper class to collect counts based on metadata information.
+ * For example, for parquet files it can be obtained from parquet footer (total row count)
+ * or from parquet metadata files (column counts).
+ */
+ private class CountsCollector {
+
+ private final PlannerSettings settings;
+ private final Set<String> implicitColumnsNames;
+ private final List<SchemaPath> columns;
+ private final List<Long> counts;
+
+ CountsCollector(PlannerSettings settings) {
+ this.settings = settings;
+ this.implicitColumnsNames = ColumnExplorer.initImplicitFileColumns(settings.getOptions()).keySet();
+ this.counts = new ArrayList<>();
+ this.columns = new ArrayList<>();
+ }
- return new RelRecordType(fields);
- }
+ /**
+ * Collects counts for each aggregation call.
+ * Will fail to collect counts if:
+ * <ol>
+ * <li>was not able to determine count for at least one aggregation call</li>
+ * <li>count if used for file system partition column</li>
+ * </ol>
+ *
+ * @param agg aggregate relational expression
+ * @param scan scan relational expression
+ * @param project project relational expression
+ *
+ * @return true if counts were collected, false otherwise
+ */
+ boolean collect(DrillAggregateRel agg, DrillScanRel scan, DrillProjectRel project) {
+ return calculateCounts(agg, scan, project) && !containsPartitionColumns();
+ }
+
+ /**
+ * @return list of counts
+ */
+ List<Long> getCounts() {
--- End diff --
In stead of returning list of Long, can we return either a map from schemaPath to counts, or list of pair schemaPath, count? Returning just the lists of counts make it hard to figure out which count corresponds to which aggregate function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #882: DRILL-4735: ConvertCountToDirectScan rule enhanceme...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/882#discussion_r131445381
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java ---
@@ -85,109 +91,231 @@ protected ConvertCountToDirectScan(RelOptRuleOperand rule, String id) {
@Override
public void onMatch(RelOptRuleCall call) {
final DrillAggregateRel agg = (DrillAggregateRel) call.rel(0);
- final DrillScanRel scan = (DrillScanRel) call.rel(call.rels.length -1);
- final DrillProjectRel proj = call.rels.length == 3 ? (DrillProjectRel) call.rel(1) : null;
+ final DrillScanRel scan = (DrillScanRel) call.rel(call.rels.length - 1);
+ final DrillProjectRel project = call.rels.length == 3 ? (DrillProjectRel) call.rel(1) : null;
final GroupScan oldGrpScan = scan.getGroupScan();
final PlannerSettings settings = PrelUtil.getPlannerSettings(call.getPlanner());
- // Only apply the rule when :
+ // Only apply the rule when:
// 1) scan knows the exact row count in getSize() call,
// 2) No GroupBY key,
- // 3) only one agg function (Check if it's count(*) below).
- // 4) No distinct agg call.
+ // 3) No distinct agg call.
if (!(oldGrpScan.getScanStats(settings).getGroupScanProperty().hasExactRowCount()
&& agg.getGroupCount() == 0
- && agg.getAggCallList().size() == 1
&& !agg.containsDistinctCall())) {
return;
}
- AggregateCall aggCall = agg.getAggCallList().get(0);
-
- if (aggCall.getAggregation().getName().equals("COUNT") ) {
-
- long cnt = 0;
- // count(*) == > empty arg ==> rowCount
- // count(Not-null-input) ==> rowCount
- if (aggCall.getArgList().isEmpty() ||
- (aggCall.getArgList().size() == 1 &&
- ! agg.getInput().getRowType().getFieldList().get(aggCall.getArgList().get(0).intValue()).getType().isNullable())) {
- cnt = (long) oldGrpScan.getScanStats(settings).getRecordCount();
- } else if (aggCall.getArgList().size() == 1) {
- // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
- int index = aggCall.getArgList().get(0);
-
- if (proj != null) {
- // project in the middle of Agg and Scan : Only when input of AggCall is a RexInputRef in Project, we find the index of Scan's field.
- // For instance,
- // Agg - count($0)
- // \
- // Proj - Exp={$1}
- // \
- // Scan (col1, col2).
- // return count of "col2" in Scan's metadata, if found.
-
- if (proj.getProjects().get(index) instanceof RexInputRef) {
- index = ((RexInputRef) proj.getProjects().get(index)).getIndex();
- } else {
- return; // do not apply for all other cases.
- }
- }
+ final CountsCollector countsCollector = new CountsCollector(settings);
+ // if counts were not collected, rule won't be applied
+ if (!countsCollector.collect(agg, scan, project)) {
+ return;
+ }
- String columnName = scan.getRowType().getFieldNames().get(index).toLowerCase();
+ final RelDataType scanRowType = constructDataType(agg);
- cnt = oldGrpScan.getColumnValueCount(SchemaPath.getSimplePath(columnName));
- if (cnt == GroupScan.NO_COLUMN_STATS) {
- // if column stats are not available don't apply this rule
- return;
- }
- } else {
- return; // do nothing.
- }
+ final DynamicPojoRecordReader<Long> reader = new DynamicPojoRecordReader<>(
+ buildSchema(scanRowType.getFieldNames()),
+ Collections.singletonList(countsCollector.getCounts()));
- RelDataType scanRowType = getCountDirectScanRowType(agg.getCluster().getTypeFactory());
+ final ScanStats scanStats = new ScanStats(ScanStats.GroupScanProperty.EXACT_ROW_COUNT, 1, 1, scanRowType.getFieldCount());
+ final GroupScan directScan = new MetadataDirectGroupScan(reader, oldGrpScan.getFiles(), scanStats);
- final ScanPrel newScan = ScanPrel.create(scan,
- scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(DrillDistributionTrait.SINGLETON), getCountDirectScan(cnt),
- scanRowType);
+ final ScanPrel newScan = ScanPrel.create(scan,
+ scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(DrillDistributionTrait.SINGLETON), directScan,
+ scanRowType);
- List<RexNode> exprs = Lists.newArrayList();
- exprs.add(RexInputRef.of(0, scanRowType));
+ final ProjectPrel newProject = new ProjectPrel(agg.getCluster(), agg.getTraitSet().plus(Prel.DRILL_PHYSICAL)
+ .plus(DrillDistributionTrait.SINGLETON), newScan, prepareFieldExpressions(scanRowType), agg.getRowType());
- final ProjectPrel newProj = new ProjectPrel(agg.getCluster(), agg.getTraitSet().plus(Prel.DRILL_PHYSICAL)
- .plus(DrillDistributionTrait.SINGLETON), newScan, exprs, agg.getRowType());
+ call.transformTo(newProject);
+ }
- call.transformTo(newProj);
+ /**
+ * For each aggregate call creates field with "count$" prefix and bigint type.
+ * Constructs record type for created fields.
+ *
+ * @param aggregateRel aggregate relation expression
+ * @return record type
+ */
+ private RelDataType constructDataType(DrillAggregateRel aggregateRel) {
+ List<RelDataTypeField> fields = new ArrayList<>();
+ for (int i = 0; i < aggregateRel.getAggCallList().size(); i++) {
+ RelDataTypeField field = new RelDataTypeFieldImpl("count$" + i, i, aggregateRel.getCluster().getTypeFactory().createSqlType(SqlTypeName.BIGINT));
--- End diff --
The fieldname in RelDataTypeField "count$0", "count$1" is not very informative. I think we could either get the field name from the aggregator operator, or use "count$colname" to indicate what column's count it represents.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #882: DRILL-4735: ConvertCountToDirectScan rule enhanceme...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/882#discussion_r131449948
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java ---
@@ -85,109 +91,231 @@ protected ConvertCountToDirectScan(RelOptRuleOperand rule, String id) {
@Override
public void onMatch(RelOptRuleCall call) {
final DrillAggregateRel agg = (DrillAggregateRel) call.rel(0);
- final DrillScanRel scan = (DrillScanRel) call.rel(call.rels.length -1);
- final DrillProjectRel proj = call.rels.length == 3 ? (DrillProjectRel) call.rel(1) : null;
+ final DrillScanRel scan = (DrillScanRel) call.rel(call.rels.length - 1);
+ final DrillProjectRel project = call.rels.length == 3 ? (DrillProjectRel) call.rel(1) : null;
final GroupScan oldGrpScan = scan.getGroupScan();
final PlannerSettings settings = PrelUtil.getPlannerSettings(call.getPlanner());
- // Only apply the rule when :
+ // Only apply the rule when:
// 1) scan knows the exact row count in getSize() call,
// 2) No GroupBY key,
- // 3) only one agg function (Check if it's count(*) below).
- // 4) No distinct agg call.
+ // 3) No distinct agg call.
if (!(oldGrpScan.getScanStats(settings).getGroupScanProperty().hasExactRowCount()
&& agg.getGroupCount() == 0
- && agg.getAggCallList().size() == 1
&& !agg.containsDistinctCall())) {
return;
}
- AggregateCall aggCall = agg.getAggCallList().get(0);
-
- if (aggCall.getAggregation().getName().equals("COUNT") ) {
-
- long cnt = 0;
- // count(*) == > empty arg ==> rowCount
- // count(Not-null-input) ==> rowCount
- if (aggCall.getArgList().isEmpty() ||
- (aggCall.getArgList().size() == 1 &&
- ! agg.getInput().getRowType().getFieldList().get(aggCall.getArgList().get(0).intValue()).getType().isNullable())) {
- cnt = (long) oldGrpScan.getScanStats(settings).getRecordCount();
- } else if (aggCall.getArgList().size() == 1) {
- // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
- int index = aggCall.getArgList().get(0);
-
- if (proj != null) {
- // project in the middle of Agg and Scan : Only when input of AggCall is a RexInputRef in Project, we find the index of Scan's field.
- // For instance,
- // Agg - count($0)
- // \
- // Proj - Exp={$1}
- // \
- // Scan (col1, col2).
- // return count of "col2" in Scan's metadata, if found.
-
- if (proj.getProjects().get(index) instanceof RexInputRef) {
- index = ((RexInputRef) proj.getProjects().get(index)).getIndex();
- } else {
- return; // do not apply for all other cases.
- }
- }
+ final CountsCollector countsCollector = new CountsCollector(settings);
+ // if counts were not collected, rule won't be applied
+ if (!countsCollector.collect(agg, scan, project)) {
+ return;
+ }
- String columnName = scan.getRowType().getFieldNames().get(index).toLowerCase();
+ final RelDataType scanRowType = constructDataType(agg);
- cnt = oldGrpScan.getColumnValueCount(SchemaPath.getSimplePath(columnName));
- if (cnt == GroupScan.NO_COLUMN_STATS) {
- // if column stats are not available don't apply this rule
- return;
- }
- } else {
- return; // do nothing.
- }
+ final DynamicPojoRecordReader<Long> reader = new DynamicPojoRecordReader<>(
+ buildSchema(scanRowType.getFieldNames()),
+ Collections.singletonList(countsCollector.getCounts()));
- RelDataType scanRowType = getCountDirectScanRowType(agg.getCluster().getTypeFactory());
+ final ScanStats scanStats = new ScanStats(ScanStats.GroupScanProperty.EXACT_ROW_COUNT, 1, 1, scanRowType.getFieldCount());
+ final GroupScan directScan = new MetadataDirectGroupScan(reader, oldGrpScan.getFiles(), scanStats);
- final ScanPrel newScan = ScanPrel.create(scan,
- scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(DrillDistributionTrait.SINGLETON), getCountDirectScan(cnt),
- scanRowType);
+ final ScanPrel newScan = ScanPrel.create(scan,
+ scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(DrillDistributionTrait.SINGLETON), directScan,
+ scanRowType);
- List<RexNode> exprs = Lists.newArrayList();
- exprs.add(RexInputRef.of(0, scanRowType));
+ final ProjectPrel newProject = new ProjectPrel(agg.getCluster(), agg.getTraitSet().plus(Prel.DRILL_PHYSICAL)
+ .plus(DrillDistributionTrait.SINGLETON), newScan, prepareFieldExpressions(scanRowType), agg.getRowType());
- final ProjectPrel newProj = new ProjectPrel(agg.getCluster(), agg.getTraitSet().plus(Prel.DRILL_PHYSICAL)
- .plus(DrillDistributionTrait.SINGLETON), newScan, exprs, agg.getRowType());
+ call.transformTo(newProject);
+ }
- call.transformTo(newProj);
+ /**
+ * For each aggregate call creates field with "count$" prefix and bigint type.
+ * Constructs record type for created fields.
+ *
+ * @param aggregateRel aggregate relation expression
+ * @return record type
+ */
+ private RelDataType constructDataType(DrillAggregateRel aggregateRel) {
+ List<RelDataTypeField> fields = new ArrayList<>();
+ for (int i = 0; i < aggregateRel.getAggCallList().size(); i++) {
+ RelDataTypeField field = new RelDataTypeFieldImpl("count$" + i, i, aggregateRel.getCluster().getTypeFactory().createSqlType(SqlTypeName.BIGINT));
+ fields.add(field);
}
-
+ return new RelRecordType(fields);
}
/**
- * Class to represent the count aggregate result.
+ * Builds schema based on given field names.
+ * Type for each schema is set to long.class.
+ *
+ * @param fieldNames field names
+ * @return schema
*/
- public static class CountQueryResult {
- public long count;
+ private LinkedHashMap<String, Class<?>> buildSchema(List<String> fieldNames) {
+ LinkedHashMap<String, Class<?>> schema = new LinkedHashMap<>();
+ for (String fieldName: fieldNames) {
+ schema.put(fieldName, long.class);
+ }
+ return schema;
+ }
- public CountQueryResult(long cnt) {
- this.count = cnt;
+ /**
+ * For each field creates row expression.
+ *
+ * @param rowType row type
+ * @return list of row expressions
+ */
+ private List<RexNode> prepareFieldExpressions(RelDataType rowType) {
+ List<RexNode> expressions = new ArrayList<>();
+ for (int i = 0; i < rowType.getFieldCount(); i++) {
+ expressions.add(RexInputRef.of(i, rowType));
}
+ return expressions;
}
- private RelDataType getCountDirectScanRowType(RelDataTypeFactory typeFactory) {
- List<RelDataTypeField> fields = Lists.newArrayList();
- fields.add(new RelDataTypeFieldImpl("count", 0, typeFactory.createSqlType(SqlTypeName.BIGINT)));
+ /**
+ * Helper class to collect counts based on metadata information.
+ * For example, for parquet files it can be obtained from parquet footer (total row count)
+ * or from parquet metadata files (column counts).
+ */
+ private class CountsCollector {
+
+ private final PlannerSettings settings;
+ private final Set<String> implicitColumnsNames;
+ private final List<SchemaPath> columns;
+ private final List<Long> counts;
+
+ CountsCollector(PlannerSettings settings) {
+ this.settings = settings;
+ this.implicitColumnsNames = ColumnExplorer.initImplicitFileColumns(settings.getOptions()).keySet();
+ this.counts = new ArrayList<>();
+ this.columns = new ArrayList<>();
+ }
- return new RelRecordType(fields);
- }
+ /**
+ * Collects counts for each aggregation call.
+ * Will fail to collect counts if:
+ * <ol>
+ * <li>was not able to determine count for at least one aggregation call</li>
+ * <li>count if used for file system partition column</li>
+ * </ol>
+ *
+ * @param agg aggregate relational expression
+ * @param scan scan relational expression
+ * @param project project relational expression
+ *
+ * @return true if counts were collected, false otherwise
+ */
+ boolean collect(DrillAggregateRel agg, DrillScanRel scan, DrillProjectRel project) {
+ return calculateCounts(agg, scan, project) && !containsPartitionColumns();
+ }
+
+ /**
+ * @return list of counts
+ */
+ List<Long> getCounts() {
+ return counts;
+ }
+
+ /**
+ * For each aggregate call if determine if count can be calculated. Collects counts only for COUNT function.
+ * For star, not null expressions and implicit columns sets count to total record number.
+ * For other cases obtains counts from group scan operator.
+ *
+ * @param agg aggregate relational expression
+ * @param scan scan relational expression
+ * @param project project relational expression
+ * @return true if counts were collected, false otherwise
+ */
+ private boolean calculateCounts(DrillAggregateRel agg, DrillScanRel scan, DrillProjectRel project) {
+ final GroupScan oldGrpScan = scan.getGroupScan();
+ final long totalRecordCount = oldGrpScan.getScanStats(settings).getRecordCount();
+
+ for (AggregateCall aggCall : agg.getAggCallList()) {
+ long cnt;
+
+ // rule can be applied only for count function
+ if (!"count".equalsIgnoreCase(aggCall.getAggregation().getName()) ) {
+ return false;
+ }
+
+ if (containsStarOrNotNullInput(aggCall, agg)) {
+ cnt = totalRecordCount;
+
+ } else if (aggCall.getArgList().size() == 1) {
+ // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
+ int index = aggCall.getArgList().get(0);
+
+ if (project != null) {
+ // project in the middle of Agg and Scan : Only when input of AggCall is a RexInputRef in Project, we find the index of Scan's field.
+ // For instance,
+ // Agg - count($0)
+ // \
+ // Proj - Exp={$1}
+ // \
+ // Scan (col1, col2).
+ // return count of "col2" in Scan's metadata, if found.
+ if (!(project.getProjects().get(index) instanceof RexInputRef)) {
+ return false; // do not apply for all other cases.
+ }
+
+ index = ((RexInputRef) project.getProjects().get(index)).getIndex();
+ }
+
+ String columnName = scan.getRowType().getFieldNames().get(index).toLowerCase();
+
+ // for implicit column count will the same as total record count
+ if (implicitColumnsNames.contains(columnName)) {
+ cnt = totalRecordCount;
+ } else {
+ SchemaPath simplePath = SchemaPath.getSimplePath(columnName);
+ columns.add(simplePath);
+
+ cnt = oldGrpScan.getColumnValueCount(simplePath);
+ if (cnt == GroupScan.NO_COLUMN_STATS) {
+ // if column stats is not available don't apply this rule
+ return false;
+ }
+ }
+ } else {
+ return false;
+ }
+ counts.add(cnt);
+ }
+ return true;
+ }
- private GroupScan getCountDirectScan(long cnt) {
- CountQueryResult res = new CountQueryResult(cnt);
+ /**
+ * Checks if aggregate call contains star or non-null expression:
+ * <pre>
+ * count(*) == > empty arg ==> rowCount
+ * count(Not-null-input) ==> rowCount
+ * </pre>
+ *
+ * @param aggregateCall aggregate call
+ * @param aggregate aggregate relation expression
+ * @return true of aggregate call contains star or non-null expression
+ */
+ private boolean containsStarOrNotNullInput(AggregateCall aggregateCall, DrillAggregateRel aggregate) {
+ return aggregateCall.getArgList().isEmpty() ||
+ (aggregateCall.getArgList().size() == 1 &&
+ !aggregate.getInput().getRowType().getFieldList().get(aggregateCall.getArgList().get(0)).getType().isNullable());
+ }
- PojoRecordReader<CountQueryResult> reader = new PojoRecordReader<CountQueryResult>(CountQueryResult.class,
- Collections.singleton(res).iterator());
+ /**
+ * Checks if stores list of columns contains file system partition columns.
+ *
+ * @return true if contains partition columns, false otherwise
+ */
+ private boolean containsPartitionColumns() {
+ final ColumnExplorer columnExplorer = new ColumnExplorer(settings.getOptions(), columns);
--- End diff --
Can ColumnExplorer provide an API to tell if one particular column is the partition column? Seems to me that with such API, we could exit from the for loop in line 238, in stead of first building column list, then call this method to check if the column lists contain a file system partition column.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #882: DRILL-4735: ConvertCountToDirectScan rule enhanceme...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/882#discussion_r131450166
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/MetadataDirectGroupScan.java ---
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.direct;
+
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.store.RecordReader;
+
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * Represents direct scan based on metadata information.
+ * For example, for parquet files it can be obtained from parquet footer (total row count)
+ * or from parquet metadata files (column counts).
+ * Contains reader, statistics and list of scanned files if present.
+ */
+@JsonTypeName("metadata-direct-scan")
+public class MetadataDirectGroupScan extends DirectGroupScan {
+
+ private final Collection<String> files;
+
+ public MetadataDirectGroupScan(RecordReader reader, Collection<String> files) {
+ super(reader);
+ this.files = files;
+ }
+
+ public MetadataDirectGroupScan(RecordReader reader, Collection<String> files, ScanStats stats) {
+ super(reader, stats);
+ this.files = files;
+ }
+
+ @Override
+ public PhysicalOperator getNewWithChildren(List<PhysicalOperator> children) throws ExecutionSetupException {
+ assert children == null || children.isEmpty();
+ return new MetadataDirectGroupScan(reader, files, stats);
+ }
+
+ @Override
+ public GroupScan clone(List<SchemaPath> columns) {
+ return this;
+ }
+
+ /**
+ * <p>
+ * Returns string representation of group scan data.
+ * Includes list of files if present.
+ * </p>
+ *
+ * <p>
+ * Example: [usedMetadata = true, files = [/tmp/0_0_0.parquet], numFiles = 1]
+ * </p>
+ *
+ * @return string representation of group scan data
+ */
+ @Override
+ public String getDigest() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("usedMetadata = true, ");
--- End diff --
This "useMetadata=true" seems to be redundant, since it's for MetadataDirectGS.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #882: DRILL-4735: ConvertCountToDirectScan rule enhancements
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/882
New PR was opened - #900
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #882: DRILL-4735: ConvertCountToDirectScan rule enhanceme...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/882#discussion_r131447579
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java ---
@@ -85,109 +91,231 @@ protected ConvertCountToDirectScan(RelOptRuleOperand rule, String id) {
@Override
public void onMatch(RelOptRuleCall call) {
final DrillAggregateRel agg = (DrillAggregateRel) call.rel(0);
- final DrillScanRel scan = (DrillScanRel) call.rel(call.rels.length -1);
- final DrillProjectRel proj = call.rels.length == 3 ? (DrillProjectRel) call.rel(1) : null;
+ final DrillScanRel scan = (DrillScanRel) call.rel(call.rels.length - 1);
+ final DrillProjectRel project = call.rels.length == 3 ? (DrillProjectRel) call.rel(1) : null;
final GroupScan oldGrpScan = scan.getGroupScan();
final PlannerSettings settings = PrelUtil.getPlannerSettings(call.getPlanner());
- // Only apply the rule when :
+ // Only apply the rule when:
// 1) scan knows the exact row count in getSize() call,
// 2) No GroupBY key,
- // 3) only one agg function (Check if it's count(*) below).
- // 4) No distinct agg call.
+ // 3) No distinct agg call.
if (!(oldGrpScan.getScanStats(settings).getGroupScanProperty().hasExactRowCount()
&& agg.getGroupCount() == 0
- && agg.getAggCallList().size() == 1
&& !agg.containsDistinctCall())) {
return;
}
- AggregateCall aggCall = agg.getAggCallList().get(0);
-
- if (aggCall.getAggregation().getName().equals("COUNT") ) {
-
- long cnt = 0;
- // count(*) == > empty arg ==> rowCount
- // count(Not-null-input) ==> rowCount
- if (aggCall.getArgList().isEmpty() ||
- (aggCall.getArgList().size() == 1 &&
- ! agg.getInput().getRowType().getFieldList().get(aggCall.getArgList().get(0).intValue()).getType().isNullable())) {
- cnt = (long) oldGrpScan.getScanStats(settings).getRecordCount();
- } else if (aggCall.getArgList().size() == 1) {
- // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
- int index = aggCall.getArgList().get(0);
-
- if (proj != null) {
- // project in the middle of Agg and Scan : Only when input of AggCall is a RexInputRef in Project, we find the index of Scan's field.
- // For instance,
- // Agg - count($0)
- // \
- // Proj - Exp={$1}
- // \
- // Scan (col1, col2).
- // return count of "col2" in Scan's metadata, if found.
-
- if (proj.getProjects().get(index) instanceof RexInputRef) {
- index = ((RexInputRef) proj.getProjects().get(index)).getIndex();
- } else {
- return; // do not apply for all other cases.
- }
- }
+ final CountsCollector countsCollector = new CountsCollector(settings);
+ // if counts were not collected, rule won't be applied
+ if (!countsCollector.collect(agg, scan, project)) {
+ return;
+ }
- String columnName = scan.getRowType().getFieldNames().get(index).toLowerCase();
+ final RelDataType scanRowType = constructDataType(agg);
- cnt = oldGrpScan.getColumnValueCount(SchemaPath.getSimplePath(columnName));
- if (cnt == GroupScan.NO_COLUMN_STATS) {
- // if column stats are not available don't apply this rule
- return;
- }
- } else {
- return; // do nothing.
- }
+ final DynamicPojoRecordReader<Long> reader = new DynamicPojoRecordReader<>(
+ buildSchema(scanRowType.getFieldNames()),
+ Collections.singletonList(countsCollector.getCounts()));
- RelDataType scanRowType = getCountDirectScanRowType(agg.getCluster().getTypeFactory());
+ final ScanStats scanStats = new ScanStats(ScanStats.GroupScanProperty.EXACT_ROW_COUNT, 1, 1, scanRowType.getFieldCount());
+ final GroupScan directScan = new MetadataDirectGroupScan(reader, oldGrpScan.getFiles(), scanStats);
- final ScanPrel newScan = ScanPrel.create(scan,
- scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(DrillDistributionTrait.SINGLETON), getCountDirectScan(cnt),
- scanRowType);
+ final ScanPrel newScan = ScanPrel.create(scan,
+ scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(DrillDistributionTrait.SINGLETON), directScan,
+ scanRowType);
- List<RexNode> exprs = Lists.newArrayList();
- exprs.add(RexInputRef.of(0, scanRowType));
+ final ProjectPrel newProject = new ProjectPrel(agg.getCluster(), agg.getTraitSet().plus(Prel.DRILL_PHYSICAL)
+ .plus(DrillDistributionTrait.SINGLETON), newScan, prepareFieldExpressions(scanRowType), agg.getRowType());
- final ProjectPrel newProj = new ProjectPrel(agg.getCluster(), agg.getTraitSet().plus(Prel.DRILL_PHYSICAL)
- .plus(DrillDistributionTrait.SINGLETON), newScan, exprs, agg.getRowType());
+ call.transformTo(newProject);
+ }
- call.transformTo(newProj);
+ /**
+ * For each aggregate call creates field with "count$" prefix and bigint type.
+ * Constructs record type for created fields.
+ *
+ * @param aggregateRel aggregate relation expression
+ * @return record type
+ */
+ private RelDataType constructDataType(DrillAggregateRel aggregateRel) {
+ List<RelDataTypeField> fields = new ArrayList<>();
+ for (int i = 0; i < aggregateRel.getAggCallList().size(); i++) {
+ RelDataTypeField field = new RelDataTypeFieldImpl("count$" + i, i, aggregateRel.getCluster().getTypeFactory().createSqlType(SqlTypeName.BIGINT));
+ fields.add(field);
}
-
+ return new RelRecordType(fields);
}
/**
- * Class to represent the count aggregate result.
+ * Builds schema based on given field names.
+ * Type for each schema is set to long.class.
+ *
+ * @param fieldNames field names
+ * @return schema
*/
- public static class CountQueryResult {
- public long count;
+ private LinkedHashMap<String, Class<?>> buildSchema(List<String> fieldNames) {
+ LinkedHashMap<String, Class<?>> schema = new LinkedHashMap<>();
+ for (String fieldName: fieldNames) {
+ schema.put(fieldName, long.class);
+ }
+ return schema;
+ }
- public CountQueryResult(long cnt) {
- this.count = cnt;
+ /**
+ * For each field creates row expression.
+ *
+ * @param rowType row type
+ * @return list of row expressions
+ */
+ private List<RexNode> prepareFieldExpressions(RelDataType rowType) {
+ List<RexNode> expressions = new ArrayList<>();
+ for (int i = 0; i < rowType.getFieldCount(); i++) {
+ expressions.add(RexInputRef.of(i, rowType));
}
+ return expressions;
}
- private RelDataType getCountDirectScanRowType(RelDataTypeFactory typeFactory) {
- List<RelDataTypeField> fields = Lists.newArrayList();
- fields.add(new RelDataTypeFieldImpl("count", 0, typeFactory.createSqlType(SqlTypeName.BIGINT)));
+ /**
+ * Helper class to collect counts based on metadata information.
+ * For example, for parquet files it can be obtained from parquet footer (total row count)
+ * or from parquet metadata files (column counts).
+ */
+ private class CountsCollector {
+
+ private final PlannerSettings settings;
+ private final Set<String> implicitColumnsNames;
+ private final List<SchemaPath> columns;
+ private final List<Long> counts;
+
+ CountsCollector(PlannerSettings settings) {
+ this.settings = settings;
+ this.implicitColumnsNames = ColumnExplorer.initImplicitFileColumns(settings.getOptions()).keySet();
+ this.counts = new ArrayList<>();
+ this.columns = new ArrayList<>();
+ }
- return new RelRecordType(fields);
- }
+ /**
+ * Collects counts for each aggregation call.
+ * Will fail to collect counts if:
+ * <ol>
+ * <li>was not able to determine count for at least one aggregation call</li>
+ * <li>count if used for file system partition column</li>
+ * </ol>
+ *
+ * @param agg aggregate relational expression
+ * @param scan scan relational expression
+ * @param project project relational expression
+ *
+ * @return true if counts were collected, false otherwise
+ */
+ boolean collect(DrillAggregateRel agg, DrillScanRel scan, DrillProjectRel project) {
+ return calculateCounts(agg, scan, project) && !containsPartitionColumns();
--- End diff --
If user calls collect(agg, scan, project), the class should clear/reset all the internal collections like "columns", "counts". Otherwise, what if user uses the same CountsCollector instance, and call collect() twice with different inputs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #882: DRILL-4735: ConvertCountToDirectScan rule enhanceme...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/882#discussion_r131446392
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java ---
@@ -85,109 +91,231 @@ protected ConvertCountToDirectScan(RelOptRuleOperand rule, String id) {
@Override
public void onMatch(RelOptRuleCall call) {
final DrillAggregateRel agg = (DrillAggregateRel) call.rel(0);
- final DrillScanRel scan = (DrillScanRel) call.rel(call.rels.length -1);
- final DrillProjectRel proj = call.rels.length == 3 ? (DrillProjectRel) call.rel(1) : null;
+ final DrillScanRel scan = (DrillScanRel) call.rel(call.rels.length - 1);
+ final DrillProjectRel project = call.rels.length == 3 ? (DrillProjectRel) call.rel(1) : null;
final GroupScan oldGrpScan = scan.getGroupScan();
final PlannerSettings settings = PrelUtil.getPlannerSettings(call.getPlanner());
- // Only apply the rule when :
+ // Only apply the rule when:
// 1) scan knows the exact row count in getSize() call,
// 2) No GroupBY key,
- // 3) only one agg function (Check if it's count(*) below).
- // 4) No distinct agg call.
+ // 3) No distinct agg call.
if (!(oldGrpScan.getScanStats(settings).getGroupScanProperty().hasExactRowCount()
&& agg.getGroupCount() == 0
- && agg.getAggCallList().size() == 1
&& !agg.containsDistinctCall())) {
return;
}
- AggregateCall aggCall = agg.getAggCallList().get(0);
-
- if (aggCall.getAggregation().getName().equals("COUNT") ) {
-
- long cnt = 0;
- // count(*) == > empty arg ==> rowCount
- // count(Not-null-input) ==> rowCount
- if (aggCall.getArgList().isEmpty() ||
- (aggCall.getArgList().size() == 1 &&
- ! agg.getInput().getRowType().getFieldList().get(aggCall.getArgList().get(0).intValue()).getType().isNullable())) {
- cnt = (long) oldGrpScan.getScanStats(settings).getRecordCount();
- } else if (aggCall.getArgList().size() == 1) {
- // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
- int index = aggCall.getArgList().get(0);
-
- if (proj != null) {
- // project in the middle of Agg and Scan : Only when input of AggCall is a RexInputRef in Project, we find the index of Scan's field.
- // For instance,
- // Agg - count($0)
- // \
- // Proj - Exp={$1}
- // \
- // Scan (col1, col2).
- // return count of "col2" in Scan's metadata, if found.
-
- if (proj.getProjects().get(index) instanceof RexInputRef) {
- index = ((RexInputRef) proj.getProjects().get(index)).getIndex();
- } else {
- return; // do not apply for all other cases.
- }
- }
+ final CountsCollector countsCollector = new CountsCollector(settings);
+ // if counts were not collected, rule won't be applied
+ if (!countsCollector.collect(agg, scan, project)) {
+ return;
+ }
- String columnName = scan.getRowType().getFieldNames().get(index).toLowerCase();
+ final RelDataType scanRowType = constructDataType(agg);
- cnt = oldGrpScan.getColumnValueCount(SchemaPath.getSimplePath(columnName));
- if (cnt == GroupScan.NO_COLUMN_STATS) {
- // if column stats are not available don't apply this rule
- return;
- }
- } else {
- return; // do nothing.
- }
+ final DynamicPojoRecordReader<Long> reader = new DynamicPojoRecordReader<>(
+ buildSchema(scanRowType.getFieldNames()),
+ Collections.singletonList(countsCollector.getCounts()));
- RelDataType scanRowType = getCountDirectScanRowType(agg.getCluster().getTypeFactory());
+ final ScanStats scanStats = new ScanStats(ScanStats.GroupScanProperty.EXACT_ROW_COUNT, 1, 1, scanRowType.getFieldCount());
+ final GroupScan directScan = new MetadataDirectGroupScan(reader, oldGrpScan.getFiles(), scanStats);
- final ScanPrel newScan = ScanPrel.create(scan,
- scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(DrillDistributionTrait.SINGLETON), getCountDirectScan(cnt),
- scanRowType);
+ final ScanPrel newScan = ScanPrel.create(scan,
+ scan.getTraitSet().plus(Prel.DRILL_PHYSICAL).plus(DrillDistributionTrait.SINGLETON), directScan,
+ scanRowType);
- List<RexNode> exprs = Lists.newArrayList();
- exprs.add(RexInputRef.of(0, scanRowType));
+ final ProjectPrel newProject = new ProjectPrel(agg.getCluster(), agg.getTraitSet().plus(Prel.DRILL_PHYSICAL)
+ .plus(DrillDistributionTrait.SINGLETON), newScan, prepareFieldExpressions(scanRowType), agg.getRowType());
- final ProjectPrel newProj = new ProjectPrel(agg.getCluster(), agg.getTraitSet().plus(Prel.DRILL_PHYSICAL)
- .plus(DrillDistributionTrait.SINGLETON), newScan, exprs, agg.getRowType());
+ call.transformTo(newProject);
+ }
- call.transformTo(newProj);
+ /**
+ * For each aggregate call creates field with "count$" prefix and bigint type.
+ * Constructs record type for created fields.
+ *
+ * @param aggregateRel aggregate relation expression
+ * @return record type
+ */
+ private RelDataType constructDataType(DrillAggregateRel aggregateRel) {
+ List<RelDataTypeField> fields = new ArrayList<>();
+ for (int i = 0; i < aggregateRel.getAggCallList().size(); i++) {
+ RelDataTypeField field = new RelDataTypeFieldImpl("count$" + i, i, aggregateRel.getCluster().getTypeFactory().createSqlType(SqlTypeName.BIGINT));
+ fields.add(field);
}
-
+ return new RelRecordType(fields);
}
/**
- * Class to represent the count aggregate result.
+ * Builds schema based on given field names.
+ * Type for each schema is set to long.class.
+ *
+ * @param fieldNames field names
+ * @return schema
*/
- public static class CountQueryResult {
- public long count;
+ private LinkedHashMap<String, Class<?>> buildSchema(List<String> fieldNames) {
+ LinkedHashMap<String, Class<?>> schema = new LinkedHashMap<>();
+ for (String fieldName: fieldNames) {
+ schema.put(fieldName, long.class);
+ }
+ return schema;
+ }
- public CountQueryResult(long cnt) {
- this.count = cnt;
+ /**
+ * For each field creates row expression.
+ *
+ * @param rowType row type
+ * @return list of row expressions
+ */
+ private List<RexNode> prepareFieldExpressions(RelDataType rowType) {
+ List<RexNode> expressions = new ArrayList<>();
+ for (int i = 0; i < rowType.getFieldCount(); i++) {
+ expressions.add(RexInputRef.of(i, rowType));
}
+ return expressions;
}
- private RelDataType getCountDirectScanRowType(RelDataTypeFactory typeFactory) {
- List<RelDataTypeField> fields = Lists.newArrayList();
- fields.add(new RelDataTypeFieldImpl("count", 0, typeFactory.createSqlType(SqlTypeName.BIGINT)));
+ /**
+ * Helper class to collect counts based on metadata information.
+ * For example, for parquet files it can be obtained from parquet footer (total row count)
+ * or from parquet metadata files (column counts).
+ */
+ private class CountsCollector {
+
+ private final PlannerSettings settings;
+ private final Set<String> implicitColumnsNames;
+ private final List<SchemaPath> columns;
+ private final List<Long> counts;
+
+ CountsCollector(PlannerSettings settings) {
+ this.settings = settings;
+ this.implicitColumnsNames = ColumnExplorer.initImplicitFileColumns(settings.getOptions()).keySet();
+ this.counts = new ArrayList<>();
+ this.columns = new ArrayList<>();
+ }
- return new RelRecordType(fields);
- }
+ /**
+ * Collects counts for each aggregation call.
+ * Will fail to collect counts if:
+ * <ol>
+ * <li>was not able to determine count for at least one aggregation call</li>
+ * <li>count if used for file system partition column</li>
+ * </ol>
+ *
+ * @param agg aggregate relational expression
+ * @param scan scan relational expression
+ * @param project project relational expression
+ *
+ * @return true if counts were collected, false otherwise
+ */
+ boolean collect(DrillAggregateRel agg, DrillScanRel scan, DrillProjectRel project) {
+ return calculateCounts(agg, scan, project) && !containsPartitionColumns();
+ }
+
+ /**
+ * @return list of counts
+ */
+ List<Long> getCounts() {
+ return counts;
--- End diff --
Minor comment: The API of this class seems to assume user would call collect() and only it return true then call getCounts(). What will happen if user call getCounts() even when collect() return false? Seems user would get some partial information. Probably return an empty collection in case requirements not meet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---