You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/08/27 01:49:52 UTC

[GitHub] [iceberg] wypoon opened a new pull request #3038: Spark: Better SparkBatchScan statistics estimation

wypoon opened a new pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038


   When estimating statistics, we should take the read schema into account.
   As explained in the [PR](https://github.com/apache/spark/pull/33825) for [SPARK-36568](https://issues.apache.org/jira/browse/SPARK-36568):
   `V2ScanRelationPushDown` can column prune `DataSourceV2ScanRelation`s and change read schema of `Scan` operations. 
   In `SparkBatchScan.estimateStatistics()`, before this change, we sum the file sizes of the files to be scanned and use that as the estimate for the size in bytes. With this change, we adjust that by the ratio of the size of the columns to be read to the size of all the columns; we also adjust that by the compression factor in case it is set (the default is 1.0). With this change, some joins that are broadcast joins on V1 tables but are SortMergeJoins on Iceberg tables can be broadcast joins as well.
   Basically this change does for Iceberg what the above-mentioned Spark PR does for the built-in V2 `FileScan`s.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] wypoon commented on pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
wypoon commented on pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#issuecomment-918798427


   @aokolnychyi thanks for seconding the change.
   Yes, I think it makes sense to do the same for Spark 2 as well. I'll put up a PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] wypoon commented on a change in pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
wypoon commented on a change in pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#discussion_r701491769



##########
File path: spark3/src/main/java/org/apache/iceberg/spark/source/SparkBatchScan.java
##########
@@ -207,22 +207,20 @@ public Statistics estimateStatistics() {
       LOG.debug("using table metadata to estimate table statistics");
       long totalRecords = PropertyUtil.propertyAsLong(table.currentSnapshot().summary(),
           SnapshotSummary.TOTAL_RECORDS_PROP, Long.MAX_VALUE);
-      Schema projectedSchema = expectedSchema != null ? expectedSchema : table.schema();

Review comment:
       I don't think `expectedSchema` is ever null. Thus, I think we can just call `readSchema()` to get the `StructType`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#issuecomment-917668408


   Thanks, @wypoon! Looks great.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#issuecomment-918814786


   Much appreciated, @wypoon!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#issuecomment-907851733


   Thanks @wypoon! I'm okay with this, but I'd prefer to return a better estimate based on number of rows and not compressed size at all. Interested to hear what @aokolnychyi thinks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] wypoon commented on pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
wypoon commented on pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#issuecomment-906932513


   Here is a part of the optimized logical plan for a TPC-DS query (q18) on V1 tables:
   ```
   +- Project [c_customer_sk#50, c_current_cdemo_sk#52, c_current_addr_sk#54, c_birth_year#63], Statistics(sizeInBytes=2.7 MiB)
      +- Filter (((c_birth_month#62 IN (1,6,8,9,12,2) AND isnotnull(c_customer_sk#50)) AND isnotnull(c_current_cdemo_sk#52)) AND isnotnull(c_current_addr_sk#54)), Statistics(sizeInBytes=25.5 MiB)
         +- Relation[c_customer_sk#50,c_customer_id#51,c_current_cdemo_sk#52,c_current_hdemo_sk#53,c_current_addr_sk#54,c_first_shipto_date_sk#55,c_first_sales_date_sk#56,c_salutation#57,c_first_name#58,c_last_name#59,c_preferred_cust_flag#60,c_birth_day#61,c_birth_month#62,c_birth_year#63,c_birth_country#64,c_login#65,c_email_address#66,c_last_review_date#67] parquet, Statistics(sizeInBytes=25.5 MiB)
   ```
   The customer table is 25.5 MiB but the Project plan (4 columns of the 18 columns of the customer table) is small enough to be broadcast.
   Here is the equivalent snippet for Iceberg tables before this change:
   ```
   +- Project [c_customer_sk#59, c_current_cdemo_sk#61, c_current_addr_sk#63, c_birth_year#72], Statistics(sizeInBytes=21.9 MiB)
      +- Filter (((c_birth_month#71 IN (1,6,8,9,12,2) AND isnotnull(c_customer_sk#59)) AND isnotnull(c_current_cdemo_sk#61)) AND isnotnull(c_current_addr_sk#63)), Statistics(sizeInBytes=25.5 MiB)
         +- RelationV2[c_customer_sk#59, c_current_cdemo_sk#61, c_current_addr_sk#63, c_birth_month#71, c_birth_year#72] spark_catalog.tpcds_10_iceberg.customer, Statistics(sizeInBytes=25.5 MiB, rowCount=5.00E+5)
   ```
   A read schema of 5 columns has already been pushed down to the Iceberg customer table, but the statistics for it is the full 25.5 MiB for the table. Consequently, the Project plan, which is estimated basically using the ratio of the size of the 4 columns to the size of the 5 columns times the 25.5 MiB, is too big to be broadcast.
   With this change, the equivalent snippet becomes:
   ```
   +- Project [c_customer_sk#59, c_current_cdemo_sk#61, c_current_addr_sk#63, c_birth_year#72], Statistics(sizeInBytes=2.0 MiB)
      +- Filter (((c_birth_month#71 IN (1,6,8,9,12,2) AND isnotnull(c_customer_sk#59)) AND isnotnull(c_current_cdemo_sk#61)) AND isnotnull(c_current_addr_sk#63)), Statistics(sizeInBytes=2.4 MiB)
         +- RelationV2[c_customer_sk#59, c_current_cdemo_sk#61, c_current_addr_sk#63, c_birth_month#71, c_birth_year#72] spark_catalog.tpcds_10_iceberg.customer, Statistics(sizeInBytes=2.4 MiB, rowCount=5.00E+5)
   ```
   Here the statistics for the Iceberg customer table is adjusted according to read schema and is now 2.4 MiB, and the estimate for the Project adjusts it by the ratio of the size of the 4 columns to the size of the 5 columns, making it 2.0 MiB, and it can be broadcast.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#issuecomment-907851594


   I'm surprised to see that we report the total data file size back to Spark. At Netflix, we switched over to reporting row size estimate * number of rows a long time ago to fix broadcast join memory problems. Spark should ignore the size on disk and use the row size * num rows, but if Spark doesn't do that then I think Iceberg should report that as the size instead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] wypoon commented on pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
wypoon commented on pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#issuecomment-912136782


   @rdblue thanks for reviewing this. I changed the statistics estimation to use (row size * number of rows) for all cases. Of course, the row size estimation can be very far from accurate. Is this version preferable? @aokolnychyi or @RussellSpitzer do you prefer this change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] wypoon commented on a change in pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
wypoon commented on a change in pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#discussion_r701491393



##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkSchemaUtil.java
##########
@@ -280,23 +280,23 @@ private static PartitionSpec identitySpec(Schema schema, List<String> partitionN
   }
 
   /**
-   * estimate approximate table size based on spark schema and total records.
+   * Estimate approximate table size based on Spark schema and total records.
    *
-   * @param tableSchema  spark schema
+   * @param tableSchema  Spark schema
    * @param totalRecords total records in the table
-   * @return approxiate size based on table schema
+   * @return approximate size based on table schema
    */
   public static long estimateSize(StructType tableSchema, long totalRecords) {
     if (totalRecords == Long.MAX_VALUE) {
       return totalRecords;
     }
 
-    long approximateSize = 0;
-    for (StructField sparkField : tableSchema.fields()) {
-      approximateSize += sparkField.dataType().defaultSize();
+    long result;
+    try {
+      result = LongMath.checkedMultiply(tableSchema.defaultSize(), totalRecords);
+    } catch (ArithmeticException e) {
+      result = Long.MAX_VALUE;

Review comment:
       `StructType` has a `defaultSize` method; there is no need to re-implement it. The main utility of the `estimateSize` static method is checking for overflow; we should just use Guava's `LongMath.checkedMultiply`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#issuecomment-918520098


   Sorry for the delay. I think this change makes sense too. Thanks, @wypoon!
   
   Shall we update Spark 2 as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] aokolnychyi commented on pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#issuecomment-918521378


   I created #3108 so that we don't forget.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] wypoon commented on pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
wypoon commented on pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038#issuecomment-907300833


   @rdblue, @aokolnychyi, can you please review this? Can you also please enlighten me why we use record count * row size as the estimate for partitioned tables without filter expressions, but file sizes for unpartitioned tables or partitioned tables with filter expressions? Should we use the record count * row size everywhere?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue merged pull request #3038: Spark: Better SparkBatchScan statistics estimation

Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #3038:
URL: https://github.com/apache/iceberg/pull/3038


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org