You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sfc-gh-asudhakar (via GitHub)" <gi...@apache.org> on 2024/03/20 18:52:15 UTC
[I] `system.add_files` utility does not support updated Partition Spec [iceberg]
sfc-gh-asudhakar opened a new issue, #10008:
URL: https://github.com/apache/iceberg/issues/10008
**Apache Iceberg version**
v2
**Query engine**
Spark
**Please describe the bug 🐞**
**The issue**
Files ingested to an Iceberg table using the `system.add_files` utility don't reflect the latest partition spec, but instead reflects the original partition spec, even if the source Parquet table matches the latest partition spec.
**Repro**
1) Create an Iceberg table with Partition Spec A:
`create table testIcebergTable (data string, p1 int, p2 int) USING ICEBERG partitioned by (p1);`
2) Modify the partition spec by adding another partition field to make Partition Spec B:
`alter table testIcebergTable add partition field p2;`
3) Create a Parquet table whose partitioning matches the new Partition Spec B:
`create table testParquetTable (data string, p1 int, p2 int) USING PARQUET partitioned by (p1, p2)`
4) Insert data into Parquet table
`insert into testParquetTable values ("hello", 10, 20)`
5) Call the `system.add_files` utility with the source as the Parquet table and the destination as the Iceberg table
`CALL system.add_files(table => 'testIcebergTable', source_table => 'testParquetTable')`
6) Run a select query on the Iceberg table
`select * from testIcebergTable`
The select query returns `NULL` for column p2.
Looking at the manifest file that gets created after the `add_files` call, it only contains the value for partition column `p1` and does not contain the value for `p2`.
Looking at the `partition-spec` in the Avro file's key-value metadata also shows a partition spec with only 1 partition column (corresponding to Partition Spec A from step 1 above). As a result, the value for partition column p2 is lost and cannot be retrieved.
Note - if the Iceberg table is originally created with 2 partition columns, then the select query returns both values. But it would face a similar issue if a 3rd partition field were to be added after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `system.add_files` utility does not support updated Partition Spec [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #10008:
URL: https://github.com/apache/iceberg/issues/10008#issuecomment-2011678115
@sfc-gh-asudhakar I believe you need to update the schema of the Iceberg table yourself. The [docs](https://iceberg.apache.org/docs/latest/spark-procedures/#add_files) of `add_files` also states:
> This procedure will not analyze the schema of the files to determine if they actually match the schema of the Iceberg table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `system.add_files` utility does not support updated Partition Spec [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #10008:
URL: https://github.com/apache/iceberg/issues/10008#issuecomment-2014209251
I looked into this a bit and I think I know the problem. Here's a sample test that can be added to `TestAddFilesProcedure` to repro
```
@TestTemplate
public void addFilesPartitionEvolved() {
createIcebergTable(
"p1 int, p2 int, data int not null", "PARTITIONED BY (p1)");
sql("ALTER TABLE %s ADD PARTITION FIELD p2", tableName);
String createParquet =
"CREATE TABLE %s (p1 int, p2 int, data int) USING %s "
+ "PARTITIONED BY (p1, p2) LOCATION '%s'";
sql(createParquet, sourceTableName, "parquet", fileTableDir.getAbsolutePath());
sql("INSERT INTO %s PARTITION (p1=1, p2=10) VALUES (100)", sourceTableName);
List<Object[]> result =
sql(
"CALL %s.system.add_files('%s', '%s')",
catalogName, tableName, sourceTableName);
sql("SELECT * FROM %s", tableName);
}
```
When we import the partitions, we derive an Icebeg partition spec from the hive style partitioning here https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java#L430.
This new partition spec will have a spec ID of 0 (the same spec ID as when you created the Iceberg table).
This is the spec that gets used when writing the manifests here https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java#L350
But in the target Iceberg table, the spec ID with (p1, p2) is actually 1.
I'll need to think more about what the right solution is, but on the surface it seems like the right thing to do here is to
1.) Derive the partition spec from the source table partitioning.
2.) See if that same schema exists in the target table
3.) If so build a copy of the derived partition spec but with the updated spec ID of the target table.
But that seems too specific of a fix for this. I'm also not sure what the behavior of the procedure is if the partition spec on the target is completely different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `system.add_files` utility does not support updated Partition Spec [iceberg]
Posted by "sfc-gh-asudhakar (via GitHub)" <gi...@apache.org>.
sfc-gh-asudhakar commented on issue #10008:
URL: https://github.com/apache/iceberg/issues/10008#issuecomment-2013045412
> @sfc-gh-asudhakar I believe you need to update the schema of the Iceberg table yourself. The [docs](https://iceberg.apache.org/docs/latest/spark-procedures/#add_files) of `add_files` also states:
>
> > This procedure will not analyze the schema of the files to determine if they actually match the schema of the Iceberg table
Hi @nastra , the schema of the Iceberg table was updated (step 2 above) and verified in the new metadata.json file generated after the `alter table` command, which shows `default-spec-id` as `1` that contains both `p1` and `p2` as partition columns.
However, the `add_files` does not seem to reflect this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `system.add_files` utility does not support updated Partition Spec [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10008:
URL: https://github.com/apache/iceberg/issues/10008#issuecomment-2430545372
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `system.add_files` utility does not support updated Partition Spec [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #10008:
URL: https://github.com/apache/iceberg/issues/10008#issuecomment-2013093976
Sorry I must have missed step 2 when reading the description. I'll take a closer look and will update the issue once I know more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `system.add_files` utility does not support updated Partition Spec [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10008:
URL: https://github.com/apache/iceberg/issues/10008#issuecomment-2461060375
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `system.add_files` utility does not support updated Partition Spec [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10008: `system.add_files` utility does not support updated Partition Spec
URL: https://github.com/apache/iceberg/issues/10008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org