You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "RussellSpitzer (via GitHub)" <gi...@apache.org> on 2026/02/20 02:29:56 UTC
[PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
RussellSpitzer opened a new pull request, #15372:
URL: https://github.com/apache/iceberg/pull/15372
Previously while we could set table identifiers there was no way of letting an engine know that it could rely on these fields for optimizations. While we do not enforce or require validation of identifier fields, some users may know that their identifier fields are in fact unique. This PR gives users an option of letting compatible engines know that they can take advantage of the uniqueness of their identifiers.
This is completely separate from true constraint validation and enforcement and is really just a hint for the optimizer that advanced users can take advantage of. Adding it as a table property also allows it to be used on all Table Versions with identifier columns.
Implementation and Tests done in conjunction with Cursor and Claude-4.6-opus-high
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#issuecomment-3935643446
I like it. Using "identifier field ids" as primary key has come up several times in varies conversations. I think this is a good compromise for when the table owner knows that primary key is enforced at write time.
heres the reference to "identifier field ids" for convenience
https://iceberg.apache.org/spec/#identifier-field-ids
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2867076719
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java:
##########
@@ -286,6 +289,25 @@ public MetadataColumn[] metadataColumns() {
return cols.toArray(SparkMetadataColumn[]::new);
}
+ @Override
+ public Constraint[] constraints() {
+ List<Constraint> constraints = Lists.newArrayList();
+
+ SparkReadConf readConf = new SparkReadConf(sparkSession(), icebergTable);
+ Set<String> identifierFieldNames = icebergTable.schema().identifierFieldNames();
+
+ if (readConf.identifierFieldsRely() && !identifierFieldNames.isEmpty()) {
+ constraints.add(
+ Constraint.primaryKey("iceberg_pk", Spark3Util.toNamedReferences(identifierFieldNames))
Review Comment:
Yeah, we should refactor our utils too. They are quite dated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2831425811
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
Is it really specific to reads?
Do we even need a property?
And are we sure it is not an engine specific property?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer merged PR #15372:
URL: https://github.com/apache/iceberg/pull/15372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2843370842
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
I would lean to enable this by default but I can understand if we want to be safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2874012186
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2843375364
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java:
##########
@@ -286,6 +290,28 @@ public MetadataColumn[] metadataColumns() {
return cols.toArray(SparkMetadataColumn[]::new);
}
+ @Override
+ public Constraint[] constraints() {
+ List<Constraint> constraints = Lists.newArrayList();
+
+ SparkReadConf readConf = new SparkReadConf(sparkSession(), icebergTable);
+ Set<String> identifierFieldNames = icebergTable.schema().identifierFieldNames();
+
+ if (readConf.identifierFieldsRely() && !identifierFieldNames.isEmpty()) {
+ NamedReference[] columns =
+ identifierFieldNames.stream().map(FieldReference::apply).toArray(NamedReference[]::new);
+
+ constraints.add(
+ Constraint.primaryKey("iceberg_pk", columns)
+ .enforced(false)
+ .validationStatus(Constraint.ValidationStatus.UNVALIDATED)
Review Comment:
Optional: What about importing `ValidationStatus`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#issuecomment-3935867490
This is what we are emulating -
https://docs.databricks.com/aws/en/sql/user/queries/query-optimization-constraints - Rely in Databricks
https://docs.snowflake.com/en/user-guide/join-elimination - Rely in Snowflake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#issuecomment-3986464515
ah I fat fingered this, I had one more commit for Anton's requests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2867075524
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
I don't feel strongly about the default, so I am okay with whatever you folks decide here.
Let me elaborate on my earlier comment about whether it is read specific. Use cases I know are indeed read-time optimizations. That said, we expose this on `Table$constraints` in Spark. At that point, we don't really know if the table is going to be used for reads or writes and we don't know what kind of read or write time optimization the engine will perform. Hence, my question was whether we strictly need the `read.` prefix or we can just go for `identifier-fields.rely`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2833989897
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
Specific to reads : Yes. We have in the spec that it isn't enforced and although we could do so, seems like a bit of a heavy ask to add here. This is similar to the "RELY" keyword which exists in Snowflake, Databricks Runtime and others. No guarantee of enforcement or validation but will tell the query engine to optimize based off the assumption that it's correct.
For Property: I have mixed feelings here, but I think table property is the right way to go. For Spark, we have to define this at Catalog Load so it has to be an engine specific property set in the spark session, or a table property. Table property feels like a better fit because users probably want to enable this per table, It's not something that should change from application run to application run. Table property is also nice for cross engine compatibility since we can then hint to multiple engines that understand this concept.
For Engine Specific: Since multiple engines support this concept, I think it's better to have it as a global property. We could prefix it with spark, but i'm not sure why we would want to do that if other engines want to take advantage of it. Specifically for me, I'm thinking about Snowflake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2843460831
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
Yeah i'm just worried about a minor version update breaking someone. I would feel ok with doing this in a 2.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2855013879
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java:
##########
@@ -286,6 +289,25 @@ public MetadataColumn[] metadataColumns() {
return cols.toArray(SparkMetadataColumn[]::new);
}
+ @Override
+ public Constraint[] constraints() {
+ List<Constraint> constraints = Lists.newArrayList();
+
+ SparkReadConf readConf = new SparkReadConf(sparkSession(), icebergTable);
+ Set<String> identifierFieldNames = icebergTable.schema().identifierFieldNames();
+
+ if (readConf.identifierFieldsRely() && !identifierFieldNames.isEmpty()) {
+ constraints.add(
+ Constraint.primaryKey("iceberg_pk", Spark3Util.toNamedReferences(identifierFieldNames))
Review Comment:
We actually have this problem in a ton of test files as well TestX3 names abound
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2843368030
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
It is a true concern. I just also worry about having another knob that people need to know and enable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2834522555
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
This is an optimizer hint: Iceberg doesn’t validate or enforce, so this should only affect reads.
I think making it a table property is reasonable since multiple engines could use this hint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2851068572
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java:
##########
@@ -286,6 +289,25 @@ public MetadataColumn[] metadataColumns() {
return cols.toArray(SparkMetadataColumn[]::new);
}
+ @Override
+ public Constraint[] constraints() {
+ List<Constraint> constraints = Lists.newArrayList();
+
+ SparkReadConf readConf = new SparkReadConf(sparkSession(), icebergTable);
+ Set<String> identifierFieldNames = icebergTable.schema().identifierFieldNames();
+
+ if (readConf.identifierFieldsRely() && !identifierFieldNames.isEmpty()) {
+ constraints.add(
+ Constraint.primaryKey("iceberg_pk", Spark3Util.toNamedReferences(identifierFieldNames))
Review Comment:
A bit odd that Spark 4 code calls `Spark3Util`. But I guess maybe the class name `Spark3Util` has become inaccurate over time.
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
default to `false` seems a safer tradeoff for a minor version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2843311530
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java:
##########
@@ -286,6 +290,28 @@ public MetadataColumn[] metadataColumns() {
return cols.toArray(SparkMetadataColumn[]::new);
}
+ @Override
+ public Constraint[] constraints() {
+ List<Constraint> constraints = Lists.newArrayList();
+
+ SparkReadConf readConf = new SparkReadConf(sparkSession(), icebergTable);
+ Set<String> identifierFieldNames = icebergTable.schema().identifierFieldNames();
+
+ if (readConf.identifierFieldsRely() && !identifierFieldNames.isEmpty()) {
+ NamedReference[] columns =
+ identifierFieldNames.stream().map(FieldReference::apply).toArray(NamedReference[]::new);
Review Comment:
Or can use `Expressions::column` directly but it would clash with Iceberg class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2843306182
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java:
##########
@@ -286,6 +290,28 @@ public MetadataColumn[] metadataColumns() {
return cols.toArray(SparkMetadataColumn[]::new);
}
+ @Override
+ public Constraint[] constraints() {
+ List<Constraint> constraints = Lists.newArrayList();
+
+ SparkReadConf readConf = new SparkReadConf(sparkSession(), icebergTable);
+ Set<String> identifierFieldNames = icebergTable.schema().identifierFieldNames();
+
+ if (readConf.identifierFieldsRely() && !identifierFieldNames.isEmpty()) {
+ NamedReference[] columns =
+ identifierFieldNames.stream().map(FieldReference::apply).toArray(NamedReference[]::new);
Review Comment:
FieldReference is a Catalyst private class. I believe `Spark3Util` has a utility that produces named references. We can extend it to accept a collection of them.
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java:
##########
@@ -286,6 +290,28 @@ public MetadataColumn[] metadataColumns() {
return cols.toArray(SparkMetadataColumn[]::new);
}
+ @Override
+ public Constraint[] constraints() {
+ List<Constraint> constraints = Lists.newArrayList();
+
+ SparkReadConf readConf = new SparkReadConf(sparkSession(), icebergTable);
+ Set<String> identifierFieldNames = icebergTable.schema().identifierFieldNames();
+
+ if (readConf.identifierFieldsRely() && !identifierFieldNames.isEmpty()) {
+ NamedReference[] columns =
+ identifierFieldNames.stream().map(FieldReference::apply).toArray(NamedReference[]::new);
Review Comment:
`FieldReference` is a Catalyst private class. I believe `Spark3Util` has a utility that produces named references. We can extend it to accept a collection of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2843294193
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
My fear was that it would be a breaking correctness issue if someone set Identifier columns by accident or unrelated to a CDC use case and just left their table broken-ish
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Adds a Table Property for Relying on Identifier Fields [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #15372:
URL: https://github.com/apache/iceberg/pull/15372#discussion_r2843284132
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -247,6 +247,15 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ /**
+ * When true, declares that the table's identifier fields can be relied upon as a primary key by
+ * query engines for optimization purposes (e.g. eliminating redundant joins or distinct). This is
+ * not enforced at write time and does not validate existing data.
+ */
+ public static final String READ_IDENTIFIER_FIELDS_RELY = "read.identifier-fields.rely";
Review Comment:
I am basically debating whether we even need this property. At least, shall we set it to true by default? I think it kind of behaves today as rely on by default (e.g. CDC case).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org