You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@iceberg.apache.org by "javrasya (via GitHub)" <gi...@apache.org> on 2024/05/06 09:25:37 UTC

[I] Spark: Schema evolution is not reflected on branches [iceberg]

javrasya opened a new issue, #10274:
URL: https://github.com/apache/iceberg/issues/10274

   ### Apache Iceberg version
   
   1.4.3
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug 🐞
   
   We have added to columns in a nested struct field by using Iceberg Java API. I can query and see the new columns when no version is passed down to the query;
   
   ```sql
   select * 
   from spark_catalog.my_db.my_table 
   where myStructField.newlyAddedField='123' ;
   ```
   
   The query above works. But not the one below
   
   
   
   ```sql
   ALTER TABLE spark_catalog.my_db.my_table CREATE BRANCH `my_test_branch`;
   
   select * 
   from spark_catalog.my_db.my_table FOR VERSION AS OF 'my_test_branch' 
   where myStructField.newlyAddedField='123' ;
   ```
   
   I get `Error: org.apache.spark.sql.AnalysisException: No such struct field newlyAddedField`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Spark: Schema evolution is not reflected on branches [iceberg]

Posted by "javrasya (via GitHub)" <gi...@apache.org>.

javrasya commented on issue #10274:
URL: https://github.com/apache/iceberg/issues/10274#issuecomment-2095823012

   An interesting finding, once there is any update on the main branch after the schema is evolved, this no longer becomes a problem.
   
   I ran the following to trigger a dummy update to test it and the problem went away;
   
   ```sql
   MERGE INTO spark_catalog.spark_catalog.my_db.my_table AS target
   USING (SELECT * FROM spark_catalog.my_db.my_table LIMIT 1) AS source
   ON target.id = source.id
   WHEN MATCHED THEN
       UPDATE SET *;
   
   ALTER TABLE spark_catalog.my_db.my_table CREATE BRANCH `my_test_branch`;
   
   SELECT * 
   FROM spark_catalog.my_db.my_table.branch_my_test_branch
   WHERE myStructField.newlyAddedField='123' ;
   ``` 
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Spark: Schema evolution is not reflected on branches [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #10274:
URL: https://github.com/apache/iceberg/issues/10274#issuecomment-2453235683

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Spark: Schema evolution is not reflected on branches [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #10274:
URL: https://github.com/apache/iceberg/issues/10274#issuecomment-2481696443

   This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Spark: Schema evolution is not reflected on branches [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] closed issue #10274: Spark: Schema evolution is not reflected on branches
URL: https://github.com/apache/iceberg/issues/10274


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org