You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/11/28 20:06:39 UTC
[GitHub] [iceberg] abmo-x opened a new pull request, #6301: Update Schema - should check if field is optional/required
abmo-x opened a new pull request, #6301:
URL: https://github.com/apache/iceberg/pull/6301
Update schema currently does not check if the field is optional/required and adds all new fields as optional. This leads to incorrect schema evolution where a required filed gets added to a table schema as optional.
With this change, a required field can only be added if 'allowIncompatibleChanges' is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on pull request #6301: Update Schema - should check if field is optional/required
Posted by GitBox <gi...@apache.org>.
dramaticlly commented on PR #6301:
URL: https://github.com/apache/iceberg/pull/6301#issuecomment-1329783777
> The only case where a field in a union result would not be optional is if both schemas have it as required, right?
If that's the case (I think I kind of agree), looks like required new field cannot be added as required later in update-schema call given prior schema will always missing new filed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] abmo-x commented on pull request #6301: Update Schema - should check if field is optional/required
Posted by GitBox <gi...@apache.org>.
abmo-x commented on PR #6301:
URL: https://github.com/apache/iceberg/pull/6301#issuecomment-1329935940
> @abmo-x, the behavior you're describing doesn't sound correct. If you union two schemas together, any field not in both schemas should be optional. The only case where a field in a union result would not be optional is if both schemas have it as required, right?
I agree that any field not in both should be optional. what's your suggestion when a user makes a incompatible change to their Avro schema `v2` by adding a required field and uses `table.updateSchema().unionByNameWith(v2)` to updated the table schema. In this case the field gets added as optional to the table where as the Avro schema has it as required.
- should the update fail instead of adding the new required field as optional?
- should `unionByNameWith` not be used by and instead user should directly call `table.updateSchema().allowIncompatibleChanges().addRequiredColumn` by diffing the v1 vs v2 schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] abmo-x closed pull request #6301: Update Schema - should check if field is optional/required
Posted by GitBox <gi...@apache.org>.
abmo-x closed pull request #6301: Update Schema - should check if field is optional/required
URL: https://github.com/apache/iceberg/pull/6301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #6301: Update Schema - should check if field is optional/required
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #6301:
URL: https://github.com/apache/iceberg/pull/6301#issuecomment-1329759677
@abmo-x, the behavior you're describing doesn't sound correct. If you union two schemas together, any field not in both schemas should be optional. The only case where a field in a union result would not be optional is if both schemas have it as required, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org