You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Ryan Blue <bl...@apache.org> on 2021/07/27 16:58:13 UTC
[VOTE] Adopt the v2 spec changes
I’d like to propose that we adopt the pending v2 spec changes as the
supported v2 spec. The full list of changes is documented in the v2 summary
section of the spec <https://iceberg.apache.org/spec/#version-2>.
The major breaking change is the addition of delete files and metadata to
track delete files. In addition, there are a few other minor breaking
changes. For example, v2 drops the block_size_in_bytes field in manifests
that was previously required and also omits fields in table metadata that
are now tracked by lists; schema is no longer written in favor of schemas.
Other changes are forward compatible, mostly tightening field requirements
where possible (e.g., schemas and current-schema-id are now required).
Adopting the changes will signal that the community intends to support the
current set of changes and will guarantee forward-compatibility for v2
tables that implement the current v2 spec. Any new breaking changes would
go into v3.
Please vote on adopting the v2 changes in the next 72 hours.
[ ] +1 Adopt the changes as v2
[ ] +0
[ ] -1 Do not adopt the changes, because…
--
Ryan Blue
Re: [VOTE] Adopt the v2 spec changes
Posted by Jack Ye <ye...@gmail.com>.
FYI, regarding the syntax to allow experimenting with the v2 format easily,
I put up a PR: https://github.com/apache/iceberg/pull/2887
-Jack
On Wed, Jul 28, 2021 at 12:23 PM Yan Yan <yy...@gmail.com> wrote:
> +1 (non-binding)
>
> On Wed, Jul 28, 2021 at 10:52 AM Yufei Gu <fl...@gmail.com> wrote:
>
>> +1 (non-binding)
>> Best,
>>
>> Yufei
>>
>> `This is not a contribution`
>>
>>
>> On Wed, Jul 28, 2021 at 8:34 AM Ryan Murray <ry...@gmail.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Wed, Jul 28, 2021 at 5:14 PM Russell Spitzer <
>>> russell.spitzer@gmail.com> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> On Jul 28, 2021, at 10:11 AM, Ryan Blue <bl...@apache.org> wrote:
>>>>
>>>> +1
>>>>
>>>> On Tue, Jul 27, 2021 at 9:58 AM Ryan Blue <bl...@apache.org> wrote:
>>>>
>>>>> I’d like to propose that we adopt the pending v2 spec changes as the
>>>>> supported v2 spec. The full list of changes is documented in the v2
>>>>> summary section of the spec
>>>>> <https://iceberg.apache.org/spec/#version-2>.
>>>>>
>>>>> The major breaking change is the addition of delete files and metadata
>>>>> to track delete files. In addition, there are a few other minor breaking
>>>>> changes. For example, v2 drops the block_size_in_bytes field in
>>>>> manifests that was previously required and also omits fields in table
>>>>> metadata that are now tracked by lists; schema is no longer written
>>>>> in favor of schemas. Other changes are forward compatible, mostly
>>>>> tightening field requirements where possible (e.g., schemas and
>>>>> current-schema-id are now required).
>>>>>
>>>>> Adopting the changes will signal that the community intends to support
>>>>> the current set of changes and will guarantee forward-compatibility for v2
>>>>> tables that implement the current v2 spec. Any new breaking changes would
>>>>> go into v3.
>>>>>
>>>>> Please vote on adopting the v2 changes in the next 72 hours.
>>>>>
>>>>> [ ] +1 Adopt the changes as v2
>>>>> [ ] +0
>>>>> [ ] -1 Do not adopt the changes, because…
>>>>> --
>>>>> Ryan Blue
>>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>>
>>>>
>>>>
Re: [VOTE] Adopt the v2 spec changes
Posted by Yan Yan <yy...@gmail.com>.
+1 (non-binding)
On Wed, Jul 28, 2021 at 10:52 AM Yufei Gu <fl...@gmail.com> wrote:
> +1 (non-binding)
> Best,
>
> Yufei
>
> `This is not a contribution`
>
>
> On Wed, Jul 28, 2021 at 8:34 AM Ryan Murray <ry...@gmail.com> wrote:
>
>> +1 (non-binding)
>>
>> On Wed, Jul 28, 2021 at 5:14 PM Russell Spitzer <
>> russell.spitzer@gmail.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Jul 28, 2021, at 10:11 AM, Ryan Blue <bl...@apache.org> wrote:
>>>
>>> +1
>>>
>>> On Tue, Jul 27, 2021 at 9:58 AM Ryan Blue <bl...@apache.org> wrote:
>>>
>>>> I’d like to propose that we adopt the pending v2 spec changes as the
>>>> supported v2 spec. The full list of changes is documented in the v2
>>>> summary section of the spec
>>>> <https://iceberg.apache.org/spec/#version-2>.
>>>>
>>>> The major breaking change is the addition of delete files and metadata
>>>> to track delete files. In addition, there are a few other minor breaking
>>>> changes. For example, v2 drops the block_size_in_bytes field in
>>>> manifests that was previously required and also omits fields in table
>>>> metadata that are now tracked by lists; schema is no longer written in
>>>> favor of schemas. Other changes are forward compatible, mostly
>>>> tightening field requirements where possible (e.g., schemas and
>>>> current-schema-id are now required).
>>>>
>>>> Adopting the changes will signal that the community intends to support
>>>> the current set of changes and will guarantee forward-compatibility for v2
>>>> tables that implement the current v2 spec. Any new breaking changes would
>>>> go into v3.
>>>>
>>>> Please vote on adopting the v2 changes in the next 72 hours.
>>>>
>>>> [ ] +1 Adopt the changes as v2
>>>> [ ] +0
>>>> [ ] -1 Do not adopt the changes, because…
>>>> --
>>>> Ryan Blue
>>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>>
>>>
>>>
Re: [VOTE] Adopt the v2 spec changes
Posted by Yufei Gu <fl...@gmail.com>.
+1 (non-binding)
Best,
Yufei
`This is not a contribution`
On Wed, Jul 28, 2021 at 8:34 AM Ryan Murray <ry...@gmail.com> wrote:
> +1 (non-binding)
>
> On Wed, Jul 28, 2021 at 5:14 PM Russell Spitzer <ru...@gmail.com>
> wrote:
>
>> +1 (non-binding)
>>
>> On Jul 28, 2021, at 10:11 AM, Ryan Blue <bl...@apache.org> wrote:
>>
>> +1
>>
>> On Tue, Jul 27, 2021 at 9:58 AM Ryan Blue <bl...@apache.org> wrote:
>>
>>> I’d like to propose that we adopt the pending v2 spec changes as the
>>> supported v2 spec. The full list of changes is documented in the v2
>>> summary section of the spec <https://iceberg.apache.org/spec/#version-2>
>>> .
>>>
>>> The major breaking change is the addition of delete files and metadata
>>> to track delete files. In addition, there are a few other minor breaking
>>> changes. For example, v2 drops the block_size_in_bytes field in
>>> manifests that was previously required and also omits fields in table
>>> metadata that are now tracked by lists; schema is no longer written in
>>> favor of schemas. Other changes are forward compatible, mostly
>>> tightening field requirements where possible (e.g., schemas and
>>> current-schema-id are now required).
>>>
>>> Adopting the changes will signal that the community intends to support
>>> the current set of changes and will guarantee forward-compatibility for v2
>>> tables that implement the current v2 spec. Any new breaking changes would
>>> go into v3.
>>>
>>> Please vote on adopting the v2 changes in the next 72 hours.
>>>
>>> [ ] +1 Adopt the changes as v2
>>> [ ] +0
>>> [ ] -1 Do not adopt the changes, because…
>>> --
>>> Ryan Blue
>>>
>>
>>
>> --
>> Ryan Blue
>>
>>
>>
Re: [VOTE] Adopt the v2 spec changes
Posted by Ryan Murray <ry...@gmail.com>.
+1 (non-binding)
On Wed, Jul 28, 2021 at 5:14 PM Russell Spitzer <ru...@gmail.com>
wrote:
> +1 (non-binding)
>
> On Jul 28, 2021, at 10:11 AM, Ryan Blue <bl...@apache.org> wrote:
>
> +1
>
> On Tue, Jul 27, 2021 at 9:58 AM Ryan Blue <bl...@apache.org> wrote:
>
>> I’d like to propose that we adopt the pending v2 spec changes as the
>> supported v2 spec. The full list of changes is documented in the v2
>> summary section of the spec <https://iceberg.apache.org/spec/#version-2>.
>>
>> The major breaking change is the addition of delete files and metadata to
>> track delete files. In addition, there are a few other minor breaking
>> changes. For example, v2 drops the block_size_in_bytes field in
>> manifests that was previously required and also omits fields in table
>> metadata that are now tracked by lists; schema is no longer written in
>> favor of schemas. Other changes are forward compatible, mostly
>> tightening field requirements where possible (e.g., schemas and
>> current-schema-id are now required).
>>
>> Adopting the changes will signal that the community intends to support
>> the current set of changes and will guarantee forward-compatibility for v2
>> tables that implement the current v2 spec. Any new breaking changes would
>> go into v3.
>>
>> Please vote on adopting the v2 changes in the next 72 hours.
>>
>> [ ] +1 Adopt the changes as v2
>> [ ] +0
>> [ ] -1 Do not adopt the changes, because…
>> --
>> Ryan Blue
>>
>
>
> --
> Ryan Blue
>
>
>
Re: [VOTE] Adopt the v2 spec changes
Posted by Russell Spitzer <ru...@gmail.com>.
+1 (non-binding)
> On Jul 28, 2021, at 10:11 AM, Ryan Blue <bl...@apache.org> wrote:
>
> +1
>
> On Tue, Jul 27, 2021 at 9:58 AM Ryan Blue <blue@apache.org <ma...@apache.org>> wrote:
> I’d like to propose that we adopt the pending v2 spec changes as the supported v2 spec. The full list of changes is documented in the v2 summary section of the spec <https://iceberg.apache.org/spec/#version-2>.
>
> The major breaking change is the addition of delete files and metadata to track delete files. In addition, there are a few other minor breaking changes. For example, v2 drops the block_size_in_bytes field in manifests that was previously required and also omits fields in table metadata that are now tracked by lists; schema is no longer written in favor of schemas. Other changes are forward compatible, mostly tightening field requirements where possible (e.g., schemas and current-schema-id are now required).
>
> Adopting the changes will signal that the community intends to support the current set of changes and will guarantee forward-compatibility for v2 tables that implement the current v2 spec. Any new breaking changes would go into v3.
>
> Please vote on adopting the v2 changes in the next 72 hours.
>
> [ ] +1 Adopt the changes as v2
> [ ] +0
> [ ] -1 Do not adopt the changes, because…
>
> --
> Ryan Blue
>
>
> --
> Ryan Blue
Re: [VOTE] Adopt the v2 spec changes
Posted by Ryan Blue <bl...@apache.org>.
+1
On Tue, Jul 27, 2021 at 9:58 AM Ryan Blue <bl...@apache.org> wrote:
> I’d like to propose that we adopt the pending v2 spec changes as the
> supported v2 spec. The full list of changes is documented in the v2
> summary section of the spec <https://iceberg.apache.org/spec/#version-2>.
>
> The major breaking change is the addition of delete files and metadata to
> track delete files. In addition, there are a few other minor breaking
> changes. For example, v2 drops the block_size_in_bytes field in manifests
> that was previously required and also omits fields in table metadata that
> are now tracked by lists; schema is no longer written in favor of schemas.
> Other changes are forward compatible, mostly tightening field requirements
> where possible (e.g., schemas and current-schema-id are now required).
>
> Adopting the changes will signal that the community intends to support the
> current set of changes and will guarantee forward-compatibility for v2
> tables that implement the current v2 spec. Any new breaking changes would
> go into v3.
>
> Please vote on adopting the v2 changes in the next 72 hours.
>
> [ ] +1 Adopt the changes as v2
> [ ] +0
> [ ] -1 Do not adopt the changes, because…
> --
> Ryan Blue
>
--
Ryan Blue
[RESULT] [VOTE] Adopt the v2 spec changes
Posted by Ryan Blue <bl...@tabular.io>.
With 10 +1 votes and no +0 or -1 votes, this passes.
Thanks, everyone for looking into the format changes and taking the time to
vote! And thanks to everyone that worked on getting v2 implemented, tested,
and documented!
I'll put up some PRs to reflect this change in the spec.
Ryan
On Sun, Aug 1, 2021 at 3:21 PM Anton Okolnychyi
<ao...@apple.com.invalid> wrote:
> +1 (binding)
>
> Anton
>
>
> On Aug 1, 2021, at 2:40 PM, Carl Steinbach <cw...@tabular.io> wrote:
>
>
> +1 (binding)
>
> On Wed, Jul 28, 2021 at 6:43 PM Daniel Weeks <dw...@apache.org> wrote:
>
>> +1
>>
>> On Wed, Jul 28, 2021, 1:36 PM Kyle Bendickson
>> <kb...@apple.com.invalid> wrote:
>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>> *Kyle Bendickson*
>>> Software Engineer
>>> Apple
>>> ACS Data
>>> One Apple Park Way,
>>> Cupertino, CA 95014, USA
>>> kbendickson@apple.com
>>>
>>> This email and any attachments may be privileged and may contain
>>> confidential information intended only for the recipient(s) named above.
>>> Any other distribution, forwarding, copying or disclosure of this message
>>> is strictly prohibited. If you have received this email in error, please
>>> notify me immediately by telephone or return email, and delete this message
>>> from your system.
>>>
>>> On Jul 27, 2021, at 9:58 AM, Ryan Blue <bl...@apache.org> wrote:
>>>
>>> I’d like to propose that we adopt the pending v2 spec changes as the
>>> supported v2 spec. The full list of changes is documented in the v2
>>> summary section of the spec <https://iceberg.apache.org/spec/#version-2>
>>> .
>>>
>>> The major breaking change is the addition of delete files and metadata
>>> to track delete files. In addition, there are a few other minor breaking
>>> changes. For example, v2 drops the block_size_in_bytes field in
>>> manifests that was previously required and also omits fields in table
>>> metadata that are now tracked by lists; schema is no longer written in
>>> favor of schemas. Other changes are forward compatible, mostly
>>> tightening field requirements where possible (e.g., schemas and
>>> current-schema-id are now required).
>>>
>>> Adopting the changes will signal that the community intends to support
>>> the current set of changes and will guarantee forward-compatibility for v2
>>> tables that implement the current v2 spec. Any new breaking changes would
>>> go into v3.
>>>
>>> Please vote on adopting the v2 changes in the next 72 hours.
>>>
>>> [ ] +1 Adopt the changes as v2
>>> [ ] +0
>>> [ ] -1 Do not adopt the changes, because…
>>> --
>>> Ryan Blue
>>>
>>>
>>>
--
Ryan Blue
Tabular
Re: [VOTE] Adopt the v2 spec changes
Posted by Anton Okolnychyi <ao...@apple.com.INVALID>.
+1 (binding)
Anton
>
> On Aug 1, 2021, at 2:40 PM, Carl Steinbach <cw...@tabular.io> wrote:
>
>
> +1 (binding)
>
>> On Wed, Jul 28, 2021 at 6:43 PM Daniel Weeks <dw...@apache.org> wrote:
>> +1
>>
>>> On Wed, Jul 28, 2021, 1:36 PM Kyle Bendickson <kb...@apple.com.invalid> wrote:
>>> +1 (non-binding)
>>>
>>>
>>> Kyle Bendickson
>>> Software Engineer
>>> Apple
>>> ACS Data
>>> One Apple Park Way,
>>> Cupertino, CA 95014, USA
>>> kbendickson@apple.com
>>>
>>> This email and any attachments may be privileged and may contain confidential information intended only for the recipient(s) named above. Any other distribution, forwarding, copying or disclosure of this message is strictly prohibited. If you have received this email in error, please notify me immediately by telephone or return email, and delete this message from your system.
>>>
>>>
>>>> On Jul 27, 2021, at 9:58 AM, Ryan Blue <bl...@apache.org> wrote:
>>>>
>>>> I’d like to propose that we adopt the pending v2 spec changes as the supported v2 spec. The full list of changes is documented in the v2 summary section of the spec.
>>>>
>>>> The major breaking change is the addition of delete files and metadata to track delete files. In addition, there are a few other minor breaking changes. For example, v2 drops the block_size_in_bytes field in manifests that was previously required and also omits fields in table metadata that are now tracked by lists; schema is no longer written in favor of schemas. Other changes are forward compatible, mostly tightening field requirements where possible (e.g., schemas and current-schema-id are now required).
>>>>
>>>> Adopting the changes will signal that the community intends to support the current set of changes and will guarantee forward-compatibility for v2 tables that implement the current v2 spec. Any new breaking changes would go into v3.
>>>>
>>>> Please vote on adopting the v2 changes in the next 72 hours.
>>>>
>>>> [ ] +1 Adopt the changes as v2
>>>> [ ] +0
>>>> [ ] -1 Do not adopt the changes, because…
>>>>
>>>> --
>>>> Ryan Blue
>>>
Re: [VOTE] Adopt the v2 spec changes
Posted by Carl Steinbach <cw...@tabular.io>.
+1 (binding)
On Wed, Jul 28, 2021 at 6:43 PM Daniel Weeks <dw...@apache.org> wrote:
> +1
>
> On Wed, Jul 28, 2021, 1:36 PM Kyle Bendickson
> <kb...@apple.com.invalid> wrote:
>
>> +1 (non-binding)
>>
>>
>>
>> *Kyle Bendickson*
>> Software Engineer
>> Apple
>> ACS Data
>> One Apple Park Way,
>> Cupertino, CA 95014, USA
>> kbendickson@apple.com
>>
>> This email and any attachments may be privileged and may contain
>> confidential information intended only for the recipient(s) named above.
>> Any other distribution, forwarding, copying or disclosure of this message
>> is strictly prohibited. If you have received this email in error, please
>> notify me immediately by telephone or return email, and delete this message
>> from your system.
>>
>> On Jul 27, 2021, at 9:58 AM, Ryan Blue <bl...@apache.org> wrote:
>>
>> I’d like to propose that we adopt the pending v2 spec changes as the
>> supported v2 spec. The full list of changes is documented in the v2
>> summary section of the spec <https://iceberg.apache.org/spec/#version-2>.
>>
>> The major breaking change is the addition of delete files and metadata to
>> track delete files. In addition, there are a few other minor breaking
>> changes. For example, v2 drops the block_size_in_bytes field in
>> manifests that was previously required and also omits fields in table
>> metadata that are now tracked by lists; schema is no longer written in
>> favor of schemas. Other changes are forward compatible, mostly
>> tightening field requirements where possible (e.g., schemas and
>> current-schema-id are now required).
>>
>> Adopting the changes will signal that the community intends to support
>> the current set of changes and will guarantee forward-compatibility for v2
>> tables that implement the current v2 spec. Any new breaking changes would
>> go into v3.
>>
>> Please vote on adopting the v2 changes in the next 72 hours.
>>
>> [ ] +1 Adopt the changes as v2
>> [ ] +0
>> [ ] -1 Do not adopt the changes, because…
>> --
>> Ryan Blue
>>
>>
>>
Re: [VOTE] Adopt the v2 spec changes
Posted by Daniel Weeks <dw...@apache.org>.
+1
On Wed, Jul 28, 2021, 1:36 PM Kyle Bendickson <kb...@apple.com.invalid>
wrote:
> +1 (non-binding)
>
>
>
> *Kyle Bendickson*
> Software Engineer
> Apple
> ACS Data
> One Apple Park Way,
> Cupertino, CA 95014, USA
> kbendickson@apple.com
>
> This email and any attachments may be privileged and may contain
> confidential information intended only for the recipient(s) named above.
> Any other distribution, forwarding, copying or disclosure of this message
> is strictly prohibited. If you have received this email in error, please
> notify me immediately by telephone or return email, and delete this message
> from your system.
>
> On Jul 27, 2021, at 9:58 AM, Ryan Blue <bl...@apache.org> wrote:
>
> I’d like to propose that we adopt the pending v2 spec changes as the
> supported v2 spec. The full list of changes is documented in the v2
> summary section of the spec <https://iceberg.apache.org/spec/#version-2>.
>
> The major breaking change is the addition of delete files and metadata to
> track delete files. In addition, there are a few other minor breaking
> changes. For example, v2 drops the block_size_in_bytes field in manifests
> that was previously required and also omits fields in table metadata that
> are now tracked by lists; schema is no longer written in favor of schemas.
> Other changes are forward compatible, mostly tightening field requirements
> where possible (e.g., schemas and current-schema-id are now required).
>
> Adopting the changes will signal that the community intends to support the
> current set of changes and will guarantee forward-compatibility for v2
> tables that implement the current v2 spec. Any new breaking changes would
> go into v3.
>
> Please vote on adopting the v2 changes in the next 72 hours.
>
> [ ] +1 Adopt the changes as v2
> [ ] +0
> [ ] -1 Do not adopt the changes, because…
> --
> Ryan Blue
>
>
>
Re: [VOTE] Adopt the v2 spec changes
Posted by Kyle Bendickson <kb...@apple.com.INVALID>.
+1 (non-binding)
Kyle Bendickson
Software Engineer
Apple
ACS Data
One Apple Park Way,
Cupertino, CA 95014, USA
kbendickson@apple.com
This email and any attachments may be privileged and may contain confidential information intended only for the recipient(s) named above. Any other distribution, forwarding, copying or disclosure of this message is strictly prohibited. If you have received this email in error, please notify me immediately by telephone or return email, and delete this message from your system.
> On Jul 27, 2021, at 9:58 AM, Ryan Blue <bl...@apache.org> wrote:
>
> I’d like to propose that we adopt the pending v2 spec changes as the supported v2 spec. The full list of changes is documented in the v2 summary section of the spec <https://iceberg.apache.org/spec/#version-2>.
>
> The major breaking change is the addition of delete files and metadata to track delete files. In addition, there are a few other minor breaking changes. For example, v2 drops the block_size_in_bytes field in manifests that was previously required and also omits fields in table metadata that are now tracked by lists; schema is no longer written in favor of schemas. Other changes are forward compatible, mostly tightening field requirements where possible (e.g., schemas and current-schema-id are now required).
>
> Adopting the changes will signal that the community intends to support the current set of changes and will guarantee forward-compatibility for v2 tables that implement the current v2 spec. Any new breaking changes would go into v3.
>
> Please vote on adopting the v2 changes in the next 72 hours.
>
> [ ] +1 Adopt the changes as v2
> [ ] +0
> [ ] -1 Do not adopt the changes, because…
>
> --
> Ryan Blue
Re: [VOTE] Adopt the v2 spec changes
Posted by Ryan Blue <bl...@tabular.io>.
Adopting the v2 spec and making support guarantees will mean that we can
add table properties to create v2 tables. The reason why it is hard to
upgrade to v2 is that we haven't adopted it or made compatibility
guarantees, so we don't want people using it in production.
On Tue, Jul 27, 2021 at 11:53 PM OpenInx <op...@gmail.com> wrote:
> > adopt the pending v2 spec changes as the supported v2 spec
>
> I assume this vote wants to reach the consistency between the community
> members that we won't introduce any breaking changes in v2 spec, not
> discuss exposing v2 to SQL tables like the following, right ?
>
> CREATE TABLE prod.db.sample (
> id bigint COMMENT 'unique id',
> data string)
> USING iceberg
> TBLPROPERTIES (
> 'format.version' = '2'
> );
>
> If so, then I will give a binding +1 from my side because I don't have
> other particular changes that need to be introduced in the v2 now.
>
> For exposing the v2 to end users, I think we could also try to merge the
> PR and clarify v2 as an experiential feature, because I found many people
> are trying to test and benchmark the v2 feature based on the practice from
> https://github.com/apache/iceberg/pull/2410. Using the
> meta.upgradeToFormatVersion(2) was quite unfriendly for users to test this
> v2 feature now.
>
> Thanks.
>
>
>
> On Wed, Jul 28, 2021 at 1:09 PM Jack Ye <ye...@gmail.com> wrote:
>
>> (non-binding) +1, this looks like a clean cut to me. We have been testing
>> v2 internally for quite a while now, hopefully it can become the new
>> default version soon to enable row level delete and update.
>>
>> -Jack Ye
>>
>>
>> On Tue, Jul 27, 2021 at 9:59 AM Ryan Blue <bl...@apache.org> wrote:
>>
>>> I’d like to propose that we adopt the pending v2 spec changes as the
>>> supported v2 spec. The full list of changes is documented in the v2
>>> summary section of the spec <https://iceberg.apache.org/spec/#version-2>
>>> .
>>>
>>> The major breaking change is the addition of delete files and metadata
>>> to track delete files. In addition, there are a few other minor breaking
>>> changes. For example, v2 drops the block_size_in_bytes field in
>>> manifests that was previously required and also omits fields in table
>>> metadata that are now tracked by lists; schema is no longer written in
>>> favor of schemas. Other changes are forward compatible, mostly
>>> tightening field requirements where possible (e.g., schemas and
>>> current-schema-id are now required).
>>>
>>> Adopting the changes will signal that the community intends to support
>>> the current set of changes and will guarantee forward-compatibility for v2
>>> tables that implement the current v2 spec. Any new breaking changes would
>>> go into v3.
>>>
>>> Please vote on adopting the v2 changes in the next 72 hours.
>>>
>>> [ ] +1 Adopt the changes as v2
>>> [ ] +0
>>> [ ] -1 Do not adopt the changes, because…
>>> --
>>> Ryan Blue
>>>
>>
--
Ryan Blue
Tabular
Re: [VOTE] Adopt the v2 spec changes
Posted by OpenInx <op...@gmail.com>.
> adopt the pending v2 spec changes as the supported v2 spec
I assume this vote wants to reach the consistency between the community
members that we won't introduce any breaking changes in v2 spec, not
discuss exposing v2 to SQL tables like the following, right ?
CREATE TABLE prod.db.sample (
id bigint COMMENT 'unique id',
data string)
USING iceberg
TBLPROPERTIES (
'format.version' = '2'
);
If so, then I will give a binding +1 from my side because I don't have
other particular changes that need to be introduced in the v2 now.
For exposing the v2 to end users, I think we could also try to merge the
PR and clarify v2 as an experiential feature, because I found many people
are trying to test and benchmark the v2 feature based on the practice from
https://github.com/apache/iceberg/pull/2410. Using the
meta.upgradeToFormatVersion(2) was quite unfriendly for users to test this
v2 feature now.
Thanks.
On Wed, Jul 28, 2021 at 1:09 PM Jack Ye <ye...@gmail.com> wrote:
> (non-binding) +1, this looks like a clean cut to me. We have been testing
> v2 internally for quite a while now, hopefully it can become the new
> default version soon to enable row level delete and update.
>
> -Jack Ye
>
>
> On Tue, Jul 27, 2021 at 9:59 AM Ryan Blue <bl...@apache.org> wrote:
>
>> I’d like to propose that we adopt the pending v2 spec changes as the
>> supported v2 spec. The full list of changes is documented in the v2
>> summary section of the spec <https://iceberg.apache.org/spec/#version-2>.
>>
>> The major breaking change is the addition of delete files and metadata to
>> track delete files. In addition, there are a few other minor breaking
>> changes. For example, v2 drops the block_size_in_bytes field in
>> manifests that was previously required and also omits fields in table
>> metadata that are now tracked by lists; schema is no longer written in
>> favor of schemas. Other changes are forward compatible, mostly
>> tightening field requirements where possible (e.g., schemas and
>> current-schema-id are now required).
>>
>> Adopting the changes will signal that the community intends to support
>> the current set of changes and will guarantee forward-compatibility for v2
>> tables that implement the current v2 spec. Any new breaking changes would
>> go into v3.
>>
>> Please vote on adopting the v2 changes in the next 72 hours.
>>
>> [ ] +1 Adopt the changes as v2
>> [ ] +0
>> [ ] -1 Do not adopt the changes, because…
>> --
>> Ryan Blue
>>
>
Re: [VOTE] Adopt the v2 spec changes
Posted by Jack Ye <ye...@gmail.com>.
(non-binding) +1, this looks like a clean cut to me. We have been testing
v2 internally for quite a while now, hopefully it can become the new
default version soon to enable row level delete and update.
-Jack Ye
On Tue, Jul 27, 2021 at 9:59 AM Ryan Blue <bl...@apache.org> wrote:
> I’d like to propose that we adopt the pending v2 spec changes as the
> supported v2 spec. The full list of changes is documented in the v2
> summary section of the spec <https://iceberg.apache.org/spec/#version-2>.
>
> The major breaking change is the addition of delete files and metadata to
> track delete files. In addition, there are a few other minor breaking
> changes. For example, v2 drops the block_size_in_bytes field in manifests
> that was previously required and also omits fields in table metadata that
> are now tracked by lists; schema is no longer written in favor of schemas.
> Other changes are forward compatible, mostly tightening field requirements
> where possible (e.g., schemas and current-schema-id are now required).
>
> Adopting the changes will signal that the community intends to support the
> current set of changes and will guarantee forward-compatibility for v2
> tables that implement the current v2 spec. Any new breaking changes would
> go into v3.
>
> Please vote on adopting the v2 changes in the next 72 hours.
>
> [ ] +1 Adopt the changes as v2
> [ ] +0
> [ ] -1 Do not adopt the changes, because…
> --
> Ryan Blue
>