You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/06/29 08:44:12 UTC
[GitHub] [incubator-kyuubi] cfmcgrady commented on pull request #2963: Bump Delta 2.0.0rc1
cfmcgrady commented on PR #2963:
URL: https://github.com/apache/incubator-kyuubi/pull/2963#issuecomment-1169701462
> Does DeltaLake provide some new features in 2.0? It's better to add some tests for those features as well
The key features in the changelog are shown as follows
1. Support Change Data Feed on Delta tables. Change Data Feed represents the row level changes between different versions of the table. When enabled, additional information is recorded regarding row level changes for every write operation on the table. See the documentation for more details.
2. Support Z-Order clustering of data to reduce the amount of data read. Z-Ordering is a technique to colocate related information in the same set of files. This data clustering allows column stats (released in Delta 1.2) to be more effective in skipping data based on filters in a query. See the documentation for more details.
3. Support for idempotent writes to Delta tables to enable fault-tolerant retry of Delta table writing jobs without writing the data multiple times to the table. See the documentation for more details.
4. Support for dropping columns in a Delta table as a metadata change operation. This command drops the column from metadata and not the column data in underlying files. See documentation for more details.
5. Support for dynamic partition overwrite. Overwrite only the partitions with data written into them at runtime. See documentation for details.
6. Experimental support for multi-part checkpoints to split the Delta Lake checkpoint into multiple parts to speed up writing the checkpoints and reading. See documentation for more details.
7. Python and Scala API support for OPTIMIZE file compaction and Z-order by.
We can add the tests in a separate PR, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org