You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by rahul c <rc...@gmail.com> on 2020/02/27 05:35:56 UTC

Clarification on the commit protocol

Hi team,

Just wanted to understand.
Is DBIO commit protocol available in open source spark version ?

Re: Clarification on the commit protocol

Posted by Michael Armbrust <mi...@databricks.com>.

No, it is not. Although the commit protocol has mostly been superseded by Delta
Lake <https://delta.io>, which is available as a separate open source
project that works natively with Apache Spark. In contrast to the commit
protocol, Delta can guarantee full ACID (rather than just partition level
atomicity). It also has better performance in many cases, as it reduces the
amount of metadata that needs to be retrieved from the storage system.

On Wed, Feb 26, 2020 at 9:36 PM rahul c <rc...@gmail.com> wrote:

> Hi team,
>
> Just wanted to understand.
> Is DBIO commit protocol available in open source spark version ?
>