You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/10 15:56:01 UTC
[GitHub] [hudi] leesf opened a new pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
leesf opened a new pull request #3964:
URL: https://github.com/apache/hudi/pull/3964
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-1038793907
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
}, {
"hash" : "3f6276e3fbe583813bcf27a5165afddac108b117",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f6276e3fbe583813bcf27a5165afddac108b117",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0c9b18702c77c910aa2a808a516f4e9fad53575 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424)
* 3f6276e3fbe583813bcf27a5165afddac108b117 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#discussion_r765378460
##########
File path: rfc/README.md
##########
@@ -59,4 +59,5 @@ The list of all RFCs can be found here.
| 33 | [Hudi supports more comprehensive Schema Evolution](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+33++Hudi+supports+more+comprehensive+Schema+Evolution) | `IN PROGRESS` |
| 34 | [Hudi BigQuery Integration (WIP)](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=188745980) | `UNDER REVIEW` |
| 35 | [Make Flink MOR table writing streaming friendly](https://cwiki.apache.org/confluence/display/HUDI/RFC-35%3A+Make+Flink+MOR+table+writing+streaming+friendly) | `UNDER REVIEW` |
-| 36 | [HUDI Metastore Server](https://cwiki.apache.org/confluence/display/HUDI/%5BWIP%5D+RFC-36%3A+HUDI+Metastore+Server) | `UNDER REVIEW` |
\ No newline at end of file
+| 36 | [HUDI Metastore Server](https://cwiki.apache.org/confluence/display/HUDI/%5BWIP%5D+RFC-36%3A+HUDI+Metastore+Server) | `UNDER REVIEW` |
+| 38 | [Spark DataSource V2 Integration](./rfc-38/rfc-38.md) | `UNDER REVIEW` |
Review comment:
In fact the PR has been merged https://github.com/apache/hudi/pull/3964, will update the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#discussion_r805597905
##########
File path: rfc/rfc-38/rfc-38.md
##########
@@ -0,0 +1,240 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-38: Spark Datasource V2 Integration
+
+## Proposers
+
+- @leesf
+
+## Approvers
+- @vinothchandar
+- @xishiyan
+- @YannByron
+
+## Status
+
+JIRA: https://issues.apache.org/jira/browse/HUDI-1297
+
+## Abstract
+
+Today, Hudi still uses V1 api and relies heavily on RDD api to index, repartition and so on given the flexibility of RDD api,
+it works fine in v1 api, using datasource V1 api, Hudi provides complete read/write, update,
+and small file auto handling capabilities, all things work well.
+However, with the continuous development and evolving of datasource V2 api,
+the datasource v2 api has stabilized.Taking into account the datasource v1 api is too old and the spark community
+no longer spends more resources to maintain v1 api, so consider migrating to DataSource V2 api,
+and use more pushdown filters provided by V2 api and
+integrate with [RFC-27](https://cwiki.apache.org/confluence/display/HUDI/RFC-27+Data+skipping+index+to+improve+query+performance)
+to provide more powerful query capabilities. Also we could leverage it after V2 api get evolved or optimized again.
+
+
+## Background
+
+The current Hudi read and write paths use DataSource V1 api, and the implementation class is `DefaultSource`
+
+```scala
+/**
+* Hoodie Spark Datasource, for reading and writing hoodie tables
+*
+*/
+class DefaultSource extends RelationProvider
+with SchemaRelationProvider
+with CreatableRelationProvider
+with DataSourceRegister
+with StreamSinkProvider
+with StreamSourceProvider
+with Serializable {
+...
+}
+```
+
+As for writing(batch write), the following method will be called.
+```scala
+override def createRelation(sqlContext: SQLContext,
+mode: SaveMode,
+optParams: Map[String, String],
+df: DataFrame): BaseRelation = {
+val parameters = HoodieWriterUtils.parametersWithWriteDefaults(optParams)
+val translatedOptions = DataSourceWriteOptions.translateSqlOptions(parameters)
+val dfWithoutMetaCols = df.drop(HoodieRecord.HOODIE_META_COLUMNS.asScala:_*)
+
+ if (translatedOptions(OPERATION.key).equals(BOOTSTRAP_OPERATION_OPT_VAL)) {
+ HoodieSparkSqlWriter.bootstrap(sqlContext, mode, translatedOptions, dfWithoutMetaCols)
+ } else {
+ HoodieSparkSqlWriter.write(sqlContext, mode, translatedOptions, dfWithoutMetaCols)
+ }
+ new HoodieEmptyRelation(sqlContext, dfWithoutMetaCols.schema)
+}
+```
+
+Regarding querying, the following method will return a `BaseRelation`（if not provide schema）
+
+```scala
+override def createRelation(sqlContext: SQLContext,
+parameters: Map[String, String]): BaseRelation = {
+createRelation(sqlContext, parameters, null)
+}
+```
+
+For streaming writing and reading, DefaultSource#createSink and DefaultSource#createSink are called respectively.
+In 0.9.0 version , the bulk_insert row mode was introduced to speed up bulk_insert, which implements the `SupportsWrite` v2 api and uses `HoodieDataSourceInternalTable` for writing,
+right now only bulk_insert operation is supported.
+
+## Implementation
+
+Spark provides a complete V2 api, such as `CatalogPlugin`, `SupportsWrite`, `SupportsRead`, and various pushdown filters,
+such as `SupportsPushDownFilters`, `SupportsPushDownAggregates`, `SupportsPushDownRequiredColumns`
+
+We would Define the key abstraction of call `HoodieDatasourceTable`, which inherits the `Table`, `SupportsWrite`, `SupportsRead`
+interfaces to provide writing and reading capabilities.
+
+### Writing Path
+
+Hudi relies heavily on some RDD APIs on the write path, such as the indexing to determine where the record is update or insert,
+this refactoring work is relatively large under datasource v2 api, so we can firstly falllback to write to v1
+since Spark provides the V1WriteBuilder interface to bridge the V1 and V2 api, we could implement this in the first stage
+and perform further reconstruction after there is a better way to deal with indexing and repartitoining with RDD api later.
+The workflow of writing process of the Datasource V2 can refer to the following figure,
+where the `DataWriterFactory` is in the Driver and the `DataWriter` that performs writing in the Executor.
+
+
+
+The writing path code sample is below
+
+```scala
+class HoodieDatasourceTable extends Table with SupportsWrite {
+
+override def name(): String = {
+//
+}
+
+override def schema(): StructType = {
+// get hudi table schema
+}
+
+override def partitioning(): Array[Transform] = {
+// get partitioning of hudi table.
+}
+
+override def capabilities(): Set[TableCapability] = {
+// Set(BATCH_WRITE, BATCH_READ,TRUNCATE,...)
+}
+
+override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
+// V1WriteBuilder
+}
+
+override def newScanBuilder(CaseInsensitiveStringMap options): ScanBuilder = {
+// HoodieScanBuilder
+}
+
+}
+```
+
+### Querying path
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-965476792
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ad8df3dd3c31f10f997e2a8d7a8a7aced720a719 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-996545185
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ad8df3dd3c31f10f997e2a8d7a8a7aced720a719 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288)
* a0c9b18702c77c910aa2a808a516f4e9fad53575 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-996623073
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0c9b18702c77c910aa2a808a516f4e9fad53575 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-989335863
@leesf Love to understand the plan going forward here and how we plan to migrate the existing v1 write path onto the v2 APIs. Specifically, current v1 upsert pipeline consists of the following logical stages ` preCombine -> index -> partition -> write` before committing out the files. In other words, we benefit from v1 API providing ways to shuffle the dataframe further before writing to disk and IIUC v2 takes this flexibility away?
Assuming I am correct (and spark has not introduced any new APIs that help us mitigate this), should we do the following?
- introduce a new `hudiv2` datasource i.e `spark.write.format("hudiv2")` that just supports bulk_insert on the datasource write path.
- We also add a new `SparkDatasetWriteClient` which exposes methods for upsert,delete, .. and we use that as the basis for our SQL/DML layer as well.
- We continue to support the v1 `hudi` datasource as-is for sometime. There are lots of users who like how they can do upserts/deletes by executing a `spark.write.format("hudi").option()...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-965565782
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ad8df3dd3c31f10f997e2a8d7a8a7aced720a719 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-1038793907
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
}, {
"hash" : "3f6276e3fbe583813bcf27a5165afddac108b117",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f6276e3fbe583813bcf27a5165afddac108b117",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0c9b18702c77c910aa2a808a516f4e9fad53575 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424)
* 3f6276e3fbe583813bcf27a5165afddac108b117 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf merged pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
leesf merged pull request #3964:
URL: https://github.com/apache/hudi/pull/3964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-996623073
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0c9b18702c77c910aa2a808a516f4e9fad53575 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-996543454
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ad8df3dd3c31f10f997e2a8d7a8a7aced720a719 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288)
* a0c9b18702c77c910aa2a808a516f4e9fad53575 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-996543454
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ad8df3dd3c31f10f997e2a8d7a8a7aced720a719 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288)
* a0c9b18702c77c910aa2a808a516f4e9fad53575 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#discussion_r805501123
##########
File path: rfc/rfc-38/rfc-38.md
##########
@@ -0,0 +1,240 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-38: Spark Datasource V2 Integration
+
+## Proposers
+
+- @leesf
+
+## Approvers
+- @vinothchandar
+- @xishiyan
+- @YannByron
+
+## Status
+
+JIRA: https://issues.apache.org/jira/browse/HUDI-1297
+
+## Abstract
+
+Today, Hudi still uses V1 api and relies heavily on RDD api to index, repartition and so on given the flexibility of RDD api,
+it works fine in v1 api, using datasource V1 api, Hudi provides complete read/write, update,
+and small file auto handling capabilities, all things work well.
+However, with the continuous development and evolving of datasource V2 api,
+the datasource v2 api has stabilized.Taking into account the datasource v1 api is too old and the spark community
+no longer spends more resources to maintain v1 api, so consider migrating to DataSource V2 api,
+and use more pushdown filters provided by V2 api and
+integrate with [RFC-27](https://cwiki.apache.org/confluence/display/HUDI/RFC-27+Data+skipping+index+to+improve+query+performance)
+to provide more powerful query capabilities. Also we could leverage it after V2 api get evolved or optimized again.
+
+
+## Background
+
+The current Hudi read and write paths use DataSource V1 api, and the implementation class is `DefaultSource`
+
+```scala
+/**
+* Hoodie Spark Datasource, for reading and writing hoodie tables
+*
+*/
+class DefaultSource extends RelationProvider
+with SchemaRelationProvider
+with CreatableRelationProvider
+with DataSourceRegister
+with StreamSinkProvider
+with StreamSourceProvider
+with Serializable {
+...
+}
+```
+
+As for writing(batch write), the following method will be called.
+```scala
+override def createRelation(sqlContext: SQLContext,
+mode: SaveMode,
+optParams: Map[String, String],
+df: DataFrame): BaseRelation = {
+val parameters = HoodieWriterUtils.parametersWithWriteDefaults(optParams)
+val translatedOptions = DataSourceWriteOptions.translateSqlOptions(parameters)
+val dfWithoutMetaCols = df.drop(HoodieRecord.HOODIE_META_COLUMNS.asScala:_*)
+
+ if (translatedOptions(OPERATION.key).equals(BOOTSTRAP_OPERATION_OPT_VAL)) {
+ HoodieSparkSqlWriter.bootstrap(sqlContext, mode, translatedOptions, dfWithoutMetaCols)
+ } else {
+ HoodieSparkSqlWriter.write(sqlContext, mode, translatedOptions, dfWithoutMetaCols)
+ }
+ new HoodieEmptyRelation(sqlContext, dfWithoutMetaCols.schema)
+}
+```
+
+Regarding querying, the following method will return a `BaseRelation`（if not provide schema）
+
+```scala
+override def createRelation(sqlContext: SQLContext,
+parameters: Map[String, String]): BaseRelation = {
+createRelation(sqlContext, parameters, null)
+}
+```
+
+For streaming writing and reading, DefaultSource#createSink and DefaultSource#createSink are called respectively.
+In 0.9.0 version , the bulk_insert row mode was introduced to speed up bulk_insert, which implements the `SupportsWrite` v2 api and uses `HoodieDataSourceInternalTable` for writing,
+right now only bulk_insert operation is supported.
+
+## Implementation
+
+Spark provides a complete V2 api, such as `CatalogPlugin`, `SupportsWrite`, `SupportsRead`, and various pushdown filters,
+such as `SupportsPushDownFilters`, `SupportsPushDownAggregates`, `SupportsPushDownRequiredColumns`
+
+We would Define the key abstraction of call `HoodieDatasourceTable`, which inherits the `Table`, `SupportsWrite`, `SupportsRead`
+interfaces to provide writing and reading capabilities.
+
+### Writing Path
+
+Hudi relies heavily on some RDD APIs on the write path, such as the indexing to determine where the record is update or insert,
+this refactoring work is relatively large under datasource v2 api, so we can firstly falllback to write to v1
+since Spark provides the V1WriteBuilder interface to bridge the V1 and V2 api, we could implement this in the first stage
+and perform further reconstruction after there is a better way to deal with indexing and repartitoining with RDD api later.
+The workflow of writing process of the Datasource V2 can refer to the following figure,
Review comment:
as discussed, we will keep falling back to v1 for writing and don't have plan to implement v2 writing. can you update this part please?
##########
File path: rfc/rfc-38/rfc-38.md
##########
@@ -0,0 +1,240 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-38: Spark Datasource V2 Integration
+
+## Proposers
+
+- @leesf
+
+## Approvers
+- @vinothchandar
+- @xishiyan
+- @YannByron
+
+## Status
+
+JIRA: https://issues.apache.org/jira/browse/HUDI-1297
+
+## Abstract
+
+Today, Hudi still uses V1 api and relies heavily on RDD api to index, repartition and so on given the flexibility of RDD api,
+it works fine in v1 api, using datasource V1 api, Hudi provides complete read/write, update,
+and small file auto handling capabilities, all things work well.
+However, with the continuous development and evolving of datasource V2 api,
+the datasource v2 api has stabilized.Taking into account the datasource v1 api is too old and the spark community
+no longer spends more resources to maintain v1 api, so consider migrating to DataSource V2 api,
+and use more pushdown filters provided by V2 api and
+integrate with [RFC-27](https://cwiki.apache.org/confluence/display/HUDI/RFC-27+Data+skipping+index+to+improve+query+performance)
+to provide more powerful query capabilities. Also we could leverage it after V2 api get evolved or optimized again.
+
+
+## Background
+
+The current Hudi read and write paths use DataSource V1 api, and the implementation class is `DefaultSource`
+
+```scala
+/**
+* Hoodie Spark Datasource, for reading and writing hoodie tables
+*
+*/
+class DefaultSource extends RelationProvider
+with SchemaRelationProvider
+with CreatableRelationProvider
+with DataSourceRegister
+with StreamSinkProvider
+with StreamSourceProvider
+with Serializable {
+...
+}
+```
+
+As for writing(batch write), the following method will be called.
+```scala
+override def createRelation(sqlContext: SQLContext,
+mode: SaveMode,
+optParams: Map[String, String],
+df: DataFrame): BaseRelation = {
+val parameters = HoodieWriterUtils.parametersWithWriteDefaults(optParams)
+val translatedOptions = DataSourceWriteOptions.translateSqlOptions(parameters)
+val dfWithoutMetaCols = df.drop(HoodieRecord.HOODIE_META_COLUMNS.asScala:_*)
+
+ if (translatedOptions(OPERATION.key).equals(BOOTSTRAP_OPERATION_OPT_VAL)) {
+ HoodieSparkSqlWriter.bootstrap(sqlContext, mode, translatedOptions, dfWithoutMetaCols)
+ } else {
+ HoodieSparkSqlWriter.write(sqlContext, mode, translatedOptions, dfWithoutMetaCols)
+ }
+ new HoodieEmptyRelation(sqlContext, dfWithoutMetaCols.schema)
+}
+```
+
+Regarding querying, the following method will return a `BaseRelation`（if not provide schema）
+
+```scala
+override def createRelation(sqlContext: SQLContext,
+parameters: Map[String, String]): BaseRelation = {
+createRelation(sqlContext, parameters, null)
+}
+```
+
+For streaming writing and reading, DefaultSource#createSink and DefaultSource#createSink are called respectively.
+In 0.9.0 version , the bulk_insert row mode was introduced to speed up bulk_insert, which implements the `SupportsWrite` v2 api and uses `HoodieDataSourceInternalTable` for writing,
+right now only bulk_insert operation is supported.
+
+## Implementation
+
+Spark provides a complete V2 api, such as `CatalogPlugin`, `SupportsWrite`, `SupportsRead`, and various pushdown filters,
+such as `SupportsPushDownFilters`, `SupportsPushDownAggregates`, `SupportsPushDownRequiredColumns`
+
+We would Define the key abstraction of call `HoodieDatasourceTable`, which inherits the `Table`, `SupportsWrite`, `SupportsRead`
+interfaces to provide writing and reading capabilities.
+
+### Writing Path
+
+Hudi relies heavily on some RDD APIs on the write path, such as the indexing to determine where the record is update or insert,
+this refactoring work is relatively large under datasource v2 api, so we can firstly falllback to write to v1
+since Spark provides the V1WriteBuilder interface to bridge the V1 and V2 api, we could implement this in the first stage
+and perform further reconstruction after there is a better way to deal with indexing and repartitoining with RDD api later.
+The workflow of writing process of the Datasource V2 can refer to the following figure,
+where the `DataWriterFactory` is in the Driver and the `DataWriter` that performs writing in the Executor.
+
+
+
+The writing path code sample is below
+
+```scala
+class HoodieDatasourceTable extends Table with SupportsWrite {
+
+override def name(): String = {
+//
+}
+
+override def schema(): StructType = {
+// get hudi table schema
+}
+
+override def partitioning(): Array[Transform] = {
+// get partitioning of hudi table.
+}
+
+override def capabilities(): Set[TableCapability] = {
+// Set(BATCH_WRITE, BATCH_READ,TRUNCATE,...)
+}
+
+override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
+// V1WriteBuilder
+}
+
+override def newScanBuilder(CaseInsensitiveStringMap options): ScanBuilder = {
+// HoodieScanBuilder
+}
+
+}
+```
+
+### Querying path
Review comment:
can you help summarize some high-level benefits and features we'll get by adopting v2 for reading? it'll be nice to have them highlighted in the RFC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#discussion_r805598314
##########
File path: rfc/rfc-38/rfc-38.md
##########
@@ -0,0 +1,240 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-38: Spark Datasource V2 Integration
+
+## Proposers
+
+- @leesf
+
+## Approvers
+- @vinothchandar
+- @xishiyan
+- @YannByron
+
+## Status
+
+JIRA: https://issues.apache.org/jira/browse/HUDI-1297
+
+## Abstract
+
+Today, Hudi still uses V1 api and relies heavily on RDD api to index, repartition and so on given the flexibility of RDD api,
+it works fine in v1 api, using datasource V1 api, Hudi provides complete read/write, update,
+and small file auto handling capabilities, all things work well.
+However, with the continuous development and evolving of datasource V2 api,
+the datasource v2 api has stabilized.Taking into account the datasource v1 api is too old and the spark community
+no longer spends more resources to maintain v1 api, so consider migrating to DataSource V2 api,
+and use more pushdown filters provided by V2 api and
+integrate with [RFC-27](https://cwiki.apache.org/confluence/display/HUDI/RFC-27+Data+skipping+index+to+improve+query+performance)
+to provide more powerful query capabilities. Also we could leverage it after V2 api get evolved or optimized again.
+
+
+## Background
+
+The current Hudi read and write paths use DataSource V1 api, and the implementation class is `DefaultSource`
+
+```scala
+/**
+* Hoodie Spark Datasource, for reading and writing hoodie tables
+*
+*/
+class DefaultSource extends RelationProvider
+with SchemaRelationProvider
+with CreatableRelationProvider
+with DataSourceRegister
+with StreamSinkProvider
+with StreamSourceProvider
+with Serializable {
+...
+}
+```
+
+As for writing(batch write), the following method will be called.
+```scala
+override def createRelation(sqlContext: SQLContext,
+mode: SaveMode,
+optParams: Map[String, String],
+df: DataFrame): BaseRelation = {
+val parameters = HoodieWriterUtils.parametersWithWriteDefaults(optParams)
+val translatedOptions = DataSourceWriteOptions.translateSqlOptions(parameters)
+val dfWithoutMetaCols = df.drop(HoodieRecord.HOODIE_META_COLUMNS.asScala:_*)
+
+ if (translatedOptions(OPERATION.key).equals(BOOTSTRAP_OPERATION_OPT_VAL)) {
+ HoodieSparkSqlWriter.bootstrap(sqlContext, mode, translatedOptions, dfWithoutMetaCols)
+ } else {
+ HoodieSparkSqlWriter.write(sqlContext, mode, translatedOptions, dfWithoutMetaCols)
+ }
+ new HoodieEmptyRelation(sqlContext, dfWithoutMetaCols.schema)
+}
+```
+
+Regarding querying, the following method will return a `BaseRelation`（if not provide schema）
+
+```scala
+override def createRelation(sqlContext: SQLContext,
+parameters: Map[String, String]): BaseRelation = {
+createRelation(sqlContext, parameters, null)
+}
+```
+
+For streaming writing and reading, DefaultSource#createSink and DefaultSource#createSink are called respectively.
+In 0.9.0 version , the bulk_insert row mode was introduced to speed up bulk_insert, which implements the `SupportsWrite` v2 api and uses `HoodieDataSourceInternalTable` for writing,
+right now only bulk_insert operation is supported.
+
+## Implementation
+
+Spark provides a complete V2 api, such as `CatalogPlugin`, `SupportsWrite`, `SupportsRead`, and various pushdown filters,
+such as `SupportsPushDownFilters`, `SupportsPushDownAggregates`, `SupportsPushDownRequiredColumns`
+
+We would Define the key abstraction of call `HoodieDatasourceTable`, which inherits the `Table`, `SupportsWrite`, `SupportsRead`
+interfaces to provide writing and reading capabilities.
+
+### Writing Path
+
+Hudi relies heavily on some RDD APIs on the write path, such as the indexing to determine where the record is update or insert,
+this refactoring work is relatively large under datasource v2 api, so we can firstly falllback to write to v1
+since Spark provides the V1WriteBuilder interface to bridge the V1 and V2 api, we could implement this in the first stage
+and perform further reconstruction after there is a better way to deal with indexing and repartitoining with RDD api later.
+The workflow of writing process of the Datasource V2 can refer to the following figure,
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-965470612
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ad8df3dd3c31f10f997e2a8d7a8a7aced720a719 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#discussion_r765339309
##########
File path: rfc/README.md
##########
@@ -59,4 +59,5 @@ The list of all RFCs can be found here.
| 33 | [Hudi supports more comprehensive Schema Evolution](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+33++Hudi+supports+more+comprehensive+Schema+Evolution) | `IN PROGRESS` |
| 34 | [Hudi BigQuery Integration (WIP)](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=188745980) | `UNDER REVIEW` |
| 35 | [Make Flink MOR table writing streaming friendly](https://cwiki.apache.org/confluence/display/HUDI/RFC-35%3A+Make+Flink+MOR+table+writing+streaming+friendly) | `UNDER REVIEW` |
-| 36 | [HUDI Metastore Server](https://cwiki.apache.org/confluence/display/HUDI/%5BWIP%5D+RFC-36%3A+HUDI+Metastore+Server) | `UNDER REVIEW` |
\ No newline at end of file
+| 36 | [HUDI Metastore Server](https://cwiki.apache.org/confluence/display/HUDI/%5BWIP%5D+RFC-36%3A+HUDI+Metastore+Server) | `UNDER REVIEW` |
+| 38 | [Spark DataSource V2 Integration](./rfc-38/rfc-38.md) | `UNDER REVIEW` |
Review comment:
lets separate the number update into a different PR? as mentioned in the process?
##########
File path: rfc/rfc-38/rfc-38.md
##########
@@ -0,0 +1,238 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-38: Spark Datasource V2 Integration
+
+## Proposers
+
+- @leesf
+
+## Approvers
+-
Review comment:
@xushiyan @YannByron and I can be approvers if you don't mind
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-965565782
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ad8df3dd3c31f10f997e2a8d7a8a7aced720a719 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-1038796696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
}, {
"hash" : "3f6276e3fbe583813bcf27a5165afddac108b117",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5980",
"triggerID" : "3f6276e3fbe583813bcf27a5165afddac108b117",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0c9b18702c77c910aa2a808a516f4e9fad53575 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424)
* 3f6276e3fbe583813bcf27a5165afddac108b117 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5980)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-1038927565
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
}, {
"hash" : "3f6276e3fbe583813bcf27a5165afddac108b117",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5980",
"triggerID" : "3f6276e3fbe583813bcf27a5165afddac108b117",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3f6276e3fbe583813bcf27a5165afddac108b117 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5980)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
leesf commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-965467633
CC @vinothchandar @bvaradar @nsivabalan @codope @umehrot2 @YannByron @li36909 @EdwinGuo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-965476792
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ad8df3dd3c31f10f997e2a8d7a8a7aced720a719 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-965470612
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ad8df3dd3c31f10f997e2a8d7a8a7aced720a719 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-1019640278
@leesf : I did not go through the lineage of this patch. But I do know we landed the another PR related to spark datasource V2. so, is this patch still valid or can we close it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-1025423884
> @leesf : I did not go through the lineage of this patch. But I do know we landed the another PR related to spark datasource V2. so, is this patch still valid or can we close it out.
@nsivabalan this is the RFC PR. Current work is in https://github.com/apache/hudi/pull/4611
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-1038796696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
}, {
"hash" : "3f6276e3fbe583813bcf27a5165afddac108b117",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5980",
"triggerID" : "3f6276e3fbe583813bcf27a5165afddac108b117",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0c9b18702c77c910aa2a808a516f4e9fad53575 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424)
* 3f6276e3fbe583813bcf27a5165afddac108b117 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5980)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
leesf commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-989431132
> @leesf Love to understand the plan going forward here and how we plan to migrate the existing v1 write path onto the v2 APIs. Specifically, current v1 upsert pipeline consists of the following logical stages ` preCombine -> index -> partition -> write` before committing out the files. In other words, we benefit from v1 API providing ways to shuffle the dataframe further before writing to disk and IIUC v2 takes this flexibility away?
>
> Assuming I am correct (and spark has not introduced any new APIs that help us mitigate this), should we do the following?
>
> * introduce a new `hudiv2` datasource i.e `spark.write.format("hudiv2")` that just supports bulk_insert on the datasource write path.
> * We also add a new `SparkDatasetWriteClient` which exposes methods for upsert,delete, .. and we use that as the basis for our SQL/DML layer as well.
> * We continue to support the v1 `hudi` datasource as-is for sometime. There are lots of users who like how they can do upserts/deletes by executing a `spark.write.format("hudi").option()...`
@vinothchandar In fact, I do not intend to introduce "hudiv2" format when introducing V2 code path, since it will make end users change their code and the "hudiv2" is not a good name("hudi" is good enough) IMO, instead I would like to change the former "hudi" format into "hudi_internal" and make "hudi" format as the v2 code path as default to make it transparent for end users, and integrate with current bulk_insert V2 write path.
And In the first phase, we would fallback to V1 write path while introduce V2 interface(HoodieCatalog and HoodieInternalTableV2), and integrate with current bulk_insert V2 write path. In the second phase, we would explore the way to integrate with `SparkDatasetWriteClient` which @xushiyan did a PoC to make it purely V2 code path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#discussion_r765378520
##########
File path: rfc/rfc-38/rfc-38.md
##########
@@ -0,0 +1,238 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-38: Spark Datasource V2 Integration
+
+## Proposers
+
+- @leesf
+
+## Approvers
+-
Review comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-990413082
>And In the first phase, we would fallback to V1 write path
Can this be done? Love to see some code for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-996545185
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288",
"triggerID" : "ad8df3dd3c31f10f997e2a8d7a8a7aced720a719",
"triggerType" : "PUSH"
}, {
"hash" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424",
"triggerID" : "a0c9b18702c77c910aa2a808a516f4e9fad53575",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ad8df3dd3c31f10f997e2a8d7a8a7aced720a719 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3288)
* a0c9b18702c77c910aa2a808a516f4e9fad53575 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf edited a comment on pull request #3964: [HUDI-2732][RFC-38] Spark Datasource V2 Integration
Posted by GitBox <gi...@apache.org>.
leesf edited a comment on pull request #3964:
URL: https://github.com/apache/hudi/pull/3964#issuecomment-965467633
CC @vinothchandar @bvaradar @nsivabalan @codope @umehrot2 @YannByron @li36909 @EdwinGuo @xushiyan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org