You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by xu...@apache.org on 2022/04/23 05:24:49 UTC
[hudi] branch release-0.11.0 updated (25501c99e9 -> 91bac98bcc)
This is an automated email from the ASF dual-hosted git repository.
xushiyan pushed a change to branch release-0.11.0
in repository https://gitbox.apache.org/repos/asf/hudi.git
from 25501c99e9 [HUDI-3934] Fix `Spark32HoodieParquetFileFormat` not being compatible w/ Spark 3.2.0 (#5378)
new 4081e99254 [DOCS] Add commit activity, twitter badgers, and Hudi logo in README (#5336)
new aee15623ce [HUDI-3947] Fixing Hive conf usage in HoodieSparkSqlWriter (#5401)
new 91bac98bcc [HUDI-3950] add parquet-avro to gcp-bundle (#5399)
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
README.md | 12 +++++++++---
.../main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala | 6 +-----
packaging/hudi-gcp-bundle/pom.xml | 1 +
3 files changed, 11 insertions(+), 8 deletions(-)
[hudi] 03/03: [HUDI-3950] add parquet-avro to gcp-bundle (#5399)
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xushiyan pushed a commit to branch release-0.11.0
in repository https://gitbox.apache.org/repos/asf/hudi.git
commit 91bac98bcc2d8b478a523bf0536259dbce223653
Author: Raymond Xu <27...@users.noreply.github.com>
AuthorDate: Fri Apr 22 20:59:49 2022 -0700
[HUDI-3950] add parquet-avro to gcp-bundle (#5399)
---
packaging/hudi-gcp-bundle/pom.xml | 1 +
1 file changed, 1 insertion(+)
diff --git a/packaging/hudi-gcp-bundle/pom.xml b/packaging/hudi-gcp-bundle/pom.xml
index 54b4bbdd67..7ef13cd180 100644
--- a/packaging/hudi-gcp-bundle/pom.xml
+++ b/packaging/hudi-gcp-bundle/pom.xml
@@ -95,6 +95,7 @@
<include>org.apache.hudi:hudi-hadoop-mr</include>
<include>org.apache.hudi:hudi-sync-common</include>
<include>org.apache.hudi:hudi-gcp</include>
+ <include>org.apache.parquet:parquet-avro</include>
<include>com.google.cloud:google-cloud-bigquery</include>
<include>com.beust:jcommander</include>
[hudi] 01/03: [DOCS] Add commit activity, twitter badgers, and Hudi logo in README (#5336)
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xushiyan pushed a commit to branch release-0.11.0
in repository https://gitbox.apache.org/repos/asf/hudi.git
commit 4081e99254a6a95f7dc367f503addf1a28bf6249
Author: Y Ethan Guo <et...@gmail.com>
AuthorDate: Fri Apr 22 01:51:07 2022 -0700
[DOCS] Add commit activity, twitter badgers, and Hudi logo in README (#5336)
---
README.md | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/README.md b/README.md
index 0bb158a816..07a2201c96 100644
--- a/README.md
+++ b/README.md
@@ -16,8 +16,11 @@
-->
# Apache Hudi
-Apache Hudi (pronounced Hoodie) stands for `Hadoop Upserts Deletes and Incrementals`.
-Hudi manages the storage of large analytical datasets on DFS (Cloud stores, HDFS or any Hadoop FileSystem compatible storage).
+
+Apache Hudi (pronounced Hoodie) stands for `Hadoop Upserts Deletes and Incrementals`. Hudi manages the storage of large
+analytical datasets on DFS (Cloud stores, HDFS or any Hadoop FileSystem compatible storage).
+
+<img src="https://hudi.apache.org/assets/images/hudi-logo-medium.png" alt="Hudi logo" height="80px" align="right" />
<https://hudi.apache.org/>
@@ -25,12 +28,15 @@ Hudi manages the storage of large analytical datasets on DFS (Cloud stores, HDFS
[](https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/latest?definitionId=3&branchName=master)
[](https://www.apache.org/licenses/LICENSE-2.0.html)
[](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.hudi%22)
+
[](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE)
+
## Features
+
* Upsert support with fast, pluggable indexing
* Atomically publish data with rollback support
-* Snapshot isolation between writer & queries
+* Snapshot isolation between writer & queries
* Savepoints for data recovery
* Manages file sizes, layout using statistics
* Async compaction of row & columnar data
[hudi] 02/03: [HUDI-3947] Fixing Hive conf usage in HoodieSparkSqlWriter (#5401)
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xushiyan pushed a commit to branch release-0.11.0
in repository https://gitbox.apache.org/repos/asf/hudi.git
commit aee15623ce74de3b04e1c542215f238877fdff03
Author: Sivabalan Narayanan <n....@gmail.com>
AuthorDate: Fri Apr 22 22:20:05 2022 -0400
[HUDI-3947] Fixing Hive conf usage in HoodieSparkSqlWriter (#5401)
---
.../src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
index c5204fcd8e..2d9f4b45a2 100644
--- a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
+++ b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
@@ -21,7 +21,6 @@ import org.apache.avro.Schema
import org.apache.avro.generic.GenericRecord
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
-import org.apache.hadoop.hive.conf.HiveConf
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.HoodieConversionUtils.toProperties
import org.apache.hudi.HoodieWriterUtils._
@@ -619,11 +618,8 @@ object HoodieSparkSqlWriter {
properties.put(HoodieSyncConfig.META_SYNC_SPARK_VERSION.key, SPARK_VERSION)
properties.put(HoodieSyncConfig.META_SYNC_USE_FILE_LISTING_FROM_METADATA.key, hoodieConfig.getBoolean(HoodieMetadataConfig.ENABLE))
- val hiveConf: HiveConf = new HiveConf()
- hiveConf.addResource(fs.getConf)
-
syncClientToolClassSet.foreach(impl => {
- SyncUtilHelpers.runHoodieMetaSync(impl.trim, properties, hiveConf, fs, basePath.toString, baseFileFormat)
+ SyncUtilHelpers.runHoodieMetaSync(impl.trim, properties, fs.getConf, fs, basePath.toString, baseFileFormat)
})
}
true