You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/01/23 21:19:39 UTC
[GitHub] [incubator-hudi] leesf opened a new pull request #1277:
[WIP][HUDI-543] release notes for 0.5.1
leesf opened a new pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
Add release notes for 0.5.1
## Brief change log
*(for example:)*
- *Modify releases.md*
## Verify this pull request
This pull request is a trivial rework / code cleanup without any test coverage.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
cc @vinothchandar @bvaradar Please review in advance though it is still WIP.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370980822
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,46 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can use "repair overwrite-hoodie-props" to append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
Review comment:
we probably need to update configurations page as well? were there other configs added in 0.5.1? @bvaradar ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r372786581
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,46 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can use "repair overwrite-hoodie-props" to append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
+* CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+* DeltaStreamer CLI parameter for capturing table type is changed from --storage-type to --table-type. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+* Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring Kafka reset offset strategies with configuration(auto.offset.reset) in deltastreamer.
Review comment:
I think the 0.8 and 0.10 refer to the underlying kafka version.. Might be good to mention once.. you can make the final call. Dont have strong opinions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370981143
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
+ * Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring kafka reset offset strategies in deltastreamer.
+ * Need shade Avro if implement custom payload, which is similar to hudi-hadoop-mr-bundle.
+ * Better delete support in DeltaStreamer, would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Support for AWS Database Migration Service(DMS) in DeltaStreamer
Review comment:
meaning? moving them to the site? if so, is that work scoped?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] smarthi commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
smarthi commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370361992
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4 supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
Review comment:
we don't support Spark 2.4, only 2.4+ - is that what this means?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370980920
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,46 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can use "repair overwrite-hoodie-props" to append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
+* CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
Review comment:
add: specified file, for one-time updates to table name or even enabling the new timeline layout above. Note that few queries may temporarily fail while the overwrite happens (few milliseconds).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370980972
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,46 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can use "repair overwrite-hoodie-props" to append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
+* CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+* DeltaStreamer CLI parameter for capturing table type is changed from --storage-type to --table-type. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+* Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring Kafka reset offset strategies with configuration(auto.offset.reset) in deltastreamer.
Review comment:
May be also call out the Kafka version change above at the top?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370486630
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
Review comment:
Include the scala version. Package name must be org.apache.spark:spark-avro_2.11:2.4.4
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370450499
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
+ * Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring kafka reset offset strategies in deltastreamer.
+ * Need shade Avro if implement custom payload, which is similar to hudi-hadoop-mr-bundle.
+ * Better delete support in DeltaStreamer, would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
Review comment:
Remove the first comma?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370981109
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,46 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can use "repair overwrite-hoodie-props" to append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
+* CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+* DeltaStreamer CLI parameter for capturing table type is changed from --storage-type to --table-type. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+* Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring Kafka reset offset strategies with configuration(auto.offset.reset) in deltastreamer.
+* When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro_2.11:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+* Key generator moved to separate package under org.apache.hudi.keygen. If you are using overridden key generator classes (configuration ("hoodie.datasource.write.keygenerator.class")) that comes with hudi package, please ensure the fully qualified class name is changed accordingly.
+* Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
Review comment:
change to: in sync config to retain the old naming without the _ro suffix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1277: [WIP][HUDI-543]
release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#issuecomment-578570505
@vinothchandar Addressed most of your comments, and left some minor questions waiting for your thoughts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r372787342
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
+ * Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring kafka reset offset strategies in deltastreamer.
+ * Need shade Avro if implement custom payload, which is similar to hudi-hadoop-mr-bundle.
+ * Better delete support in DeltaStreamer, would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Support for AWS Database Migration Service(DMS) in DeltaStreamer
Review comment:
I was trying to understand what you mean by "refreshed".. both the blogs are up at cwiki site already right.. can we link them here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370450670
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
Review comment:
`scala` -> `Scala`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r372786581
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,46 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can use "repair overwrite-hoodie-props" to append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
+* CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+* DeltaStreamer CLI parameter for capturing table type is changed from --storage-type to --table-type. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+* Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring Kafka reset offset strategies with configuration(auto.offset.reset) in deltastreamer.
Review comment:
> I think the 0.8 and 0.10 refer to the underlying kafka version.. Might be good to mention once.. you can make the final call. Dont have strong opinions.
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1277:
[WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r371047596
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,46 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can use "repair overwrite-hoodie-props" to append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
+* CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+* DeltaStreamer CLI parameter for capturing table type is changed from --storage-type to --table-type. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+* Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring Kafka reset offset strategies with configuration(auto.offset.reset) in deltastreamer.
Review comment:
> May be also call out the Kafka version change above at the top?
I found we just change the artifactId from spark-streaming-kafka-0-8_2.11/2.12 to spark-streaming-kafka-0-10_2.11/2.12, the version is not changed. need to call out still ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370981143
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
+ * Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring kafka reset offset strategies in deltastreamer.
+ * Need shade Avro if implement custom payload, which is similar to hudi-hadoop-mr-bundle.
+ * Better delete support in DeltaStreamer, would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Support for AWS Database Migration Service(DMS) in DeltaStreamer
Review comment:
> meaning? moving them to the site? if so, is that work scoped?
sorry, I would not fully understand the meaning, which site to move them to? remove DMS support in release notes? @vinothchandar
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370483510
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
Review comment:
Reword:
"are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12."
change to
"are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}. Note that scala_version here is one of (2.11, 2.12)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370483039
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
Review comment:
Minor : Formatting changes. Can you change the layout so the version changes is more visible.
Like
* Dependency Version Upgrades
** Spark from 2.1.0 to 2.4.4
** Avro from .....
** parquet from .... to ...
* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370981064
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,46 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can use "repair overwrite-hoodie-props" to append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
+* CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+* DeltaStreamer CLI parameter for capturing table type is changed from --storage-type to --table-type. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+* Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring Kafka reset offset strategies with configuration(auto.offset.reset) in deltastreamer.
+* When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro_2.11:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
Review comment:
lets use (backtick)--packages org.apache.spark:spark-avro_2.11:2.4.4(backtick) , to get it rendered nicely like `--packages org.apache.spark:spark-avro_2.11:2.4.4` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1277: [WIP][HUDI-543]
release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#issuecomment-578061664
Thanks for your review @bvaradar @yanghua . Updated the PR to address your comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1277: [WIP][HUDI-543]
release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#issuecomment-578463652
@bvaradar Thanks for your review and suggestion, just updated the PR to address the new comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1277:
[WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370375801
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4 supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
Review comment:
> we don't support Spark 2.4, only 2.4+ - is that what this means?
Yes, fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf merged pull request #1277: [HUDI-543]
release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf merged pull request #1277: [HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370486513
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
Review comment:
We need to describe what needs to change in configuration. You need to list down what the user will have to change when they use the following APIs for ingesting
You should add a statement saying that : If you are using overridden key generator classes (configuration ("hoodie.datasource.write.keygenerator.class")) that comes with hudi package, please make change the fully qualified class name is changed accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370450262
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
+ * Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring kafka reset offset strategies in deltastreamer.
Review comment:
The second `kafka` -> `Kafka`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370487125
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
Review comment:
Can you move this to second point ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370487026
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
Review comment:
Reword:
DeltaStreamer CLI parameter for capturing table type is changed from --storage-type to --table-type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1277:
[WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r372991116
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,47 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+ * Upgrade from Kafka 0.8.2.1 to Kafka 2.0.0 as a result of updating spark-streaming-kafka artifact from 0.8_2.11/2.12 to 0.10_2.11/2.12.
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](https://hudi.apache.org/docs/deployment.html#upgrading), before enabling this feature for existing hudi tables.
Review comment:
I have updated the section link here(https://hudi.apache.org/docs/deployment.html#upgrading) according to the new website, it should work once the website is updated. cc @vinothchandar
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370483722
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
+ * Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring kafka reset offset strategies in deltastreamer.
Review comment:
Can you point to the configuration name also for this ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1277: [WIP][HUDI-543]
release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#issuecomment-580288717
All the comments addressed and is ready to be merged. @vinothchandar @bvaradar @yanghua Would take another look to get a final pass. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370947587
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,47 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
+* CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+* DeltaStreamer CLI parameter for capturing table type is changed from --storage-type to --table-type. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+* Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring Kafka reset offset strategies with configuration(auto.offset.reset) in deltastreamer.
+* When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro_2.11:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+* Key generator moved to separate package under org.apache.hudi.keygen. If you are using overridden key generator classes (configuration ("hoodie.datasource.write.keygenerator.class")) that comes with hudi package, please make change the fully qualified class name is changed accordingly.
+* Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+* With 0.5.1, hudi-hadoop-mr-bundle which is used by query engines such as presto and hive includes shaded avro package to support hudi real time queries through these engines. Hudi supports pluggable logic for merging of records. Users provide their own implementation of [HoodieRecordPayload](https://github.com/apache/incubator-hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java).
+If you are using this feature, you need to relocate the avro dependencies in your custom record payload class to be consistent with internal hudi shading. You need to add the following relocation when shading the package containing the record payload implementation.
+
+ ```xml
+<relocation>
+ <pattern>org.apache.avro.</pattern>
+ <shadedPattern>org.apache.hudi.org.apache.avro.</shadedPattern>
+</relocation>
+ ```
+
+ * Better delete support in DeltaStreamer would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Support for AWS Database Migration Service(DMS) in DeltaStreamer
+ * Support for DynamicBloomFilter.
+ * Support option to overwrite payload implementation in hoodie.properties file.
Review comment:
This is already covered in the point above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370980759
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,46 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
Review comment:
nit : existing hudi tables
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf edited a comment on issue #1277: [HUDI-543]
release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf edited a comment on issue #1277: [HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#issuecomment-580288717
All the comments addressed and is ready to be merged. @vinothchandar @bvaradar @yanghua @smarthi Would take another look to get a final pass. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370490915
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
+ * Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring kafka reset offset strategies in deltastreamer.
+ * Need shade Avro if implement custom payload, which is similar to hudi-hadoop-mr-bundle.
+ * Better delete support in DeltaStreamer, would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Support for AWS Database Migration Service(DMS) in DeltaStreamer
Review comment:
You need to link the blogs that were added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on a change in pull request #1277:
[WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370547632
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
+ * Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring kafka reset offset strategies in deltastreamer.
+ * Need shade Avro if implement custom payload, which is similar to hudi-hadoop-mr-bundle.
+ * Better delete support in DeltaStreamer, would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Support for AWS Database Migration Service(DMS) in DeltaStreamer
Review comment:
> You need to link the blogs that were added.
Will update after the blogs refreshed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r372787342
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
+ * Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring kafka reset offset strategies in deltastreamer.
+ * Need shade Avro if implement custom payload, which is similar to hudi-hadoop-mr-bundle.
+ * Better delete support in DeltaStreamer, would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Support for AWS Database Migration Service(DMS) in DeltaStreamer
Review comment:
> I was trying to understand what you mean by "refreshed".. both the blogs are up at cwiki site already right.. can we link them here?
Done @vinothchandar
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370947406
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,47 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
+* CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+* DeltaStreamer CLI parameter for capturing table type is changed from --storage-type to --table-type. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+* Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring Kafka reset offset strategies with configuration(auto.offset.reset) in deltastreamer.
+* When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro_2.11:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+* Key generator moved to separate package under org.apache.hudi.keygen. If you are using overridden key generator classes (configuration ("hoodie.datasource.write.keygenerator.class")) that comes with hudi package, please make change the fully qualified class name is changed accordingly.
Review comment:
Reword:
"please make change the fully qualified class name is changed accordingly."
to
"please ensure the fully qualified class name is changed accordingly."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370490658
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,31 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4 and upgrade from avro 1.7.7 to avro 1.8.2 accordingly. Spark 2.4+ supports drop and please use Spark 2.4+ for Hudi 0.5.1+ above.
+ * When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Key generator moved to separate package under org.apache.hudi.keygen.
+ * CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+ * Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+ * DeltaStreamer configs changed including from `storage-type` to `table-type`. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+ * Hudi now supports both scala 2.11 and scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with scala 2.12. Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle
+ are changed according hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}, scala_version here includes 2.11 and 2.12.
+ * Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring kafka reset offset strategies in deltastreamer.
+ * Need shade Avro if implement custom payload, which is similar to hudi-hadoop-mr-bundle.
Review comment:
We need to elaborate on this. I would move this to last point as I am not expecting many users to be impacted by this.
Please reword to:
"With 0.5.1, hudi-hadoop-mr-bundle which is used by query engines such as presto and hive includes shaded avro package to support hudi real time queries through these engines. Hudi supports pluggable logic for merging of records. Users provide their own implementation of [HoodieRecordPayload](https://github.com/apache/incubator-hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java). If you are using this feature, you need to relocate the avro dependencies in your custom record payload class to be consistent with internal hudi shading. You need to add the following relocation when shading the package containing the record payload implementation
```
<relocation>
<pattern>org.apache.avro.</pattern>
<shadedPattern>org.apache.hudi.org.apache.avro.</shadedPattern>
</relocation>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] leesf commented on issue #1277: [HUDI-543] release
notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
leesf commented on issue #1277: [HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#issuecomment-580617430
All comments addressed and merge this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370947637
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,47 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
+* CLI supports `repair overwrite-hoodie-props` to overwrite the table's hoodie.properties with specified file.
+* DeltaStreamer CLI parameter for capturing table type is changed from --storage-type to --table-type. Refer to [wiki](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture) with more latest terminologies.
+* Configuration Value change for Kafka Reset Offset Strategies. Enum values are changed from LARGEST to LATEST, SMALLEST to EARLIEST for configuring Kafka reset offset strategies with configuration(auto.offset.reset) in deltastreamer.
+* When using spark-shell to give a quick peek at Hudi, please provide --packages org.apache.spark:spark-avro_2.11:2.4.4, more details would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+* Key generator moved to separate package under org.apache.hudi.keygen. If you are using overridden key generator classes (configuration ("hoodie.datasource.write.keygenerator.class")) that comes with hudi package, please make change the fully qualified class name is changed accordingly.
+* Hive Sync tool will register RO tables for MOR with a _ro suffix, so query with _ro suffix. You would use `--skip-ro-suffix` in sync config to control suffix.
+* With 0.5.1, hudi-hadoop-mr-bundle which is used by query engines such as presto and hive includes shaded avro package to support hudi real time queries through these engines. Hudi supports pluggable logic for merging of records. Users provide their own implementation of [HoodieRecordPayload](https://github.com/apache/incubator-hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java).
+If you are using this feature, you need to relocate the avro dependencies in your custom record payload class to be consistent with internal hudi shading. You need to add the following relocation when shading the package containing the record payload implementation.
+
+ ```xml
+<relocation>
+ <pattern>org.apache.avro.</pattern>
+ <shadedPattern>org.apache.hudi.org.apache.avro.</shadedPattern>
+</relocation>
+ ```
+
+ * Better delete support in DeltaStreamer would refer to [latest quickstart docs](https://hudi.apache.org/docs/quick-start-guide.html)
+ * Support for AWS Database Migration Service(DMS) in DeltaStreamer
+ * Support for DynamicBloomFilter.
Review comment:
We need to describe that this is turned off by default and how to turn this on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1277: [WIP][HUDI-543] release notes for 0.5.1
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1277: [WIP][HUDI-543] release notes for 0.5.1
URL: https://github.com/apache/incubator-hudi/pull/1277#discussion_r370947534
##########
File path: docs/_pages/releases.md
##########
@@ -6,6 +6,47 @@ toc: true
last_modified_at: 2019-12-30T15:59:57-04:00
---
+## [Release 0.5.1-incubating]
+
+### Download Information
+ * Source Release : [Apache Hudi(incubating) 0.5.1-incubating Source Release](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz) ([asc](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.asc), [sha512](https://www.apache.org/dist/incubator/hudi/0.5.1-incubating/hudi-0.5.1-incubating.src.tgz.sha512))
+ * Apache Hudi (incubating) jars corresponding to this release is available [here](https://repository.apache.org/#nexus-search;quick~hudi)
+
+### Release Highlights
+* Dependency Version Upgrades
+ * Upgrade from Spark 2.1.0 to Spark 2.4.4
+ * Upgrade from Avro 1.7.7 to Avro 1.8.2
+ * Upgrade from Parquet 1.8.1 to Parquet 1.10.1
+* **IMPORTANT** This version requires your runtime spark version to be upgraded to 2.4+.
+* Hudi now supports both Scala 2.11 and Scala 2.12, please refer to [Build with Scala 2.12](https://github.com/apache/incubator-hudi#build-with-scala-212) to build with Scala 2.12.
+Also, the packages hudi-spark, hudi-utilities, hudi-spark-bundle and hudi-utilities-bundle are changed correspondingly to hudi-spark_{scala_version}, hudi-spark_{scala_version}, hudi-utilities_{scala_version}, hudi-spark-bundle_{scala_version} and hudi-utilities-bundle_{scala_version}.
+Note that scala_version here is one of (2.11, 2.12).
+* With 0.5.1, we added functionality to stop using renames for Hudi timeline metadata operations. This feature is automatically enabled for newly created Hudi tables. For existing tables, this feature is turned off by default. Please read this [section](deployment_link), before enabling this feature for existing hudi table.
+To enable the new hudi timeline layout which avoids renames, use the write config "hoodie.timeline.layout.version=1". Alternatively, you can append the line "hoodie.timeline.layout.version=1" to hoodie.properties. Note that in any case, upgrade hudi readers (query engines) first with 0.5.1-incubating release before upgrading writer.
Review comment:
Alternatively, you can append the line "hoodie.timeline.layout.version=1" to hoodie.properties
needs to change to
Alternatively, you can use "repair overwrite-hoodie-props" to append the line "hoodie.timeline.layout.version=1" to hoodie.properties
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services