You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ckonehouse (via GitHub)" <gi...@apache.org> on 2023/11/13 22:31:53 UTC
[PR] Added video resources to Concepts and Services Sections [hudi]
ckonehouse opened a new pull request, #10080:
URL: https://github.com/apache/hudi/pull/10080
### Change Logs
Added video resources section for pages in the Concepts and Services sections of the Hudi docs. Not all pages had associated videos so there may not be a video for every page.
### Impact
N/A
### Risk level (write none, low medium or high below)
Low
### Documentation Update
N/A
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] Added video resources to Concepts and Services Sections [hudi]
Posted by "ckonehouse (via GitHub)" <gi...@apache.org>.
ckonehouse commented on code in PR #10080:
URL: https://github.com/apache/hudi/pull/10080#discussion_r1399898447
##########
website/docs/transforms.md:
##########
@@ -64,3 +64,9 @@ Set the config as:
### Custom Transformer Implementation
You can write your own custom transformer by extending [this class](https://github.com/apache/hudi/tree/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/transform)
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Learn about Apache Hudi Transformers with Hands on Lab](https://www.youtube.com/watch?v=AprlZ8hGdJo)
+* [Apache Hudi with DBT Hands on Lab.Transform Raw Hudi tables with DBT and Glue Interactive Session](https://youtu.be/DH3LEaPG6ss)
Review Comment:
I think having linked resources to non-Hudi data transformations will be more beneficial than not having them. People will come here for any transformers even if the Hudi ones are not best fit so this could point them in the right direction (which is why I included the DBT video here).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] Added video resources to Concepts and Services Sections [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10080:
URL: https://github.com/apache/hudi/pull/10080#discussion_r1396818335
##########
website/docs/compaction.md:
##########
@@ -231,3 +231,5 @@ Offline compaction needs to submit the Flink task on the command line. The progr
| `--seq` | `LIFO` (Optional) | The order in which compaction tasks are executed. Executing from the latest compaction plan by default. `LIFO`: executing from the latest plan. `FIFO`: executing from the oldest plan. |
| `--service` | `false` (Optional) | Whether to start a monitoring service that checks and schedules new compaction task in configured interval. |
| `--min-compaction-interval-seconds` | `600(s)` (optional) | The checking interval for service mode, by default 10 minutes. |
+
Review Comment:
Avoid whitespace changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] Added video resources to Concepts and Services Sections [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10080:
URL: https://github.com/apache/hudi/pull/10080#discussion_r1396823127
##########
website/docs/key_generation.md:
##########
@@ -209,3 +209,8 @@ Partition path generated from key generator: "2020040118"
Input field value: "20200401" <br/>
Partition path generated from key generator: "04/01/2020"
+
+## Related Resources
Review Comment:
This video might be relevant for de-duping, does not address key generation. Lets remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] Added video resources to Concepts and Services Sections [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha merged PR #10080:
URL: https://github.com/apache/hudi/pull/10080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] Added video resources to Concepts and Services Sections [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10080:
URL: https://github.com/apache/hudi/pull/10080#discussion_r1396832031
##########
website/docs/timeline.md:
##########
@@ -77,3 +77,7 @@ Flink jobs using the SQL can be configured through the options in WITH clause. T
Refer [here](https://hudi.apache.org/docs/next/configurations#Flink-Options) for more details.
+## Related Resources
+<h3>Videos</h3>
+
+* [How to Query Hudi Tables in Incremental Fashion and Get only New data on AWS Glue | Hands on Lab](https://www.youtube.com/watch?v=c6DCJR91rBQx)
Review Comment:
May be a good fit for https://hudi.apache.org/docs/next/table_types page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] Added video resources to Concepts and Services Sections [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10080:
URL: https://github.com/apache/hudi/pull/10080#discussion_r1396826296
##########
website/docs/indexing.md:
##########
@@ -155,4 +155,11 @@ to finally check the incoming updates against all files. The `SIMPLE` Index will
`HBASE` index can be employed, if the operational overhead is acceptable and would provide much better lookup times for these tables.
When using a global index, users should also consider setting `hoodie.bloom.index.update.partition.path=true` or `hoodie.simple.index.update.partition.path=true` to deal with cases where the
-partition path value could change due to an update e.g users table partitioned by home city; user relocates to a different city. These tables are also excellent candidates for the Merge-On-Read table type.
\ No newline at end of file
+partition path value could change due to an update e.g users table partitioned by home city; user relocates to a different city. These tables are also excellent candidates for the Merge-On-Read table type.
+
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Global Bloom Index: Remove duplicates & guarantee uniquness - Hudi Labs](https://youtu.be/XlRvMFJ7g9c)
+* [Advantages of Metadata Indexing and Asynchronous Indexing in Hudi - Hands on Lab](https://www.youtube.com/watch?v=TSphQCsY4pY)
Review Comment:
This one is a good fit for the page - https://hudi.apache.org/docs/next/metadata_indexing
##########
website/docs/indexing.md:
##########
@@ -155,4 +155,11 @@ to finally check the incoming updates against all files. The `SIMPLE` Index will
`HBASE` index can be employed, if the operational overhead is acceptable and would provide much better lookup times for these tables.
When using a global index, users should also consider setting `hoodie.bloom.index.update.partition.path=true` or `hoodie.simple.index.update.partition.path=true` to deal with cases where the
-partition path value could change due to an update e.g users table partitioned by home city; user relocates to a different city. These tables are also excellent candidates for the Merge-On-Read table type.
\ No newline at end of file
+partition path value could change due to an update e.g users table partitioned by home city; user relocates to a different city. These tables are also excellent candidates for the Merge-On-Read table type.
+
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Global Bloom Index: Remove duplicates & guarantee uniquness - Hudi Labs](https://youtu.be/XlRvMFJ7g9c)
+* [Advantages of Metadata Indexing and Asynchronous Indexing in Hudi - Hands on Lab](https://www.youtube.com/watch?v=TSphQCsY4pY)
Review Comment:
Lets move it over to there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] Added video resources to Concepts and Services Sections [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10080:
URL: https://github.com/apache/hudi/pull/10080#discussion_r1396835800
##########
website/docs/transforms.md:
##########
@@ -64,3 +64,9 @@ Set the config as:
### Custom Transformer Implementation
You can write your own custom transformer by extending [this class](https://github.com/apache/hudi/tree/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/transform)
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Learn about Apache Hudi Transformers with Hands on Lab](https://www.youtube.com/watch?v=AprlZ8hGdJo)
+* [Apache Hudi with DBT Hands on Lab.Transform Raw Hudi tables with DBT and Glue Interactive Session](https://youtu.be/DH3LEaPG6ss)
Review Comment:
Not sure if Hudi transformers are used here actually. Seem like this is generally talking about how to transform data using Hudi DBT. Remove?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] Added video resources to Concepts and Services Sections [hudi]
Posted by "ckonehouse (via GitHub)" <gi...@apache.org>.
ckonehouse commented on code in PR #10080:
URL: https://github.com/apache/hudi/pull/10080#discussion_r1399898447
##########
website/docs/transforms.md:
##########
@@ -64,3 +64,9 @@ Set the config as:
### Custom Transformer Implementation
You can write your own custom transformer by extending [this class](https://github.com/apache/hudi/tree/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/transform)
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Learn about Apache Hudi Transformers with Hands on Lab](https://www.youtube.com/watch?v=AprlZ8hGdJo)
+* [Apache Hudi with DBT Hands on Lab.Transform Raw Hudi tables with DBT and Glue Interactive Session](https://youtu.be/DH3LEaPG6ss)
Review Comment:
I think having linked resources to non-Hudi data transformations will be more beneficial than not having them. People will come here for any transformers even if the Hudi ones are not best fit so this could point them in the right direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] Added video resources to Concepts and Services Sections [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10080:
URL: https://github.com/apache/hudi/pull/10080#discussion_r1396826530
##########
website/docs/indexing.md:
##########
@@ -155,4 +155,11 @@ to finally check the incoming updates against all files. The `SIMPLE` Index will
`HBASE` index can be employed, if the operational overhead is acceptable and would provide much better lookup times for these tables.
When using a global index, users should also consider setting `hoodie.bloom.index.update.partition.path=true` or `hoodie.simple.index.update.partition.path=true` to deal with cases where the
-partition path value could change due to an update e.g users table partitioned by home city; user relocates to a different city. These tables are also excellent candidates for the Merge-On-Read table type.
\ No newline at end of file
+partition path value could change due to an update e.g users table partitioned by home city; user relocates to a different city. These tables are also excellent candidates for the Merge-On-Read table type.
+
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Global Bloom Index: Remove duplicates & guarantee uniquness - Hudi Labs](https://youtu.be/XlRvMFJ7g9c)
+* [Advantages of Metadata Indexing and Asynchronous Indexing in Hudi - Hands on Lab](https://www.youtube.com/watch?v=TSphQCsY4pY)
Review Comment:
First one is okay. Lets move second over to there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] Added video resources to Concepts and Services Sections [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10080:
URL: https://github.com/apache/hudi/pull/10080#discussion_r1396818699
##########
website/docs/concurrency_control.md:
##########
@@ -279,4 +279,10 @@ hoodie.cleaner.policy.failed.writes=EAGER
## Caveats
If you are using the `WriteClient` API, please note that multiple writes to the table need to be initiated from 2 different instances of the write client.
-It is **NOT** recommended to use the same instance of the write client to perform multi writing.
\ No newline at end of file
+It is **NOT** recommended to use the same instance of the write client to perform multi writing.
+
+## Related Resources
+<h3>Videos</h3>
+
+* [Efficient Data Ingestion with Glue Concurrency and Hudi Data Lake](https://www.youtube.com/watch?v=G_Xd1haycqE)
Review Comment:
This might be just Glue concurrency. Not Hudi. We can remove the first one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org