You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2023/11/14 12:59:10 UTC
[PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
codope opened a new pull request, #10087:
URL: https://github.com/apache/hudi/pull/10087
### Change Logs
- Update `CREATE INDEX` syntax and show example fo creating functional index in SQL DDL page.
- Update about non-blocking concurrency control in concurrency mode page.
- Update flink quickstart page with an example of new concurrency mode.
- Update timeline page with details about LSM timeline.
### Impact
Add docs for new features in 1.0.0-beta1.
### Risk level (write none, low medium or high below)
none
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393231248
##########
website/docs/sql_ddl.md:
##########
@@ -178,6 +178,145 @@ TBLPROPERTIES (
AS SELECT * FROM parquet_table;
```
+### Create Index
+
+Hudi supports creating and dropping indexes, including functional indexes, on a table.
+
+:::note
+Creating indexes through SQL is in preview in version 1.0.0-beta only. It will be generally available in version 1.0.0.
+Please report any issues you find either via [GitHub issues](https://github.com/apache/hudi/issues) or creating a [JIRA](https://issues.apache.org/jira/projects/HUDI/issues).
+:::
+
+```sql
+-- Create Index
+CREATE INDEX [IF NOT EXISTS] index_name ON [TABLE] table_name
+[USING index_type]
+(column_name1 [OPTIONS(key1=value1, key2=value2, ...)], column_name2 [OPTIONS(key1=value1, key2=value2, ...)], ...)
+[OPTIONS (key1=value1, key2=value2, ...)]
+
+-- Drop Index
+DROP INDEX [IF EXISTS] index_name ON [TABLE] table_name
+```
+
+- `index_name` is the name of the index to be created or dropped.
+- `table_name` is the name of the table on which the index is created or dropped.
+- `index_type` is the type of the index to be created. Currently, only `files`, `column_stats` and `bloom_filters` is supported.
+- `column_name` is the name of the column on which the index is created.
+- Both index and column on which the index is created can be qualified with some options in the form of key-value pairs.
+ We will see this with an example of functional index below.
+
+#### Create Functional Index
+
+A [functional index](https://github.com/apache/hudi/blob/00ece7bce0a4a8d0019721a28049723821e01842/rfc/rfc-63/rfc-63.md)
+is an index on a function of a column. It is a new addition to Hudi's [multi-modal indexing](https://hudi.apache.org/blog/2022/05/17/Introducing-Multi-Modal-Index-for-the-Lakehouse-in-Apache-Hudi)
+subsystem which provides faster access method and also absorb partitioning as part of the indexing system. Let us see
+some examples of creating a functional index.
+
+```sql
+-- Create a functional index on the column `ts` (unix epoch) of the table `hudi_table` using the function `from_unixtime` with the format `yyyy-MM-dd`
+CREATE INDEX IF NOT EXISTS ts_datestr ON hudi_table USING column_stats(ts) OPTIONS(func='from_unixtime', format='yyyy-MM-dd');
+-- Create a functional index on the column `ts` (timestamp in yyyy-MM-dd HH:mm:ss) of the table `hudi_table` using the function `hour`
+CREATE INDEX ts_hour ON hudi_table_func_index USING column_stats(ts) options(func='hour');
+```
+
+Few things to note:
+- The `func` option is required for creating functional index, and it should be a valid Spark SQL function. Currently,
+ only the functions that take a single column as input are supported. Some useful functions that are supported are listed below.
+ - `identity`
+ - `from_unixtime`
+ - `date_format`
+ - `to_date`
+ - `to_timestamp`
+ - `year`
+ - `month`
+ - `day`
+ - `hour`
+ - `lower`
+ - `upper`
+ - `substring`
+ - `regexp_extract`
+ - `regexp_replace`
+ - `concat`
+ - `length`
+- Please check the syntax for the above functions in
+ the [Spark SQL documentation](https://spark.apache.org/docs/latest/sql-ref-functions.html) and provide the options
+ accordingly. For example, the `format` option is required for `from_unixtime` function.
+- UDFs are not supported.
+
+<details>
+ <summary>Example of creating and using functional index</summary>
+
+```sql
+DROP TABLE IF EXISTS hudi_table_func_index;
Review Comment:
Ah okay got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393182248
##########
website/docs/sql_ddl.md:
##########
@@ -178,6 +178,145 @@ TBLPROPERTIES (
AS SELECT * FROM parquet_table;
```
+### Create Index
+
+Hudi supports creating and dropping indexes, including functional indexes, on a table.
+
+:::note
+Creating indexes through SQL is in preview in version 1.0.0-beta only. It will be generally available in version 1.0.0.
+Please report any issues you find either via [GitHub issues](https://github.com/apache/hudi/issues) or creating a [JIRA](https://issues.apache.org/jira/projects/HUDI/issues).
+:::
+
+```sql
+-- Create Index
+CREATE INDEX [IF NOT EXISTS] index_name ON [TABLE] table_name
+[USING index_type]
+(column_name1 [OPTIONS(key1=value1, key2=value2, ...)], column_name2 [OPTIONS(key1=value1, key2=value2, ...)], ...)
+[OPTIONS (key1=value1, key2=value2, ...)]
+
+-- Drop Index
+DROP INDEX [IF EXISTS] index_name ON [TABLE] table_name
+```
+
+- `index_name` is the name of the index to be created or dropped.
+- `table_name` is the name of the table on which the index is created or dropped.
+- `index_type` is the type of the index to be created. Currently, only `files`, `column_stats` and `bloom_filters` is supported.
+- `column_name` is the name of the column on which the index is created.
+- Both index and column on which the index is created can be qualified with some options in the form of key-value pairs.
+ We will see this with an example of functional index below.
+
+#### Create Functional Index
+
+A [functional index](https://github.com/apache/hudi/blob/00ece7bce0a4a8d0019721a28049723821e01842/rfc/rfc-63/rfc-63.md)
+is an index on a function of a column. It is a new addition to Hudi's [multi-modal indexing](https://hudi.apache.org/blog/2022/05/17/Introducing-Multi-Modal-Index-for-the-Lakehouse-in-Apache-Hudi)
+subsystem which provides faster access method and also absorb partitioning as part of the indexing system. Let us see
+some examples of creating a functional index.
+
+```sql
+-- Create a functional index on the column `ts` (unix epoch) of the table `hudi_table` using the function `from_unixtime` with the format `yyyy-MM-dd`
+CREATE INDEX IF NOT EXISTS ts_datestr ON hudi_table USING column_stats(ts) OPTIONS(func='from_unixtime', format='yyyy-MM-dd');
+-- Create a functional index on the column `ts` (timestamp in yyyy-MM-dd HH:mm:ss) of the table `hudi_table` using the function `hour`
+CREATE INDEX ts_hour ON hudi_table_func_index USING column_stats(ts) options(func='hour');
+```
+
+Few things to note:
+- The `func` option is required for creating functional index, and it should be a valid Spark SQL function. Currently,
+ only the functions that take a single column as input are supported. Some useful functions that are supported are listed below.
+ - `identity`
+ - `from_unixtime`
+ - `date_format`
+ - `to_date`
+ - `to_timestamp`
+ - `year`
+ - `month`
+ - `day`
+ - `hour`
+ - `lower`
+ - `upper`
+ - `substring`
+ - `regexp_extract`
+ - `regexp_replace`
+ - `concat`
+ - `length`
+- Please check the syntax for the above functions in
+ the [Spark SQL documentation](https://spark.apache.org/docs/latest/sql-ref-functions.html) and provide the options
+ accordingly. For example, the `format` option is required for `from_unixtime` function.
+- UDFs are not supported.
+
+<details>
+ <summary>Example of creating and using functional index</summary>
+
+```sql
+DROP TABLE IF EXISTS hudi_table_func_index;
Review Comment:
`DROP INDEX` ? and name of the index is `ts_hour` instead ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393619389
##########
website/docs/sql_ddl.md:
##########
@@ -237,8 +375,10 @@ Users can set table properties while creating a table. The important table prope
:::
#### Passing Lock Providers for Concurrent Writers
-Hudi requires a lock provider to support concurrent writers or asynchronous table services. Users can pass these table
-properties into *TBLPROPERTIES* as well. Below is an example for a Zookeeper based configuration.
+
+Hudi requires a lock provider to support concurrent writers or asynchronous table services when using OCC concurrency
+mode. Users can pass these table properties into *TBLPROPERTIES* as well. Below is an example for a Zookeeper based
Review Comment:
When using OCC and NB-CC concurrency modes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #10087:
URL: https://github.com/apache/hudi/pull/10087
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393139965
##########
website/docs/sql_ddl.md:
##########
@@ -178,6 +178,145 @@ TBLPROPERTIES (
AS SELECT * FROM parquet_table;
```
+### Create Index
Review Comment:
Tag the Header with `(Preview)` or `(Experimental)` suffix, so it is more visible to users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393618310
##########
website/docs/concurrency_control.md:
##########
@@ -61,6 +68,18 @@ With multiple writers using OCC, these are the write guarantees to expect:
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order. If there are inflight commits
(due to multi-writing), incremental queries will not expose the completed commits following the inflight commits.
+#### Multi Writer Guarantees with Non-Blocking Concurrency Control Mode (Experimental)
+
+`NON_BLOCKING_CONCURRENCY_CONTROL`, offers the same set of guarantees as mentioned in the case of OCC but without
+explicit locks for serializing the writes. Lock is only needed for writing the commit metadata to the Hudi timeline. The
+completion time for the commits reflects the serialization order and file slicing is done based on completion time.
+Multiple writers can operate on the table with non-blocking conflict resolution. The writers can write into the same
+file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
+currently available for preview in version 1.0.0-beta only with following caveats:
+
+- It is only supported for Flink writers currently.
Review Comment:
Spark writers also support it right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393136432
##########
website/docs/flink-quick-start-guide.md:
##########
@@ -401,6 +401,82 @@ select * from hudi_table/*+ OPTIONS('read.streaming.enabled'='true')*/;
This will give all changes that happened after the `read.start-commit` commit. The unique thing about this
feature is that it now lets you author streaming pipelines on streaming or batch data source.
+## Non-Blocking Concurrency Control
Review Comment:
Refer to the comment above on moving this section to other Flink related docs. We can call this out in the https://hudi.apache.org/docs/next/flink-quick-start-guide#where-to-go-from-here section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393236163
##########
website/docs/timeline.md:
##########
@@ -54,6 +54,35 @@ after certain thresholds the archival kicks in to move older timeline events to
timeline is never contacted for regular operations of the table and is merely used for book-keeping and debugging purposes.
Any instants seen under “.hoodie” directory refers to active timeline and those archived goes into “.hoodie/archived” folder.
+#### LSM Timeline
+
+As mentioned above, active timeline has limited log history to be fast, while archived timeline is expensive to access
Review Comment:
@codope Can we clarify in a sentence or two on how users can/should transition to this new timeline. Should there be any action items from the users to use this new timeline. That part is not clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393131984
##########
website/docs/concurrency_control.md:
##########
@@ -61,6 +68,18 @@ With multiple writers using OCC, these are the write guarantees to expect:
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order. If there are inflight commits
(due to multi-writing), incremental queries will not expose the completed commits following the inflight commits.
+#### Multi Writer Guarantees with Non-Blocking Concurrency Control Mode
+
+`NON_BLOCKING_CONCURRENCY_CONTROL`, offers the same set of guarantees as mentioned in the case of OCC but without
+explicit locks for serializing the writes. Lock is only needed for writing the commit metadata to the Hudi timeline. The
+completion time for the commits reflects the serialization order and file slicing is done based on completion time.
+Multiple writers can operate on the table with non-blocking conflict resolution. The writers can write into the same
+file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
+currently available for preview in version 1.0.0-beta only with following caveats:
+
+- It is only supported for Flink writers currently.
+- Conflict resolution is not supported yet between clustering and ingestion. It works for compaction and ingestion, and
+ we can see an example of that in the [Flink quickstart guide](/docs/next/flink-quick-start-guide#non-blocking-concurrency-control).
Review Comment:
Can we move the example of NBCC from quickstart to either https://hudi.apache.org/docs/hoodie_streaming_ingestion#flink-ingestion or to the https://hudi.apache.org/docs/writing_data#flink-sql-writer. Quickstart should be simple and focus only on CRUD operations for easy start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393221202
##########
website/docs/sql_ddl.md:
##########
@@ -178,6 +178,145 @@ TBLPROPERTIES (
AS SELECT * FROM parquet_table;
```
+### Create Index
+
+Hudi supports creating and dropping indexes, including functional indexes, on a table.
+
+:::note
+Creating indexes through SQL is in preview in version 1.0.0-beta only. It will be generally available in version 1.0.0.
+Please report any issues you find either via [GitHub issues](https://github.com/apache/hudi/issues) or creating a [JIRA](https://issues.apache.org/jira/projects/HUDI/issues).
+:::
+
+```sql
+-- Create Index
+CREATE INDEX [IF NOT EXISTS] index_name ON [TABLE] table_name
+[USING index_type]
+(column_name1 [OPTIONS(key1=value1, key2=value2, ...)], column_name2 [OPTIONS(key1=value1, key2=value2, ...)], ...)
+[OPTIONS (key1=value1, key2=value2, ...)]
+
+-- Drop Index
+DROP INDEX [IF EXISTS] index_name ON [TABLE] table_name
+```
+
+- `index_name` is the name of the index to be created or dropped.
+- `table_name` is the name of the table on which the index is created or dropped.
+- `index_type` is the type of the index to be created. Currently, only `files`, `column_stats` and `bloom_filters` is supported.
+- `column_name` is the name of the column on which the index is created.
+- Both index and column on which the index is created can be qualified with some options in the form of key-value pairs.
+ We will see this with an example of functional index below.
+
+#### Create Functional Index
+
+A [functional index](https://github.com/apache/hudi/blob/00ece7bce0a4a8d0019721a28049723821e01842/rfc/rfc-63/rfc-63.md)
+is an index on a function of a column. It is a new addition to Hudi's [multi-modal indexing](https://hudi.apache.org/blog/2022/05/17/Introducing-Multi-Modal-Index-for-the-Lakehouse-in-Apache-Hudi)
+subsystem which provides faster access method and also absorb partitioning as part of the indexing system. Let us see
+some examples of creating a functional index.
+
+```sql
+-- Create a functional index on the column `ts` (unix epoch) of the table `hudi_table` using the function `from_unixtime` with the format `yyyy-MM-dd`
+CREATE INDEX IF NOT EXISTS ts_datestr ON hudi_table USING column_stats(ts) OPTIONS(func='from_unixtime', format='yyyy-MM-dd');
+-- Create a functional index on the column `ts` (timestamp in yyyy-MM-dd HH:mm:ss) of the table `hudi_table` using the function `hour`
+CREATE INDEX ts_hour ON hudi_table_func_index USING column_stats(ts) options(func='hour');
+```
+
+Few things to note:
+- The `func` option is required for creating functional index, and it should be a valid Spark SQL function. Currently,
+ only the functions that take a single column as input are supported. Some useful functions that are supported are listed below.
+ - `identity`
+ - `from_unixtime`
+ - `date_format`
+ - `to_date`
+ - `to_timestamp`
+ - `year`
+ - `month`
+ - `day`
+ - `hour`
+ - `lower`
+ - `upper`
+ - `substring`
+ - `regexp_extract`
+ - `regexp_replace`
+ - `concat`
+ - `length`
+- Please check the syntax for the above functions in
+ the [Spark SQL documentation](https://spark.apache.org/docs/latest/sql-ref-functions.html) and provide the options
+ accordingly. For example, the `format` option is required for `from_unixtime` function.
+- UDFs are not supported.
+
+<details>
+ <summary>Example of creating and using functional index</summary>
+
+```sql
+DROP TABLE IF EXISTS hudi_table_func_index;
Review Comment:
Actually `hudi_table_func_index` is the name of the table and not the index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393181396
##########
website/docs/sql_ddl.md:
##########
@@ -178,6 +178,145 @@ TBLPROPERTIES (
AS SELECT * FROM parquet_table;
```
+### Create Index
+
+Hudi supports creating and dropping indexes, including functional indexes, on a table.
+
+:::note
+Creating indexes through SQL is in preview in version 1.0.0-beta only. It will be generally available in version 1.0.0.
+Please report any issues you find either via [GitHub issues](https://github.com/apache/hudi/issues) or creating a [JIRA](https://issues.apache.org/jira/projects/HUDI/issues).
+:::
+
+```sql
+-- Create Index
+CREATE INDEX [IF NOT EXISTS] index_name ON [TABLE] table_name
+[USING index_type]
+(column_name1 [OPTIONS(key1=value1, key2=value2, ...)], column_name2 [OPTIONS(key1=value1, key2=value2, ...)], ...)
+[OPTIONS (key1=value1, key2=value2, ...)]
+
+-- Drop Index
+DROP INDEX [IF EXISTS] index_name ON [TABLE] table_name
+```
+
+- `index_name` is the name of the index to be created or dropped.
+- `table_name` is the name of the table on which the index is created or dropped.
+- `index_type` is the type of the index to be created. Currently, only `files`, `column_stats` and `bloom_filters` is supported.
+- `column_name` is the name of the column on which the index is created.
+- Both index and column on which the index is created can be qualified with some options in the form of key-value pairs.
+ We will see this with an example of functional index below.
+
+#### Create Functional Index
+
+A [functional index](https://github.com/apache/hudi/blob/00ece7bce0a4a8d0019721a28049723821e01842/rfc/rfc-63/rfc-63.md)
+is an index on a function of a column. It is a new addition to Hudi's [multi-modal indexing](https://hudi.apache.org/blog/2022/05/17/Introducing-Multi-Modal-Index-for-the-Lakehouse-in-Apache-Hudi)
+subsystem which provides faster access method and also absorb partitioning as part of the indexing system. Let us see
+some examples of creating a functional index.
+
+```sql
+-- Create a functional index on the column `ts` (unix epoch) of the table `hudi_table` using the function `from_unixtime` with the format `yyyy-MM-dd`
+CREATE INDEX IF NOT EXISTS ts_datestr ON hudi_table USING column_stats(ts) OPTIONS(func='from_unixtime', format='yyyy-MM-dd');
+-- Create a functional index on the column `ts` (timestamp in yyyy-MM-dd HH:mm:ss) of the table `hudi_table` using the function `hour`
+CREATE INDEX ts_hour ON hudi_table_func_index USING column_stats(ts) options(func='hour');
Review Comment:
`ON hudi_table` instead of `ON hudi_table_func_index` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393221994
##########
website/docs/flink-quick-start-guide.md:
##########
@@ -401,6 +401,82 @@ select * from hudi_table/*+ OPTIONS('read.streaming.enabled'='true')*/;
This will give all changes that happened after the `read.start-commit` commit. The unique thing about this
feature is that it now lets you author streaming pipelines on streaming or batch data source.
+## Non-Blocking Concurrency Control
Review Comment:
done.. moved to flink-sql-writer section in writeing_data page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393526846
##########
website/docs/concurrency_control.md:
##########
@@ -61,6 +68,18 @@ With multiple writers using OCC, these are the write guarantees to expect:
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order. If there are inflight commits
(due to multi-writing), incremental queries will not expose the completed commits following the inflight commits.
+#### Multi Writer Guarantees with Non-Blocking Concurrency Control Mode
Review Comment:
good point.. i'll mark it in the heading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393643946
##########
website/docs/concurrency_control.md:
##########
@@ -61,6 +68,18 @@ With multiple writers using OCC, these are the write guarantees to expect:
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order. If there are inflight commits
(due to multi-writing), incremental queries will not expose the completed commits following the inflight commits.
+#### Multi Writer Guarantees with Non-Blocking Concurrency Control Mode (Experimental)
+
+`NON_BLOCKING_CONCURRENCY_CONTROL`, offers the same set of guarantees as mentioned in the case of OCC but without
+explicit locks for serializing the writes. Lock is only needed for writing the commit metadata to the Hudi timeline. The
+completion time for the commits reflects the serialization order and file slicing is done based on completion time.
+Multiple writers can operate on the table with non-blocking conflict resolution. The writers can write into the same
+file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
+currently available for preview in version 1.0.0-beta only with following caveats:
+
+- It is only supported for Flink writers currently.
Review Comment:
yeah just checked the commit is in 1.0.0-beta1. i thought it was merged last minute and in mast but not in beta. i'll update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393526570
##########
website/docs/timeline.md:
##########
@@ -54,6 +54,35 @@ after certain thresholds the archival kicks in to move older timeline events to
timeline is never contacted for regular operations of the table and is merely used for book-keeping and debugging purposes.
Any instants seen under “.hoodie” directory refers to active timeline and those archived goes into “.hoodie/archived” folder.
+#### LSM Timeline
+
+As mentioned above, active timeline has limited log history to be fast, while archived timeline is expensive to access
Review Comment:
For the beta release it is enabled by default. Users will be automatically transitioned to the new timeline and can't go back unless they downgrade the hudi version. I can mention about a config which controls the timeline compaction - `hoodie.timeline.compaction.batch.size`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393237084
##########
website/docs/concurrency_control.md:
##########
@@ -61,6 +68,18 @@ With multiple writers using OCC, these are the write guarantees to expect:
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order. If there are inflight commits
(due to multi-writing), incremental queries will not expose the completed commits following the inflight commits.
+#### Multi Writer Guarantees with Non-Blocking Concurrency Control Mode
Review Comment:
Should we mark this experimental as well int eh heading ?
##########
website/docs/concurrency_control.md:
##########
@@ -61,6 +68,18 @@ With multiple writers using OCC, these are the write guarantees to expect:
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order. If there are inflight commits
(due to multi-writing), incremental queries will not expose the completed commits following the inflight commits.
+#### Multi Writer Guarantees with Non-Blocking Concurrency Control Mode
Review Comment:
Should we mark this experimental as well in the heading ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7089] Add docs for new features in 1.0.0-beta [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10087:
URL: https://github.com/apache/hudi/pull/10087#discussion_r1393643946
##########
website/docs/concurrency_control.md:
##########
@@ -61,6 +68,18 @@ With multiple writers using OCC, these are the write guarantees to expect:
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order. If there are inflight commits
(due to multi-writing), incremental queries will not expose the completed commits following the inflight commits.
+#### Multi Writer Guarantees with Non-Blocking Concurrency Control Mode (Experimental)
+
+`NON_BLOCKING_CONCURRENCY_CONTROL`, offers the same set of guarantees as mentioned in the case of OCC but without
+explicit locks for serializing the writes. Lock is only needed for writing the commit metadata to the Hudi timeline. The
+completion time for the commits reflects the serialization order and file slicing is done based on completion time.
+Multiple writers can operate on the table with non-blocking conflict resolution. The writers can write into the same
+file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
+currently available for preview in version 1.0.0-beta only with following caveats:
+
+- It is only supported for Flink writers currently.
Review Comment:
yeah just checked the commit is in 1.0.0-beta1. i thought it was merged last minute and in master but not in beta. i'll update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org