You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/07/05 01:55:40 UTC
[GitHub] [flink-table-store] SteNicholas opened a new pull request, #195: [FLINK-26901] Introduce config options document for table store
SteNicholas opened a new pull request, #195:
URL: https://github.com/apache/flink-table-store/pull/195
Flink Table Store provides the configuration of LogStore and FileStore, which config options can be documented. It's necessary to introduce config options document for table store.
**The brief chang log**
- Introduces the `flink-table-store-docs` module to generate the configuration document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #195: [FLINK-26901] Introduce config options document for table store
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on code in PR #195:
URL: https://github.com/apache/flink-table-store/pull/195#discussion_r913367319
##########
docs/content/docs/deployment/configuration.md:
##########
@@ -0,0 +1,49 @@
+---
+title: "Configuration"
+weight: 1
+type: docs
+aliases:
+- /deployment/configuration.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Configuration
+
+Flink Table Store provides the configuration of LogStore and FileStore for hybrid storage to build dynamic tables.
Review Comment:
I think we can remove this line
##########
docs/content/docs/deployment/configuration.md:
##########
@@ -0,0 +1,49 @@
+---
+title: "Configuration"
+weight: 1
+type: docs
+aliases:
+- /deployment/configuration.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Configuration
+
+Flink Table Store provides the configuration of LogStore and FileStore for hybrid storage to build dynamic tables.
+You can specify the below config options in ‘with’ for the Managed Table to configure the LogStore and FileStore.
+
+# Common Setup Options
+
+**Table Store Factory**
Review Comment:
We will introduce 4 options:
- `CoreOptions`: Core options for table store.
- `CatalogOptions`: Options for table store catalog.
- `FlinkConnectorOptions`: Flink connector options for table store.
- `KafkaLogOptions`: Kafka log options provided after configuring `log.system` for `kafka`.
##########
docs/content/docs/deployment/_index.md:
##########
@@ -0,0 +1,27 @@
+---
+title: Deployment
Review Comment:
Just `Configuration`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on pull request #195: [FLINK-26901] Introduce config options document for table store
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on PR #195:
URL: https://github.com/apache/flink-table-store/pull/195#issuecomment-1174583780
Thanks @SteNicholas for the contribution, we can finish some configuration refactor first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #195: [FLINK-26901] Introduce config options document for table store
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on code in PR #195:
URL: https://github.com/apache/flink-table-store/pull/195#discussion_r913466400
##########
docs/content/docs/development/configuration.md:
##########
@@ -0,0 +1,53 @@
+---
+title: "Configuration"
+weight: 6
+type: docs
+aliases:
+- /development/configuration.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Configuration
+
+# Configuration Options
Review Comment:
Remove this line too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi closed pull request #195: [FLINK-26901] Introduce config options document for table store
Posted by GitBox <gi...@apache.org>.
JingsongLi closed pull request #195: [FLINK-26901] Introduce config options document for table store
URL: https://github.com/apache/flink-table-store/pull/195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #195: [FLINK-26901] Introduce config options document for table store
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on code in PR #195:
URL: https://github.com/apache/flink-table-store/pull/195#discussion_r913468538
##########
docs/layouts/shortcodes/generated/core_configuration.html:
##########
@@ -0,0 +1,204 @@
+<table class="configuration table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 10%">Type</th>
+ <th class="text-left" style="width: 55%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>bucket</h5></td>
+ <td style="word-wrap: break-word;">1</td>
+ <td>Integer</td>
+ <td>Bucket number for file store.</td>
+ </tr>
+ <tr>
+ <td><h5>changelog-file</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>Whether to double write to a changelog file when flushing memory table. This changelog file keeps the order of data input and the details of data changes, it can be read directly during stream reads.</td>
+ </tr>
+ <tr>
+ <td><h5>commit.force-compact</h5></td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>Whether to force a compaction before commit.</td>
+ </tr>
+ <tr>
+ <td><h5>compaction.max-size-amplification-percent</h5></td>
+ <td style="word-wrap: break-word;">200</td>
+ <td>Integer</td>
+ <td>The size amplification is defined as the amount (in percentage) of additional storage needed to store a single byte of data in the merge tree.</td>
+ </tr>
+ <tr>
+ <td><h5>compaction.size-ratio</h5></td>
+ <td style="word-wrap: break-word;">1</td>
+ <td>Integer</td>
+ <td>Percentage flexibility while comparing sorted run size. If the candidate sorted run(s) size is 1% smaller than the next sorted run's size, then include next sorted run into this candidate set.</td>
+ </tr>
+ <tr>
+ <td><h5>continuous.discovery-interval</h5></td>
+ <td style="word-wrap: break-word;">1 s</td>
+ <td>Duration</td>
+ <td>The discovery interval of continuous reading.</td>
+ </tr>
+ <tr>
+ <td><h5>file.format</h5></td>
+ <td style="word-wrap: break-word;">"orc"</td>
+ <td>String</td>
+ <td>Specify the message format of data files.</td>
+ </tr>
+ <tr>
+ <td><h5>log.changelog-mode</h5></td>
+ <td style="word-wrap: break-word;">auto</td>
+ <td><p>Enum</p></td>
+ <td>Specify the log changelog mode for table.<br /><ul><li>"auto": Upsert for table with primary key, all for table without primary key..</li><li>"all": The log system stores all changes including UPDATE_BEFORE.</li><li>"upsert": The log system does not store the UPDATE_BEFORE changes, the log consumed job will automatically add the normalized node, relying on the state to generate the required update_before.</li></ul><br /><br />Possible values:<ul><li>"auto": Upsert for table with primary key, all for table without primary key..</li><li>"all": The log system stores all changes including UPDATE_BEFORE.</li><li>"upsert": The log system does not store the UPDATE_BEFORE changes, the log consumed job will automatically add the normalized node, relying on the state to generate the required update_before.</li></ul></td>
+ </tr>
+ <tr>
+ <td><h5>log.consistency</h5></td>
+ <td style="word-wrap: break-word;">transactional</td>
+ <td><p>Enum</p></td>
+ <td>Specify the log consistency mode for table.<br /><ul><li>"transactional": Only the data after the checkpoint can be seen by readers, the latency depends on checkpoint interval.</li><li>"eventual": Immediate data visibility, you may see some intermediate states, but eventually the right results will be produced, only works for table with primary key.</li></ul><br /><br />Possible values:<ul><li>"transactional": Only the data after the checkpoint can be seen by readers, the latency depends on checkpoint interval.</li><li>"eventual": Immediate data visibility, you may see some intermediate states, but eventually the right results will be produced, only works for table with primary key.</li></ul></td>
+ </tr>
+ <tr>
+ <td><h5>log.format</h5></td>
+ <td style="word-wrap: break-word;">"debezium-json"</td>
+ <td>String</td>
+ <td>Specify the message format of log system.</td>
+ </tr>
+ <tr>
+ <td><h5>log.key.format</h5></td>
+ <td style="word-wrap: break-word;">"json"</td>
+ <td>String</td>
+ <td>Specify the key message format of log system with primary key.</td>
+ </tr>
+ <tr>
+ <td><h5>log.retention</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>Duration</td>
+ <td>It means how long changes log will be kept. The default value is from the log system cluster.</td>
+ </tr>
+ <tr>
+ <td><h5>log.scan</h5></td>
+ <td style="word-wrap: break-word;">full</td>
+ <td><p>Enum</p></td>
+ <td>Specify the startup mode for log consumer.<br /><ul><li>"full": Perform a snapshot on the table upon first startup, and continue to read the latest changes.</li></ul><ul><li>"latest": Start from the latest.</li></ul><ul><li>"from-timestamp": Start from user-supplied timestamp.</li></ul><br /><br />Possible values:<ul><li>"full": Perform a snapshot on the table upon first startup, and continue to read the latest changes.</li><li>"latest": Start from the latest.</li><li>"from-timestamp": Start from user-supplied timestamp.</li></ul></td>
+ </tr>
+ <tr>
+ <td><h5>log.scan.timestamp-millis</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>Long</td>
+ <td>Optional timestamp used in case of "from-timestamp" scan mode</td>
+ </tr>
+ <tr>
+ <td><h5>manifest.format</h5></td>
+ <td style="word-wrap: break-word;">"avro"</td>
+ <td>String</td>
+ <td>Specify the message format of manifest files.</td>
+ </tr>
+ <tr>
+ <td><h5>manifest.merge-min-count</h5></td>
+ <td style="word-wrap: break-word;">30</td>
+ <td>Integer</td>
+ <td>To avoid frequent manifest merges, this parameter specifies the minimum number of ManifestFileMeta to merge.</td>
+ </tr>
+ <tr>
+ <td><h5>manifest.target-file-size</h5></td>
+ <td style="word-wrap: break-word;">8 mb</td>
+ <td>MemorySize</td>
+ <td>Suggested file size of a manifest file.</td>
+ </tr>
+ <tr>
+ <td><h5>merge-engine</h5></td>
+ <td style="word-wrap: break-word;">deduplicate</td>
+ <td><p>Enum</p></td>
+ <td>Specify the merge engine for table with primary key.<br /><ul><li>"deduplicate": De-duplicate and keep the last row.</li><li>"partial-update": Partial update non-null fields.</li></ul><br /><br />Possible values:<ul><li>"deduplicate": De-duplicate and keep the last row.</li><li>"partial-update": Partial update non-null fields.</li></ul></td>
+ </tr>
+ <tr>
+ <td><h5>num-levels</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>Integer</td>
+ <td>Total level number, for example, there are 3 levels, including 0,1,2 levels.</td>
+ </tr>
+ <tr>
+ <td><h5>num-sorted-run.compaction-trigger</h5></td>
+ <td style="word-wrap: break-word;">5</td>
+ <td>Integer</td>
+ <td>The sorted run number to trigger compaction. Includes level0 files (one file one sorted run) and high-level runs (one level one sorted run).</td>
+ </tr>
+ <tr>
+ <td><h5>num-sorted-run.stop-trigger</h5></td>
+ <td style="word-wrap: break-word;">10</td>
+ <td>Integer</td>
+ <td>The number of sorted-runs that trigger the stopping of writes.</td>
+ </tr>
+ <tr>
+ <td><h5>page-size</h5></td>
+ <td style="word-wrap: break-word;">64 kb</td>
+ <td>MemorySize</td>
+ <td>Memory page size.</td>
+ </tr>
+ <tr>
+ <td><h5>partition.default-name</h5></td>
+ <td style="word-wrap: break-word;">"__DEFAULT_PARTITION__"</td>
+ <td>String</td>
+ <td>The default partition name in case the dynamic partition column value is null/empty string.</td>
+ </tr>
+ <tr>
+ <td><h5>path</h5></td>
Review Comment:
Contains `org.apache.flink.annotation.Internal` option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org