[jira] [Commented] (IOTDB-3611) Support "Modify Time Series Encoding and Compression Type" interface/command

["Pengfei Liu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IOTDB-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17582760#comment-17582760 ] 

Pengfei Liu commented on IOTDB-3611:
------------------------------------

Phase III thinking modification code thinking:
1. Modify the SCHEMA command-remove the rewriting code
2. Modify the SCHEMA command-rewriting the log (here you only need to record the modification command record and the overwriting complete record)
3. Modify the SCHEMA command-increase the rewriting memory record
4. Modify the SCHEMA command-remove the original rewriting lock and use the new rewriting lock (only control the modification and finishing process, not the merger)
5. Restore code-Remove all
6. Restore code-increase the memory record through the log
7. Consolidation code (inside and outside) -On the original rewriting lock
8. Merge code (inside and outside) -writer's SCHEMA acquisition conditions: There is modification-> modification sequence-> query schema, otherwise use the file in the file
9. Sort the command-file scan-whether there are modified sequences in the file, and the sequence coding has a list of changes.
10. Sort out command-new rewriting lock control
11. Sort out command-virtual storage group rewriting tasks
12. Sort the command-Single file rewriting code modification
13. Sort the command-alignment optimization

> Support "Modify Time Series Encoding and Compression Type" interface/command
> ----------------------------------------------------------------------------
>
>                 Key: IOTDB-3611
>                 URL: https://issues.apache.org/jira/browse/IOTDB-3611
>             Project: Apache IoTDB
>          Issue Type: New Feature
>            Reporter: Pengfei Liu
>            Assignee: Pengfei Liu
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: master branch
>
>   Original Estimate: 960h
>  Remaining Estimate: 960h
>
> Want to do:
> Added "Modify Time Series Encoding and Compression Type" interface/command
> Application scenarios:
> In IoTDB application projects, reasonable setting of encoding and compression algorithms can effectively reduce disk space occupancy and reduce server costs in disguise. Modifying encoding and compression algorithms is an ideal method.
> Encoding and compression type data location:
> 1. Metadata - File
> 2. Metadata - Memory
> 3. tsfile file - unsealed
> 4. tsfile file - sealed
> 5. Wirter in memory
> 6. The reader in memory
> 7. Files and memory in merge
> 7. Others in memory to be added...
> Divided by data segment:
> Sealed historical tsfile file data - unsealed tsfile file data - incremental data
> plan selection:
> Option 1: The modification command only names incremental data, and provides historical data modification commands at the same time
> Advantages: fast execution of real-time commands, little impact on online operation
> Disadvantage: If the prompt is inappropriate, the user may mistakenly think that all data has been changed by executing the first command
>  
> Option 2: The modification command can modify incremental data and historical data at the same time
> Advantages: One command completes all operations, immediate results, and the reduction of disk space usage can be directly observed
> Disadvantages: The execution time of a single command is too long, which has a great impact on the online, and needs to be controlled
> List of possible tasks:
> 1. When querying and reading tsfile, use the encoding/compression type in the chunk header (to be investigated)
> 2. Modify metadata, including memory and files, so that incremental data directly uses the latest encoding/compression type
> 3. Modify the related Decoder Encoder Compressor in memory
> 4. Modify the historical tsfile file, the original tsfile->temporary tsfile, do the switching operation after the writing is completed, and write in the positive sequence
> 5. When the command is executed, the sealing operation is performed immediately, and the sealed file will be rewritten when the history file is modified (to be discussed)
> 6. If currently in merge, reject the command? (to be discussed)
> 7. Interface development, first modify the part that affects incremental data, and then modify the historical tsfile file
> 8. SQL development (does not support the first version?)
> 9. Service restart command recovery operation requires persistent execution process
> 10. Perform pre- and post-checking, possibly saving a summary of each tsfile before and after modification for verification
> Possibly noteworthy:
> 1. Lock control
> 2. Concurrency control
> 3. Permission control
> question:
> 1. If this is feasible, which version is better to develop in, 12/13/14



--
This message was sent by Atlassian Jira
(v8.20.10#820010)