[jira] [Commented] (KYLIN-5991) Multiple abnormal errors in internal tables

["Guoliang Sun (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927980#comment-17927980 ] 

Guoliang Sun commented on KYLIN-5991:
-------------------------------------

h3. Root Cause & Dev Design
 # When creating an internal table, the case where the tablePartition field of the internal table is null was not considered, which led to the NPE issue.
 # When Gluten is not enabled in the configuration, during the process of loading a data source and simultaneously loading it as an internal table, validate the configuration and throw the corresponding exception.
 # When submitting an incremental refresh task, validate whether the refresh time range exceeds the existing time range.
 # Due to the metadata structure upgrade in KE5, some older Tool classes are no longer supported. During the upgrade script execution, some Tools cannot be found, causing the upgrade to exit halfway and leaving subsequent processes incomplete.
 ## Skip some upgrade Tool processes required for version 4.x
 ### UpdateUserAclTool
 ### UpgradeExcludedTableTool
 # The logic of converting a regular table to an internal table essentially involves saving empty data to the delta directory, which is divided into two steps:
 ## Create delta_log.
 ## Save empty data.
 ## If the second step fails and the delta_log file is not promptly cleared, it can lead to a mismatch in the schema within the delta_log if the table structure changes or related sort keys and primary keys are adjusted later.
 # Capture exceptions thrown during delta.save() and delete the already created delta_log directory on HDFS.
 # When a single-table model corresponding to an internal table exists, the metrics and CC (Calculated Columns) on the model are treated as columns of the table during execution plan generation. This causes queries on the internal table data to throw "column does not exist" exceptions.

 ## Refer to the snapshot logic. When querying internal table data, use 1 to populate the CC columns and metrics.

 # Add validation to check if the internal table exists.
 # In the parameter validation phase, add logic to validate partition columns when *date_partition_format* is not empty.
 # In internal table operations, some error code messages do not comply with the standards. xxxx have not been added to the corresponding configuration files, causing the correct error_code to be unrecognized when an error occurs.
 ## Solution: Add the error codes xxxx to the corresponding configuration files, *kylin_errorcode_conf_en.properties* and {*}kylin_errorcode_conf_zh.properties{*}. This will ensure accurate error codes are generated.

> Multiple abnormal errors in internal tables
> -------------------------------------------
>
>                 Key: KYLIN-5991
>                 URL: https://issues.apache.org/jira/browse/KYLIN-5991
>             Project: Kylin
>          Issue Type: Bug
>    Affects Versions: 5.0.0
>            Reporter: Guoliang Sun
>            Priority: Major
>
> # [Single-table Load/Refresh API] Incremental load without partition columns, API does not throw an error.
>  # Failed to delete table without internal table
>  # [Single-table Refresh] Date out of range not validated
>  ## When a time partition column exists and the selected date exceeds the range of existing data, the refresh request does not throw an error and loads data beyond the time range.
>  # [KE5 Upgrade] KE5 upgraded using the upgrade process, but the old version directory was retained and not deleted.
>  # [Create Internal Table] Inconsistent transaction: Delta log was successfully written, but the table metadata was not updated, causing subsequent creation of this internal table to fail.
>  # When an internal table exists, the model cannot be hit by a query.
>  ## Create a project and enable internal tables.
>  ## Load the data source table.
>  ## Create an internal table.
>  ## Use this table for modeling and building.
>  ## After the model is built, query the model. Expected to hit the model, but the query shows pushdown (and pushdown does not hit the internal table).
>  # [Internal Table API] The URL parameters for table and database in the data clearing interface are not separated.
>  ## Need to be consistent with other internal table APIs, and pass the parameters separately.
>  # [Source Table - Reload] After adding or deleting columns in the source table, it is necessary to check whether the table is an internal table with data during loading.
>  ## If it is an internal table with data, reloading is not allowed. The user should be prompted to clear the internal table data before reloading.
>  # [Internal Table API] The delete table partition interface does not validate the 'partitions' parameter.
>  # [Internal Table API] The interface for getting internal table partition details does not validate the database name and table name parameters.
>  ## Passing non-existent values for the database name and table name results in a successful response.
>  # [Internal Table API] When creating an internal table, the partition column and time partition format are not cross-validated.
>  ## The request succeeds when no partition column is passed, but only the time partition format is provided.
>  # Normalizes the Error Code of the inner table apis
>  ## The error message is unreasonable. When Accept-Language=cn is set, the error message is still in English.
>  # Support Internal Table Open API
>  ## Add a parameter to the table reload interface to support directly clearing data and reloading the table (the UI will prompt users to clear the data themselves before reloading).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)