[jira] [Updated] (FLINK-15570) Support inserting into non-empty csv tables

["Caizhi Weng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Caizhi Weng updated FLINK-15570:
--------------------------------
    Description: 
Currently we only support inserting into empty csv tables. It would be nice for a new user from traditional database to try out Flink if we support inserting into non-empty csv tables.

In other words, it would be nice if the following SQL produces a valid result:
{code:java}
CREATE TABLE myTable(
    a INT,
    b DOUBLE
) WITH (
    'connector.type' = 'filesystem',
    'connector.path' = an existing csv file,
    'format.type' = 'csv',
    'format.derive-schema' = 'true'
);

SELECT * FROM myTable; // produces 3 rows

INSERT INTO myTable VALUES (4, 4.4);

SELECT * FROM myTable; // produces 4 rows{code}
We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In NO_OVERWRITE mode we can only insert into empty csv tables, while in OVERWRITE mode inserting into a csv table will wipe all existing data in it (in the above example, the last SELECT will produce only 1 instead of 4 rows) which is really strange for a mere INSERT operation.

We need to add a new APPEND write mode, or change the behavior for OVERWRITE mode to something like appending to files. Supporting specifying write mode in the WITH clause is also needed.

  was:
Currently we only support inserting into empty csv tables. It would be nice for a new user from traditional database to try out Flink if we support inserting into non-empty csv tables.

In other words, it would be nice if the following SQL produces a valid result:
{code:java}
CREATE TABLE myTable(
    a INT,
    b DOUBLE
) WITH (
    'connector.type' = 'filesystem',
    'connector.path' = an existing csv file,
    'format.type' = 'csv',
    'format.derive-schema' = 'true'
);

SELECT * FROM myTable; // produces 3 rows

INSERT INTO myTable VALUES (4, 4.4);

SELECT * FROM myTable; // produces 4 rows{code}
We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In NO_OVERWRITE mode we can only insert into empty csv tables, while in OVERWRITE mode inserting into a csv table will wipe all existing data in it (in the above example, the last SELECT will produce only 1 instead of 4 rows) which is really strange for a mere INSERT operation. We need to add a new APPEND write mode, or change the behavior for OVERWRITE mode to something like appending to files.


> Support inserting into non-empty csv tables
> -------------------------------------------
>
>                 Key: FLINK-15570
>                 URL: https://issues.apache.org/jira/browse/FLINK-15570
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>            Reporter: Caizhi Weng
>            Priority: Minor
>
> Currently we only support inserting into empty csv tables. It would be nice for a new user from traditional database to try out Flink if we support inserting into non-empty csv tables.
> In other words, it would be nice if the following SQL produces a valid result:
> {code:java}
> CREATE TABLE myTable(
>     a INT,
>     b DOUBLE
> ) WITH (
>     'connector.type' = 'filesystem',
>     'connector.path' = an existing csv file,
>     'format.type' = 'csv',
>     'format.derive-schema' = 'true'
> );
> SELECT * FROM myTable; // produces 3 rows
> INSERT INTO myTable VALUES (4, 4.4);
> SELECT * FROM myTable; // produces 4 rows{code}
> We currently only have two write modes, namely NO_OVERWRITE and OVERWRITE. In NO_OVERWRITE mode we can only insert into empty csv tables, while in OVERWRITE mode inserting into a csv table will wipe all existing data in it (in the above example, the last SELECT will produce only 1 instead of 4 rows) which is really strange for a mere INSERT operation.
> We need to add a new APPEND write mode, or change the behavior for OVERWRITE mode to something like appending to files. Supporting specifying write mode in the WITH clause is also needed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)