You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/11/03 16:44:24 UTC

[GitHub] [flink] wuchong opened a new pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

wuchong opened a new pull request #13909:
URL: https://github.com/apache/flink/pull/13909


   <!--
   *Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   *Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
     
     - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
     Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message (including the JIRA id)
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   
   
   **(The sections below can be removed for hotfixes of typos)**
   -->
   
   ## What is the purpose of the change
   
   This is a code improvement based on #11896. 
   
   This pull request introduct `single-field` format to (de)serialize message to a single field.
   
   ## Brief change log
   
   Modifications based on  #11896:
   - Rename format from `single-value` to `single-field`, because it's more literal that it's used for single field/column. And personally, 'value.format' = 'single-field' is better than 'value.format' = 'single-value' when it's used in Kafka connector. KSQL also has a similar feature for [single field (un)wrapping](https://docs.ksqldb.io/en/latest/developer-guide/serialization/#single-field-unwrapping).
   - Move clases to `flink-table-runtime-blink` for easy testing. 
   - Using Flink's primitive TypeSerializers for encoding/decoding primitive values.
   - Encode `null` values as `null` `byte[]`. This may have limitation when used in `upsert-kafka`, because `upsert-kafka` treats null values as a tombstone message. Therefore, we recommend avoiding using `upsert-kafka` connector and `single-field` format if the field can have a null value. KSQL also has this [limitation](https://docs.ksqldb.io/en/latest/developer-guide/serialization/#controlling-deserializing-of-single-fields).
   
   ## Verifying this change
   
   - `SingleFieldFormatFactoryTest` for verifing format factory
   - `SingleFieldSerDeSchemaTest` for verifying serializing and deserializing values with different types.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / **no**)
     - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de5115aa0bbc4bc486dced3d2bc67d34823659f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de5115aa0bbc4bc486dced3d2bc67d34823659f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de5115aa0bbc4bc486dced3d2bc67d34823659f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912) 
   * 6051bbecc6f91a54f72d7982b5390e8569189caa UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164",
       "triggerID" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175",
       "triggerID" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de5115aa0bbc4bc486dced3d2bc67d34823659f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912) 
   * 6051bbecc6f91a54f72d7982b5390e8569189caa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164) 
   * fe7ef3f93e33dbe1eae8965707f83432e1e9859d UNKNOWN
   * d60a354989d3e3a3e6cea3fcce6903a35fa72965 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164",
       "triggerID" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de5115aa0bbc4bc486dced3d2bc67d34823659f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912) 
   * 6051bbecc6f91a54f72d7982b5390e8569189caa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164) 
   * fe7ef3f93e33dbe1eae8965707f83432e1e9859d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de5115aa0bbc4bc486dced3d2bc67d34823659f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wuchong commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518120299



##########
File path: docs/dev/table/connectors/formats/single-field.md
##########
@@ -0,0 +1,151 @@
+---
+title: "Single Field Format"
+nav-title: SingleField
+nav-parent_id: sql-formats
+nav-pos: 7
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<span class="label label-info">Format: Serialization Schema</span>
+<span class="label label-info">Format: Deserialization Schema</span>
+
+* This will be replaced by the TOC
+{:toc}
+
+The SingleField format allows to read and write data where the data contains only a single field, and that field is not wrapped within a JSON object, or an Avro record.
+
+Currently, the SingleField format supports `String`, `byte[]` and primitive type.
+
+Note: this format encodes `null` values as `null` `byte[]`. This may have limitation when used in `upsert-kafka`, because `upsert-kafka` treats `null` values as a tombstone message (DELETE on the key). Therefore, we recommend avoiding using `upsert-kafka` connector and `single-field` format if the field can have a `null` value.
+
+Example
+----------------
+
+For example, you may have following raw log data in Kafka and want to read and analyse such data using Flink SQL.
+
+```
+47.29.201.179 - - [28/Feb/2019:13:17:10 +0000] "GET /?p=1 HTTP/2.0" 200 5316 "https://domain.com/?p=1" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"
+```
+
+The following creates a table where it reads from (and writes to) the underlying Kafka topic as an anonymous string value by using `single-field` format:
+
+<div class="codetabs" markdown="1">
+<div data-lang="SQL" markdown="1">
+{% highlight sql %}
+CREATE TABLE nginx_log (
+  log STRING
+) WITH (
+  'connector' = 'kafka',
+  'topic' = 'nginx_log',
+  'properties.bootstrap.servers' = 'localhost:9092',
+  'properties.group.id' = 'testGroup',
+  'format' = 'single-field'
+)
+{% endhighlight %}
+</div>
+</div>
+
+Then you can read out the raw data as a pure string, and split it into multiple fields using user-defined-function for further analysing, e.g. `my_split` in the example.
+
+<div class="codetabs" markdown="1">
+<div data-lang="SQL" markdown="1">
+{% highlight sql %}
+SELECT t.hostname, t.datetime, t.url, t.browser, ...
+FROM(
+  SELECT my_split(log) as t FROM nginx_log
+);
+{% endhighlight %}
+</div>
+</div>
+
+In contrast, you can also write a single field of STRING type into Kafka topic as an anonymous string value.
+
+Format Options
+----------------
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left" style="width: 25%">Option</th>
+        <th class="text-center" style="width: 8%">Required</th>
+        <th class="text-center" style="width: 7%">Default</th>
+        <th class="text-center" style="width: 10%">Type</th>
+        <th class="text-center" style="width: 50%">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+    <tr>
+      <td><h5>format</h5></td>
+      <td>required</td>
+      <td style="word-wrap: break-word;">(none)</td>
+      <td>String</td>
+      <td>Specify what format to use, here should be 'single-field'.</td>
+    </tr>
+    </tbody>
+</table>
+
+Data Type Mapping
+----------------
+
+The table below details the SQL types the format supports, including details of the serializer and deserializer class for encoding and decoding.
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left">Flink SQL type</th>
+        <th class="text-left">Value</th>
+      </tr>
+    </thead>
+    <tbody>
+    <tr>
+      <td><code>CHAR / VARCHAR / STRING</code></td>
+      <td>A UTF-8 encoded text string</td>
+    </tr>
+    <tr>
+      <td><code>BOOLEAN</code></td>
+      <td>A single byte to indicate boolean value, 0 means false, 1 means true.</td>
+    </tr>
+    <tr>
+      <td><code>TINYINT</code></td>
+      <td>A 8-bit signed number</td>
+    </tr>
+    <tr>
+      <td><code>SMALLINT</code></td>
+      <td>A 16-bit signed number</td>
+    </tr>
+    <tr>
+      <td><code>INT</code></td>
+      <td>A 32-bit signed integer</td>
+    </tr>
+    <tr>
+      <td><code>BIGINT</code></td>
+      <td>A 64-bit signed integer</td>
+    </tr>
+    <tr>
+      <td><code>FLOAT</code></td>
+      <td>A 32-bit floating point number</td>
+    </tr>
+    <tr>

Review comment:
       I'm not sure about `RAW` type, because currently it's hard to declare RAW type in DDL.
   As an alternative, users can declare it as `BYTES` and use UDF to deserialize the bytes. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164",
       "triggerID" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175",
       "triggerID" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6051bbecc6f91a54f72d7982b5390e8569189caa Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164) 
   * fe7ef3f93e33dbe1eae8965707f83432e1e9859d UNKNOWN
   * d60a354989d3e3a3e6cea3fcce6903a35fa72965 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wuchong commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518104838



##########
File path: docs/dev/table/connectors/formats/single-field.md
##########
@@ -0,0 +1,151 @@
+---
+title: "Single Field Format"
+nav-title: SingleField
+nav-parent_id: sql-formats
+nav-pos: 7
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<span class="label label-info">Format: Serialization Schema</span>
+<span class="label label-info">Format: Deserialization Schema</span>
+
+* This will be replaced by the TOC
+{:toc}
+
+The SingleField format allows to read and write data where the data contains only a single field, and that field is not wrapped within a JSON object, or an Avro record.
+
+Currently, the SingleField format supports `String`, `byte[]` and primitive type.
+
+Note: this format encodes `null` values as `null` `byte[]`. This may have limitation when used in `upsert-kafka`, because `upsert-kafka` treats `null` values as a tombstone message (DELETE on the key). Therefore, we recommend avoiding using `upsert-kafka` connector and `single-field` format if the field can have a `null` value.

Review comment:
       Yes.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wuchong commented on pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
wuchong commented on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-723387225


   The failed e2e should not be related to this PR. I created https://issues.apache.org/jira/browse/FLINK-20039 to track the unstable e2e test. 
   
   Will merge this. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wuchong commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518833730



##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/raw/RawFormatDeserializationSchema.java
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.raw;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.HeapMemorySegment;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RawValueData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RawType;
+import org.apache.flink.types.DeserializationException;
+import org.apache.flink.util.function.FunctionUtils;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+import java.util.function.Consumer;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Deserialization schema from raw (byte based) value to Flink Table/SQL internal data structure {@link RowData}.
+ */
+@Internal
+public class RawFormatDeserializationSchema implements DeserializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType deserializedType;
+
+	private final DeserializationRuntimeConverter converter;
+
+	private final TypeInformation<RowData> producedTypeInfo;
+
+	private final String charsetName;
+
+	private final boolean isBigEndian;
+
+	private transient GenericRowData reuse;
+
+	public RawFormatDeserializationSchema(
+			LogicalType deserializedType,
+			TypeInformation<RowData> producedTypeInfo,
+			String charsetName,
+			boolean isBigEndian) {
+		this.deserializedType = checkNotNull(deserializedType);
+		this.producedTypeInfo = checkNotNull(producedTypeInfo);
+		this.converter = createConverter(deserializedType, charsetName, isBigEndian);
+		this.charsetName = charsetName;
+		this.isBigEndian = isBigEndian;
+	}
+
+	@Override
+	public void open(InitializationContext context) throws Exception {
+		reuse = new GenericRowData(1);
+		converter.open();
+	}
+
+	@Override
+	public RowData deserialize(byte[] message) throws IOException {
+		reuse.setField(0, converter.convert(message));
+		return reuse;
+	}
+
+	@Override
+	public boolean isEndOfStream(RowData nextElement) {
+		return false;
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return producedTypeInfo;
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		RawFormatDeserializationSchema that = (RawFormatDeserializationSchema) o;
+		return producedTypeInfo.equals(that.producedTypeInfo) &&
+			deserializedType.equals(that.deserializedType) &&
+			charsetName.equals(that.charsetName) &&
+			isBigEndian == that.isBigEndian;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(producedTypeInfo, deserializedType, charsetName, isBigEndian);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert byte[] to internal data structure object.
+	 */
+	@FunctionalInterface
+	private interface DeserializationRuntimeConverter extends Serializable {
+
+		default void open() {}
+
+		Object convert(byte[] data) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private static DeserializationRuntimeConverter createConverter(
+			LogicalType type, String charsetName, boolean isBigEndian) {
+		final DeserializationRuntimeConverter converter = createNotNullConverter(type, charsetName, isBigEndian);
+		final Consumer<byte[]> validator = createDataLengthValidator(type);
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+
+			@Override
+			public void open() {
+				converter.open();
+			}
+
+			@Override
+			public Object convert(byte[] data) throws IOException {
+				if (data == null) {

Review comment:
       Why? The Kafka message value might be null.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721246593


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit 8de5115aa0bbc4bc486dced3d2bc67d34823659f (Tue Nov 03 16:46:37 UTC 2020)
   
    ✅no warnings
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164",
       "triggerID" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175",
       "triggerID" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fe7ef3f93e33dbe1eae8965707f83432e1e9859d UNKNOWN
   * d60a354989d3e3a3e6cea3fcce6903a35fa72965 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wuchong commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518833855



##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/raw/RawFormatDeserializationSchema.java
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.raw;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.HeapMemorySegment;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RawValueData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RawType;
+import org.apache.flink.types.DeserializationException;
+import org.apache.flink.util.function.FunctionUtils;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+import java.util.function.Consumer;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Deserialization schema from raw (byte based) value to Flink Table/SQL internal data structure {@link RowData}.
+ */
+@Internal
+public class RawFormatDeserializationSchema implements DeserializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType deserializedType;
+
+	private final DeserializationRuntimeConverter converter;
+
+	private final TypeInformation<RowData> producedTypeInfo;
+
+	private final String charsetName;
+
+	private final boolean isBigEndian;
+
+	private transient GenericRowData reuse;
+
+	public RawFormatDeserializationSchema(
+			LogicalType deserializedType,
+			TypeInformation<RowData> producedTypeInfo,
+			String charsetName,
+			boolean isBigEndian) {
+		this.deserializedType = checkNotNull(deserializedType);
+		this.producedTypeInfo = checkNotNull(producedTypeInfo);
+		this.converter = createConverter(deserializedType, charsetName, isBigEndian);
+		this.charsetName = charsetName;
+		this.isBigEndian = isBigEndian;
+	}
+
+	@Override
+	public void open(InitializationContext context) throws Exception {
+		reuse = new GenericRowData(1);
+		converter.open();
+	}
+
+	@Override
+	public RowData deserialize(byte[] message) throws IOException {
+		reuse.setField(0, converter.convert(message));
+		return reuse;
+	}
+
+	@Override
+	public boolean isEndOfStream(RowData nextElement) {
+		return false;
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return producedTypeInfo;
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		RawFormatDeserializationSchema that = (RawFormatDeserializationSchema) o;
+		return producedTypeInfo.equals(that.producedTypeInfo) &&
+			deserializedType.equals(that.deserializedType) &&
+			charsetName.equals(that.charsetName) &&
+			isBigEndian == that.isBigEndian;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(producedTypeInfo, deserializedType, charsetName, isBigEndian);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert byte[] to internal data structure object.
+	 */
+	@FunctionalInterface
+	private interface DeserializationRuntimeConverter extends Serializable {
+
+		default void open() {}
+
+		Object convert(byte[] data) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private static DeserializationRuntimeConverter createConverter(
+			LogicalType type, String charsetName, boolean isBigEndian) {
+		final DeserializationRuntimeConverter converter = createNotNullConverter(type, charsetName, isBigEndian);
+		final Consumer<byte[]> validator = createDataLengthValidator(type);
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+
+			@Override
+			public void open() {
+				converter.open();
+			}
+
+			@Override
+			public Object convert(byte[] data) throws IOException {
+				if (data == null) {
+					return null;
+				}
+				validator.accept(data);
+				return converter.convert(data);
+			}
+		};
+	}
+
+	private static DeserializationRuntimeConverter createNotNullConverter(
+			LogicalType type, String charsetName, boolean isBigEndian) {
+		switch (type.getTypeRoot()) {
+			case CHAR:
+			case VARCHAR:
+				return createStringConverter(charsetName);
+
+			case VARBINARY:
+			case BINARY:
+				return data -> data;
+
+			case RAW:
+				return createRawValueConverter((RawType<?>) type);
+
+			case BOOLEAN:
+				return data -> data[0] != 0;
+
+			case TINYINT:
+				return data -> data[0];
+
+			case SMALLINT:
+				return createEndiannessAwareConverter(
+					isBigEndian,
+					segment -> segment.getShortBigEndian(0),
+					segment -> segment.getShortLittleEndian(0)
+				);
+
+			case INTEGER:
+				return createEndiannessAwareConverter(
+					isBigEndian,
+					segment -> segment.getIntBigEndian(0),
+					segment -> segment.getIntLittleEndian(0)
+				);
+
+			case BIGINT:
+				return createEndiannessAwareConverter(
+					isBigEndian,
+					segment -> segment.getLongBigEndian(0),
+					segment -> segment.getLongLittleEndian(0)
+				);
+
+			case FLOAT:
+				return createEndiannessAwareConverter(
+					isBigEndian,
+					segment -> segment.getFloatBigEndian(0),
+					segment -> segment.getFloatLittleEndian(0)
+				);
+
+			case DOUBLE:
+				return createEndiannessAwareConverter(
+					isBigEndian,
+					segment -> segment.getDoubleBigEndian(0),
+					segment -> segment.getDoubleLittleEndian(0)
+				);
+
+			default:
+				throw new UnsupportedOperationException("'raw' format currently doesn't support type: " + type);
+		}
+	}
+
+	private static DeserializationRuntimeConverter createStringConverter(final String charsetName) {
+		// this also checks the charsetName is valid
+		Charset charset = Charset.forName(charsetName);
+		if (charset == StandardCharsets.UTF_8) {
+			// avoid UTF-8 decoding if the given charset is UTF-8
+			// because the underlying bytes of StringData is in UTF-8 encoding
+			return StringData::fromBytes;
+		}
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+			private transient Charset charset;
+			@Override
+			public void open() {
+				charset = Charset.forName(charsetName);
+			}
+
+			@Override
+			public Object convert(byte[] data) {
+				String str = new String(data, charset);
+				return StringData.fromString(str);
+			}
+		};
+	}
+
+	private static DeserializationRuntimeConverter createRawValueConverter(RawType<?> rawType) {
+		final TypeSerializer<?> serializer = rawType.getTypeSerializer();
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+			private final DataInputDeserializer source = new DataInputDeserializer();
+			@Override
+			public Object convert(byte[] data) throws IOException {
+				source.setBuffer(data);
+				return RawValueData.fromObject(serializer.deserialize(source));

Review comment:
       Yes. I think we can. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] twalthr commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518852188



##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/raw/RawFormatDeserializationSchema.java
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.raw;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.HeapMemorySegment;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RawValueData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RawType;
+import org.apache.flink.types.DeserializationException;
+import org.apache.flink.util.function.FunctionUtils;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+import java.util.function.Consumer;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Deserialization schema from raw (byte based) value to Flink Table/SQL internal data structure {@link RowData}.
+ */
+@Internal
+public class RawFormatDeserializationSchema implements DeserializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType deserializedType;
+
+	private final DeserializationRuntimeConverter converter;
+
+	private final TypeInformation<RowData> producedTypeInfo;
+
+	private final String charsetName;
+
+	private final boolean isBigEndian;
+
+	private transient GenericRowData reuse;
+
+	public RawFormatDeserializationSchema(
+			LogicalType deserializedType,
+			TypeInformation<RowData> producedTypeInfo,
+			String charsetName,
+			boolean isBigEndian) {
+		this.deserializedType = checkNotNull(deserializedType);
+		this.producedTypeInfo = checkNotNull(producedTypeInfo);
+		this.converter = createConverter(deserializedType, charsetName, isBigEndian);
+		this.charsetName = charsetName;
+		this.isBigEndian = isBigEndian;
+	}
+
+	@Override
+	public void open(InitializationContext context) throws Exception {
+		reuse = new GenericRowData(1);
+		converter.open();
+	}
+
+	@Override
+	public RowData deserialize(byte[] message) throws IOException {
+		reuse.setField(0, converter.convert(message));
+		return reuse;
+	}
+
+	@Override
+	public boolean isEndOfStream(RowData nextElement) {
+		return false;
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return producedTypeInfo;
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		RawFormatDeserializationSchema that = (RawFormatDeserializationSchema) o;
+		return producedTypeInfo.equals(that.producedTypeInfo) &&
+			deserializedType.equals(that.deserializedType) &&
+			charsetName.equals(that.charsetName) &&
+			isBigEndian == that.isBigEndian;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(producedTypeInfo, deserializedType, charsetName, isBigEndian);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert byte[] to internal data structure object.
+	 */
+	@FunctionalInterface
+	private interface DeserializationRuntimeConverter extends Serializable {
+
+		default void open() {}
+
+		Object convert(byte[] data) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private static DeserializationRuntimeConverter createConverter(
+			LogicalType type, String charsetName, boolean isBigEndian) {
+		final DeserializationRuntimeConverter converter = createNotNullConverter(type, charsetName, isBigEndian);
+		final Consumer<byte[]> validator = createDataLengthValidator(type);
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+
+			@Override
+			public void open() {
+				converter.open();
+			}
+
+			@Override
+			public Object convert(byte[] data) throws IOException {
+				if (data == null) {

Review comment:
       but let's keep the null check for now




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] twalthr commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518007361



##########
File path: docs/dev/table/connectors/formats/index.md
##########
@@ -68,5 +68,10 @@ Flink supports the following formats:
          <td><a href="{% link dev/table/connectors/formats/orc.md %}">Apache ORC</a></td>
          <td><a href="{% link dev/table/connectors/filesystem.md %}">Filesystem</a></td>
         </tr>
+        <tr>
+        <td><a href="{% link dev/table/connectors/formats/singleValue.md %}">Single Value</a></td>

Review comment:
       According to the offline discussion, call the format `raw` instead.

##########
File path: docs/dev/table/connectors/formats/single-field.md
##########
@@ -0,0 +1,151 @@
+---
+title: "Single Field Format"
+nav-title: SingleField
+nav-parent_id: sql-formats
+nav-pos: 7
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<span class="label label-info">Format: Serialization Schema</span>
+<span class="label label-info">Format: Deserialization Schema</span>
+
+* This will be replaced by the TOC
+{:toc}
+
+The SingleField format allows to read and write data where the data contains only a single field, and that field is not wrapped within a JSON object, or an Avro record.
+
+Currently, the SingleField format supports `String`, `byte[]` and primitive type.
+
+Note: this format encodes `null` values as `null` `byte[]`. This may have limitation when used in `upsert-kafka`, because `upsert-kafka` treats `null` values as a tombstone message (DELETE on the key). Therefore, we recommend avoiding using `upsert-kafka` connector and `single-field` format if the field can have a `null` value.

Review comment:
       ```
   we recommend avoiding using `upsert-kafka` connector and the `raw` format as a `value.format`
   ```
   
   We should emphasize that a key format is fine.

##########
File path: docs/dev/table/connectors/formats/single-field.md
##########
@@ -0,0 +1,151 @@
+---
+title: "Single Field Format"
+nav-title: SingleField
+nav-parent_id: sql-formats
+nav-pos: 7
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<span class="label label-info">Format: Serialization Schema</span>
+<span class="label label-info">Format: Deserialization Schema</span>
+
+* This will be replaced by the TOC
+{:toc}
+
+The SingleField format allows to read and write data where the data contains only a single field, and that field is not wrapped within a JSON object, or an Avro record.
+
+Currently, the SingleField format supports `String`, `byte[]` and primitive type.
+
+Note: this format encodes `null` values as `null` `byte[]`. This may have limitation when used in `upsert-kafka`, because `upsert-kafka` treats `null` values as a tombstone message (DELETE on the key). Therefore, we recommend avoiding using `upsert-kafka` connector and `single-field` format if the field can have a `null` value.
+
+Example
+----------------
+
+For example, you may have following raw log data in Kafka and want to read and analyse such data using Flink SQL.
+
+```
+47.29.201.179 - - [28/Feb/2019:13:17:10 +0000] "GET /?p=1 HTTP/2.0" 200 5316 "https://domain.com/?p=1" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"
+```
+
+The following creates a table where it reads from (and writes to) the underlying Kafka topic as an anonymous string value by using `single-field` format:
+
+<div class="codetabs" markdown="1">
+<div data-lang="SQL" markdown="1">
+{% highlight sql %}
+CREATE TABLE nginx_log (
+  log STRING
+) WITH (
+  'connector' = 'kafka',
+  'topic' = 'nginx_log',
+  'properties.bootstrap.servers' = 'localhost:9092',
+  'properties.group.id' = 'testGroup',
+  'format' = 'single-field'
+)
+{% endhighlight %}
+</div>
+</div>
+
+Then you can read out the raw data as a pure string, and split it into multiple fields using user-defined-function for further analysing, e.g. `my_split` in the example.
+
+<div class="codetabs" markdown="1">
+<div data-lang="SQL" markdown="1">
+{% highlight sql %}
+SELECT t.hostname, t.datetime, t.url, t.browser, ...
+FROM(
+  SELECT my_split(log) as t FROM nginx_log
+);
+{% endhighlight %}
+</div>
+</div>
+
+In contrast, you can also write a single field of STRING type into Kafka topic as an anonymous string value.
+
+Format Options
+----------------
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left" style="width: 25%">Option</th>
+        <th class="text-center" style="width: 8%">Required</th>
+        <th class="text-center" style="width: 7%">Default</th>
+        <th class="text-center" style="width: 10%">Type</th>
+        <th class="text-center" style="width: 50%">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+    <tr>
+      <td><h5>format</h5></td>
+      <td>required</td>
+      <td style="word-wrap: break-word;">(none)</td>
+      <td>String</td>
+      <td>Specify what format to use, here should be 'single-field'.</td>
+    </tr>
+    </tbody>
+</table>
+
+Data Type Mapping
+----------------
+
+The table below details the SQL types the format supports, including details of the serializer and deserializer class for encoding and decoding.
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left">Flink SQL type</th>
+        <th class="text-left">Value</th>
+      </tr>
+    </thead>
+    <tbody>
+    <tr>
+      <td><code>CHAR / VARCHAR / STRING</code></td>
+      <td>A UTF-8 encoded text string</td>
+    </tr>
+    <tr>
+      <td><code>BOOLEAN</code></td>
+      <td>A single byte to indicate boolean value, 0 means false, 1 means true.</td>
+    </tr>
+    <tr>
+      <td><code>TINYINT</code></td>
+      <td>A 8-bit signed number</td>
+    </tr>
+    <tr>
+      <td><code>SMALLINT</code></td>
+      <td>A 16-bit signed number</td>
+    </tr>
+    <tr>
+      <td><code>INT</code></td>
+      <td>A 32-bit signed integer</td>
+    </tr>
+    <tr>
+      <td><code>BIGINT</code></td>
+      <td>A 64-bit signed integer</td>
+    </tr>
+    <tr>
+      <td><code>FLOAT</code></td>
+      <td>A 32-bit floating point number</td>
+    </tr>
+    <tr>

Review comment:
       let's also support BINARY/VARBINARY/BYTES and the RAW type itself to finalize this story

##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/singlefield/SingleFieldSerializationSchema.java
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.singlefield;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.BooleanSerializer;
+import org.apache.flink.api.common.typeutils.base.ByteSerializer;
+import org.apache.flink.api.common.typeutils.base.DoubleSerializer;
+import org.apache.flink.api.common.typeutils.base.FloatSerializer;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeutils.base.ShortSerializer;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.RowData.FieldGetter;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Objects;
+
+/**
+ * Serialization schema that serializes an {@link RowData} object into a single field bytes.
+ */
+@Internal
+public class SingleFieldSerializationSchema implements SerializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType fieldType;
+
+	private final SerializationRuntimeConverter converter;
+
+	private final FieldGetter fieldGetter;
+
+	public SingleFieldSerializationSchema(LogicalType fieldType) {
+		this.fieldType = fieldType;
+		this.fieldGetter = RowData.createFieldGetter(fieldType, 0);
+		this.converter = createConverter(fieldType);
+	}
+
+	@Override
+	public byte[] serialize(RowData row) {
+		try {
+			return converter.convert(fieldGetter.getFieldOrNull(row));
+		} catch (IOException e) {
+			throw new RuntimeException("Could not serialize row '" + row + "'. ", e);
+		}
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		SingleFieldSerializationSchema that = (SingleFieldSerializationSchema) o;
+		return fieldType.equals(that.fieldType);
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(fieldType);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert an object of internal data structure to byte[].
+	 */
+	@FunctionalInterface
+	private interface SerializationRuntimeConverter extends Serializable {
+		byte[] convert(Object value) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private SerializationRuntimeConverter createConverter(LogicalType type) {
+		switch (type.getTypeRoot()) {
+			case CHAR:
+			case VARCHAR:
+				return value -> {
+					// put null check logic in the lambda instead of wrapping outside
+					// to avoid virtual method invoking.
+					if (value == null) {
+						return null;
+					}
+					return ((StringData) value).toBytes();
+				};
+			case VARBINARY:
+			case BINARY:
+				return value -> {
+					if (value == null) {
+						return null;
+					}
+					return (byte[]) value;
+				};
+			case TINYINT:
+				return createConverterUsingSerializer(ByteSerializer.INSTANCE);
+			case SMALLINT:
+				return createConverterUsingSerializer(ShortSerializer.INSTANCE);
+			case INTEGER:
+				return createConverterUsingSerializer(IntSerializer.INSTANCE);
+			case BIGINT:
+				return createConverterUsingSerializer(LongSerializer.INSTANCE);
+			case FLOAT:
+				return createConverterUsingSerializer(FloatSerializer.INSTANCE);
+			case DOUBLE:
+				return createConverterUsingSerializer(DoubleSerializer.INSTANCE);
+			case BOOLEAN:
+				return createConverterUsingSerializer(BooleanSerializer.INSTANCE);
+			default:
+				throw new UnsupportedOperationException("'single-format' currently doesn't support type: " + type);
+		}
+	}
+
+	@SuppressWarnings("unchecked")
+	private static SerializationRuntimeConverter createConverterUsingSerializer(
+			TypeSerializer<?> serializer) {
+		return new DelegatingSerializationConverter((TypeSerializer<Object>) serializer);
+	}
+
+	private static final class DelegatingSerializationConverter
+		implements SerializationRuntimeConverter {
+		private static final long serialVersionUID = 1L;
+		private final DataOutputSerializer dos = new DataOutputSerializer(16);
+		private final TypeSerializer<Object> delegatingSerializer;
+
+		protected DelegatingSerializationConverter(TypeSerializer<Object> delegatingSerializer) {
+			this.delegatingSerializer = delegatingSerializer;
+		}
+
+		@Override
+		public byte[] convert(Object value) throws IOException {
+			if (value == null) {
+				return null;
+			}
+			delegatingSerializer.serialize(value, dos);
+			byte[] ret = dos.getCopyOfBuffer();

Review comment:
       Same comment as before, this looks overly complicated to just convert a a couple of data types to bytes. We should think about doing it manually.

##########
File path: flink-table/flink-table-runtime-blink/src/test/java/org/apache/flink/table/formats/singlefield/SingleFieldSerDeSchemaTest.java
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.singlefield;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.conversion.DataStructureConverter;
+import org.apache.flink.table.data.conversion.DataStructureConverters;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static org.apache.flink.table.api.DataTypes.BIGINT;
+import static org.apache.flink.table.api.DataTypes.BINARY;
+import static org.apache.flink.table.api.DataTypes.BOOLEAN;
+import static org.apache.flink.table.api.DataTypes.BYTES;
+import static org.apache.flink.table.api.DataTypes.DOUBLE;
+import static org.apache.flink.table.api.DataTypes.FIELD;
+import static org.apache.flink.table.api.DataTypes.FLOAT;
+import static org.apache.flink.table.api.DataTypes.INT;
+import static org.apache.flink.table.api.DataTypes.ROW;
+import static org.apache.flink.table.api.DataTypes.SMALLINT;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.DataTypes.TINYINT;
+import static org.apache.flink.table.api.DataTypes.VARCHAR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+
+/**
+ * Tests for {@link SingleFieldDeserializationSchema} {@link SingleFieldSerializationSchema}.
+ */
+public class SingleFieldSerDeSchemaTest {
+
+	@Test
+	public void testSerializationAndDeserialization() throws Exception {
+		for (TestSpec testSpec : testData) {
+			runTest(testSpec);
+		}
+	}
+
+	private void runTest(TestSpec testSpec) throws Exception {
+		SingleFieldDeserializationSchema deserializationSchema = new SingleFieldDeserializationSchema(
+			testSpec.type.getLogicalType(), TypeInformation.of(RowData.class));
+		SingleFieldSerializationSchema serializationSchema = new SingleFieldSerializationSchema(
+			testSpec.type.getLogicalType());
+		deserializationSchema.open(mock(DeserializationSchema.InitializationContext.class));
+		serializationSchema.open(mock(SerializationSchema.InitializationContext.class));
+
+		Row row = Row.of(testSpec.value);
+		DataStructureConverter<Object, Object> converter = DataStructureConverters.getConverter(
+			ROW(FIELD("single", testSpec.type)));
+		RowData originalRowData = (RowData) converter.toInternal(row);
+
+		byte[] serializedBytes = serializationSchema.serialize(originalRowData);
+		RowData deserializeRowData = deserializationSchema.deserialize(serializedBytes);
+
+		Row actual = (Row) converter.toExternal(deserializeRowData);
+		assertEquals(row, actual);
+	}
+
+	private static List<TestSpec> testData = Arrays.asList(

Review comment:
       Use a JUnit `@Parameterized` instead?

##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/singlefield/SingleFieldDeserializationSchema.java
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.singlefield;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.BooleanSerializer;
+import org.apache.flink.api.common.typeutils.base.ByteSerializer;
+import org.apache.flink.api.common.typeutils.base.DoubleSerializer;
+import org.apache.flink.api.common.typeutils.base.FloatSerializer;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeutils.base.ShortSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Deserialization schema from single field data to Flink Table/SQL internal data structure {@link RowData}.
+ */
+@Internal
+public class SingleFieldDeserializationSchema implements DeserializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType fieldType;
+	private final DeserializationRuntimeConverter converter;
+	private TypeInformation<RowData> typeInfo;

Review comment:
       nit: call this`producedTypeInfo` for consistency
   
   final?

##########
File path: docs/dev/table/connectors/formats/single-field.md
##########
@@ -0,0 +1,151 @@
+---
+title: "Single Field Format"
+nav-title: SingleField
+nav-parent_id: sql-formats
+nav-pos: 7
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<span class="label label-info">Format: Serialization Schema</span>
+<span class="label label-info">Format: Deserialization Schema</span>
+
+* This will be replaced by the TOC
+{:toc}
+
+The SingleField format allows to read and write data where the data contains only a single field, and that field is not wrapped within a JSON object, or an Avro record.
+
+Currently, the SingleField format supports `String`, `byte[]` and primitive type.
+
+Note: this format encodes `null` values as `null` `byte[]`. This may have limitation when used in `upsert-kafka`, because `upsert-kafka` treats `null` values as a tombstone message (DELETE on the key). Therefore, we recommend avoiding using `upsert-kafka` connector and `single-field` format if the field can have a `null` value.
+
+Example
+----------------
+
+For example, you may have following raw log data in Kafka and want to read and analyse such data using Flink SQL.
+
+```
+47.29.201.179 - - [28/Feb/2019:13:17:10 +0000] "GET /?p=1 HTTP/2.0" 200 5316 "https://domain.com/?p=1" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"
+```
+
+The following creates a table where it reads from (and writes to) the underlying Kafka topic as an anonymous string value by using `single-field` format:
+
+<div class="codetabs" markdown="1">
+<div data-lang="SQL" markdown="1">
+{% highlight sql %}
+CREATE TABLE nginx_log (
+  log STRING
+) WITH (
+  'connector' = 'kafka',
+  'topic' = 'nginx_log',
+  'properties.bootstrap.servers' = 'localhost:9092',
+  'properties.group.id' = 'testGroup',
+  'format' = 'single-field'
+)
+{% endhighlight %}
+</div>
+</div>
+
+Then you can read out the raw data as a pure string, and split it into multiple fields using user-defined-function for further analysing, e.g. `my_split` in the example.
+
+<div class="codetabs" markdown="1">
+<div data-lang="SQL" markdown="1">
+{% highlight sql %}
+SELECT t.hostname, t.datetime, t.url, t.browser, ...
+FROM(
+  SELECT my_split(log) as t FROM nginx_log
+);
+{% endhighlight %}
+</div>
+</div>
+
+In contrast, you can also write a single field of STRING type into Kafka topic as an anonymous string value.
+
+Format Options
+----------------
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left" style="width: 25%">Option</th>
+        <th class="text-center" style="width: 8%">Required</th>
+        <th class="text-center" style="width: 7%">Default</th>
+        <th class="text-center" style="width: 10%">Type</th>
+        <th class="text-center" style="width: 50%">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+    <tr>
+      <td><h5>format</h5></td>
+      <td>required</td>
+      <td style="word-wrap: break-word;">(none)</td>
+      <td>String</td>
+      <td>Specify what format to use, here should be 'single-field'.</td>
+    </tr>
+    </tbody>
+</table>
+
+Data Type Mapping
+----------------
+
+The table below details the SQL types the format supports, including details of the serializer and deserializer class for encoding and decoding.
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left">Flink SQL type</th>
+        <th class="text-left">Value</th>
+      </tr>
+    </thead>
+    <tbody>
+    <tr>
+      <td><code>CHAR / VARCHAR / STRING</code></td>
+      <td>A UTF-8 encoded text string</td>

Review comment:
       We should also allow to specify a different charset. This should be configurable.

##########
File path: docs/dev/table/connectors/formats/single-field.md
##########
@@ -0,0 +1,151 @@
+---
+title: "Single Field Format"
+nav-title: SingleField
+nav-parent_id: sql-formats
+nav-pos: 7
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<span class="label label-info">Format: Serialization Schema</span>
+<span class="label label-info">Format: Deserialization Schema</span>
+
+* This will be replaced by the TOC
+{:toc}
+
+The SingleField format allows to read and write data where the data contains only a single field, and that field is not wrapped within a JSON object, or an Avro record.
+
+Currently, the SingleField format supports `String`, `byte[]` and primitive type.
+
+Note: this format encodes `null` values as `null` `byte[]`. This may have limitation when used in `upsert-kafka`, because `upsert-kafka` treats `null` values as a tombstone message (DELETE on the key). Therefore, we recommend avoiding using `upsert-kafka` connector and `single-field` format if the field can have a `null` value.
+
+Example
+----------------
+
+For example, you may have following raw log data in Kafka and want to read and analyse such data using Flink SQL.
+
+```
+47.29.201.179 - - [28/Feb/2019:13:17:10 +0000] "GET /?p=1 HTTP/2.0" 200 5316 "https://domain.com/?p=1" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"
+```
+
+The following creates a table where it reads from (and writes to) the underlying Kafka topic as an anonymous string value by using `single-field` format:
+
+<div class="codetabs" markdown="1">
+<div data-lang="SQL" markdown="1">
+{% highlight sql %}
+CREATE TABLE nginx_log (
+  log STRING
+) WITH (
+  'connector' = 'kafka',
+  'topic' = 'nginx_log',
+  'properties.bootstrap.servers' = 'localhost:9092',
+  'properties.group.id' = 'testGroup',
+  'format' = 'single-field'
+)
+{% endhighlight %}
+</div>
+</div>
+
+Then you can read out the raw data as a pure string, and split it into multiple fields using user-defined-function for further analysing, e.g. `my_split` in the example.
+
+<div class="codetabs" markdown="1">
+<div data-lang="SQL" markdown="1">
+{% highlight sql %}
+SELECT t.hostname, t.datetime, t.url, t.browser, ...
+FROM(
+  SELECT my_split(log) as t FROM nginx_log
+);
+{% endhighlight %}
+</div>
+</div>
+
+In contrast, you can also write a single field of STRING type into Kafka topic as an anonymous string value.
+
+Format Options
+----------------
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left" style="width: 25%">Option</th>
+        <th class="text-center" style="width: 8%">Required</th>
+        <th class="text-center" style="width: 7%">Default</th>
+        <th class="text-center" style="width: 10%">Type</th>
+        <th class="text-center" style="width: 50%">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+    <tr>
+      <td><h5>format</h5></td>

Review comment:
       We should also allow to configure the endianness. Every format that encodes to bytes requires this information.

##########
File path: docs/dev/table/connectors/formats/single-field.md
##########
@@ -0,0 +1,151 @@
+---
+title: "Single Field Format"
+nav-title: SingleField
+nav-parent_id: sql-formats
+nav-pos: 7
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<span class="label label-info">Format: Serialization Schema</span>
+<span class="label label-info">Format: Deserialization Schema</span>
+
+* This will be replaced by the TOC
+{:toc}
+
+The SingleField format allows to read and write data where the data contains only a single field, and that field is not wrapped within a JSON object, or an Avro record.
+
+Currently, the SingleField format supports `String`, `byte[]` and primitive type.
+
+Note: this format encodes `null` values as `null` `byte[]`. This may have limitation when used in `upsert-kafka`, because `upsert-kafka` treats `null` values as a tombstone message (DELETE on the key). Therefore, we recommend avoiding using `upsert-kafka` connector and `single-field` format if the field can have a `null` value.

Review comment:
       I don't understand this comment:
   ```
   encodes `null` values as `null` `byte[]`
   ```
   
   Do you mean:
   ```
   encodes `null` values as `null` of `byte[]` type
   ```

##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/singlefield/SingleFieldDeserializationSchema.java
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.singlefield;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.BooleanSerializer;
+import org.apache.flink.api.common.typeutils.base.ByteSerializer;
+import org.apache.flink.api.common.typeutils.base.DoubleSerializer;
+import org.apache.flink.api.common.typeutils.base.FloatSerializer;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeutils.base.ShortSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Deserialization schema from single field data to Flink Table/SQL internal data structure {@link RowData}.
+ */
+@Internal
+public class SingleFieldDeserializationSchema implements DeserializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType fieldType;

Review comment:
       nit: call this `deserializedType`?

##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/singlefield/SingleFieldDeserializationSchema.java
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.singlefield;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.BooleanSerializer;
+import org.apache.flink.api.common.typeutils.base.ByteSerializer;
+import org.apache.flink.api.common.typeutils.base.DoubleSerializer;
+import org.apache.flink.api.common.typeutils.base.FloatSerializer;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeutils.base.ShortSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Deserialization schema from single field data to Flink Table/SQL internal data structure {@link RowData}.
+ */
+@Internal
+public class SingleFieldDeserializationSchema implements DeserializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType fieldType;
+	private final DeserializationRuntimeConverter converter;
+	private TypeInformation<RowData> typeInfo;
+
+	private transient GenericRowData reuse;
+
+	public SingleFieldDeserializationSchema(
+			LogicalType fieldType,
+			TypeInformation<RowData> resultTypeInfo) {
+		this.fieldType = checkNotNull(fieldType);
+		this.typeInfo = checkNotNull(resultTypeInfo);
+		this.converter = createConverter(fieldType);
+	}
+
+	@Override
+	public void open(InitializationContext context) throws Exception {
+		reuse = new GenericRowData(1);
+	}
+
+	@Override
+	public RowData deserialize(byte[] message) throws IOException {
+		reuse.setField(0, converter.convert(message));
+		return reuse;
+	}
+
+	@Override
+	public boolean isEndOfStream(RowData nextElement) {
+		return false;
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return typeInfo;
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		SingleFieldDeserializationSchema that = (SingleFieldDeserializationSchema) o;
+		return typeInfo.equals(that.typeInfo) && fieldType.equals(that.fieldType);
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(typeInfo, fieldType);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert byte[] to internal data structure object.
+	 */
+	@FunctionalInterface
+	private interface DeserializationRuntimeConverter extends Serializable {
+		Object convert(byte[] data) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private static DeserializationRuntimeConverter createConverter(LogicalType type) {
+		switch (type.getTypeRoot()) {
+			case CHAR:
+			case VARCHAR:
+				return data -> {
+					// put null check logic in the lambda instead of wrapping outside
+					// to avoid virtual method invoking.
+					if (data == null) {
+						return null;
+					}
+					return StringData.fromBytes(data);
+				};
+			case VARBINARY:
+			case BINARY:
+				return data -> data;
+			case TINYINT:
+				return createConverterUsingSerializer(ByteSerializer.INSTANCE);
+			case SMALLINT:
+				return createConverterUsingSerializer(ShortSerializer.INSTANCE);
+			case INTEGER:
+				return createConverterUsingSerializer(IntSerializer.INSTANCE);
+			case BIGINT:
+				return createConverterUsingSerializer(LongSerializer.INSTANCE);
+			case FLOAT:
+				return createConverterUsingSerializer(FloatSerializer.INSTANCE);
+			case DOUBLE:
+				return createConverterUsingSerializer(DoubleSerializer.INSTANCE);
+			case BOOLEAN:
+				return createConverterUsingSerializer(BooleanSerializer.INSTANCE);
+			default:
+				throw new UnsupportedOperationException("'single-format' currently doesn't support type: " + type);
+		}
+	}
+
+	private static DeserializationRuntimeConverter createConverterUsingSerializer(
+			final TypeSerializer<?> serializer) {
+		return new DelegatingDeserializationConverter(serializer);
+	}
+
+	private static final class DelegatingDeserializationConverter
+		implements DeserializationRuntimeConverter {
+		private static final long serialVersionUID = 1L;
+		private final DataInputDeserializer source = new DataInputDeserializer();
+		private final TypeSerializer<?> serializer;
+
+		protected DelegatingDeserializationConverter(TypeSerializer<?> serializer) {
+			this.serializer = serializer;
+		}
+
+		@Override
+		public Object convert(byte[] data) throws IOException {
+			if (data == null) {
+				return null;
+			}
+			source.setBuffer(data);

Review comment:
       Would we have a performance benefit of implementing the deserialization logic ourselves? instead of setting fields and delegating to other classes.

##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/singlefield/SingleFieldSerializationSchema.java
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.singlefield;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.BooleanSerializer;
+import org.apache.flink.api.common.typeutils.base.ByteSerializer;
+import org.apache.flink.api.common.typeutils.base.DoubleSerializer;
+import org.apache.flink.api.common.typeutils.base.FloatSerializer;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeutils.base.ShortSerializer;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.RowData.FieldGetter;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Objects;
+
+/**
+ * Serialization schema that serializes an {@link RowData} object into a single field bytes.
+ */
+@Internal
+public class SingleFieldSerializationSchema implements SerializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType fieldType;
+
+	private final SerializationRuntimeConverter converter;
+
+	private final FieldGetter fieldGetter;
+
+	public SingleFieldSerializationSchema(LogicalType fieldType) {
+		this.fieldType = fieldType;
+		this.fieldGetter = RowData.createFieldGetter(fieldType, 0);
+		this.converter = createConverter(fieldType);
+	}
+
+	@Override
+	public byte[] serialize(RowData row) {
+		try {
+			return converter.convert(fieldGetter.getFieldOrNull(row));
+		} catch (IOException e) {
+			throw new RuntimeException("Could not serialize row '" + row + "'. ", e);
+		}
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		SingleFieldSerializationSchema that = (SingleFieldSerializationSchema) o;
+		return fieldType.equals(that.fieldType);
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(fieldType);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert an object of internal data structure to byte[].
+	 */
+	@FunctionalInterface
+	private interface SerializationRuntimeConverter extends Serializable {
+		byte[] convert(Object value) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private SerializationRuntimeConverter createConverter(LogicalType type) {
+		switch (type.getTypeRoot()) {
+			case CHAR:
+			case VARCHAR:
+				return value -> {
+					// put null check logic in the lambda instead of wrapping outside
+					// to avoid virtual method invoking.
+					if (value == null) {
+						return null;
+					}
+					return ((StringData) value).toBytes();

Review comment:
       Is this Flink's representation of strings in bytes? I could imagine that most people assume `"string".getBytes()` semantics. Is the string length included in the bytes are as well?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] twalthr commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518805037



##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/raw/RawFormatFactory.java
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.raw;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory.Context;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Format factory for raw format which allows to read and write raw (byte based) values as a single column.
+ */
+public class RawFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+
+	public static final String IDENTIFIER = "raw";
+	private static final String BIG_ENDIAN = "big-endian";
+	private static final String LITTLE_ENDIAN = "little-endian";
+
+	public static final ConfigOption<String> ENDIANNESS = ConfigOptions
+		.key("endianness")
+		.stringType()
+		.defaultValue(BIG_ENDIAN)
+		.withDescription("Defines the endianness of the bytes of the numeric value.");

Review comment:
       nit: `Defines the endianness for bytes of numeric values.`

##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/raw/RawFormatDeserializationSchema.java
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.raw;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.HeapMemorySegment;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RawValueData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RawType;
+import org.apache.flink.types.DeserializationException;
+import org.apache.flink.util.function.FunctionUtils;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+import java.util.function.Consumer;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Deserialization schema from raw (byte based) value to Flink Table/SQL internal data structure {@link RowData}.
+ */
+@Internal
+public class RawFormatDeserializationSchema implements DeserializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType deserializedType;
+
+	private final DeserializationRuntimeConverter converter;
+
+	private final TypeInformation<RowData> producedTypeInfo;
+
+	private final String charsetName;
+
+	private final boolean isBigEndian;
+
+	private transient GenericRowData reuse;
+
+	public RawFormatDeserializationSchema(
+			LogicalType deserializedType,
+			TypeInformation<RowData> producedTypeInfo,
+			String charsetName,
+			boolean isBigEndian) {
+		this.deserializedType = checkNotNull(deserializedType);
+		this.producedTypeInfo = checkNotNull(producedTypeInfo);
+		this.converter = createConverter(deserializedType, charsetName, isBigEndian);
+		this.charsetName = charsetName;
+		this.isBigEndian = isBigEndian;
+	}
+
+	@Override
+	public void open(InitializationContext context) throws Exception {
+		reuse = new GenericRowData(1);
+		converter.open();
+	}
+
+	@Override
+	public RowData deserialize(byte[] message) throws IOException {
+		reuse.setField(0, converter.convert(message));
+		return reuse;
+	}
+
+	@Override
+	public boolean isEndOfStream(RowData nextElement) {
+		return false;
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return producedTypeInfo;
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		RawFormatDeserializationSchema that = (RawFormatDeserializationSchema) o;
+		return producedTypeInfo.equals(that.producedTypeInfo) &&
+			deserializedType.equals(that.deserializedType) &&
+			charsetName.equals(that.charsetName) &&
+			isBigEndian == that.isBigEndian;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(producedTypeInfo, deserializedType, charsetName, isBigEndian);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert byte[] to internal data structure object.
+	 */
+	@FunctionalInterface
+	private interface DeserializationRuntimeConverter extends Serializable {
+
+		default void open() {}
+
+		Object convert(byte[] data) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private static DeserializationRuntimeConverter createConverter(
+			LogicalType type, String charsetName, boolean isBigEndian) {
+		final DeserializationRuntimeConverter converter = createNotNullConverter(type, charsetName, isBigEndian);
+		final Consumer<byte[]> validator = createDataLengthValidator(type);
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+
+			@Override
+			public void open() {
+				converter.open();
+			}
+
+			@Override
+			public Object convert(byte[] data) throws IOException {
+				if (data == null) {
+					return null;
+				}
+				validator.accept(data);
+				return converter.convert(data);
+			}
+		};
+	}
+
+	private static DeserializationRuntimeConverter createNotNullConverter(
+			LogicalType type, String charsetName, boolean isBigEndian) {
+		switch (type.getTypeRoot()) {
+			case CHAR:
+			case VARCHAR:
+				return createStringConverter(charsetName);
+
+			case VARBINARY:
+			case BINARY:
+				return data -> data;
+
+			case RAW:
+				return createRawValueConverter((RawType<?>) type);
+
+			case BOOLEAN:
+				return data -> data[0] != 0;
+
+			case TINYINT:
+				return data -> data[0];
+
+			case SMALLINT:
+				return createEndiannessAwareConverter(
+					isBigEndian,
+					segment -> segment.getShortBigEndian(0),
+					segment -> segment.getShortLittleEndian(0)
+				);
+
+			case INTEGER:
+				return createEndiannessAwareConverter(
+					isBigEndian,
+					segment -> segment.getIntBigEndian(0),
+					segment -> segment.getIntLittleEndian(0)
+				);
+
+			case BIGINT:
+				return createEndiannessAwareConverter(
+					isBigEndian,
+					segment -> segment.getLongBigEndian(0),
+					segment -> segment.getLongLittleEndian(0)
+				);
+
+			case FLOAT:
+				return createEndiannessAwareConverter(
+					isBigEndian,
+					segment -> segment.getFloatBigEndian(0),
+					segment -> segment.getFloatLittleEndian(0)
+				);
+
+			case DOUBLE:
+				return createEndiannessAwareConverter(
+					isBigEndian,
+					segment -> segment.getDoubleBigEndian(0),
+					segment -> segment.getDoubleLittleEndian(0)
+				);
+
+			default:
+				throw new UnsupportedOperationException("'raw' format currently doesn't support type: " + type);
+		}
+	}
+
+	private static DeserializationRuntimeConverter createStringConverter(final String charsetName) {
+		// this also checks the charsetName is valid
+		Charset charset = Charset.forName(charsetName);
+		if (charset == StandardCharsets.UTF_8) {
+			// avoid UTF-8 decoding if the given charset is UTF-8
+			// because the underlying bytes of StringData is in UTF-8 encoding
+			return StringData::fromBytes;
+		}
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+			private transient Charset charset;
+			@Override
+			public void open() {
+				charset = Charset.forName(charsetName);
+			}
+
+			@Override
+			public Object convert(byte[] data) {
+				String str = new String(data, charset);
+				return StringData.fromString(str);
+			}
+		};
+	}
+
+	private static DeserializationRuntimeConverter createRawValueConverter(RawType<?> rawType) {
+		final TypeSerializer<?> serializer = rawType.getTypeSerializer();
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+			private final DataInputDeserializer source = new DataInputDeserializer();
+			@Override
+			public Object convert(byte[] data) throws IOException {
+				source.setBuffer(data);
+				return RawValueData.fromObject(serializer.deserialize(source));

Review comment:
       can't we use `RawValueData.fromBytes` here and do lazy deserialization?

##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/raw/RawFormatFactory.java
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.raw;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory.Context;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Format factory for raw format which allows to read and write raw (byte based) values as a single column.
+ */
+public class RawFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+
+	public static final String IDENTIFIER = "raw";
+	private static final String BIG_ENDIAN = "big-endian";
+	private static final String LITTLE_ENDIAN = "little-endian";
+
+	public static final ConfigOption<String> ENDIANNESS = ConfigOptions
+		.key("endianness")
+		.stringType()
+		.defaultValue(BIG_ENDIAN)
+		.withDescription("Defines the endianness of the bytes of the numeric value.");
+
+	public static final ConfigOption<String> CHARSET = ConfigOptions
+		.key("charset")
+		.stringType()
+		.defaultValue(StandardCharsets.UTF_8.displayName())
+		.withDescription("Defines the string charset.");
+
+	@Override
+	public String factoryIdentifier() {
+		return IDENTIFIER;
+	}
+
+	@Override
+	public Set<ConfigOption<?>> requiredOptions() {
+		return Collections.emptySet();
+	}
+
+	@Override
+	public Set<ConfigOption<?>> optionalOptions() {
+		Set<ConfigOption<?>> options = new HashSet<>();
+		options.add(ENDIANNESS);
+		options.add(CHARSET);
+		return options;
+	}
+
+	@Override
+	public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(
+			Context context,
+			ReadableConfig formatOptions) {
+		FactoryUtil.validateFactoryOptions(this, formatOptions);
+		final String charsetName = validateAndGetCharsetName(formatOptions);
+		final boolean isBigEndian = isBigEndian(formatOptions);
+
+		return new DecodingFormat<DeserializationSchema<RowData>>() {
+			@Override
+			public DeserializationSchema<RowData> createRuntimeDecoder(
+					DynamicTableSource.Context context,
+					DataType producedDataType) {
+				final RowType physicalRowType = (RowType) producedDataType.getLogicalType();
+				final LogicalType fieldType = validateAndExtractSingleField(physicalRowType);
+				final TypeInformation<RowData> producedTypeInfo = context.createTypeInformation(producedDataType);
+				return new RawFormatDeserializationSchema(fieldType, producedTypeInfo, charsetName, isBigEndian);
+			}
+
+			@Override
+			public ChangelogMode getChangelogMode() {
+				return ChangelogMode.insertOnly();
+			}
+		};
+	}
+
+	@Override
+	public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(
+			Context context, ReadableConfig formatOptions) {
+		FactoryUtil.validateFactoryOptions(this, formatOptions);
+		final String charsetName = validateAndGetCharsetName(formatOptions);
+		final boolean isBigEndian = isBigEndian(formatOptions);
+
+		return new EncodingFormat<SerializationSchema<RowData>>() {
+			@Override
+			public SerializationSchema<RowData> createRuntimeEncoder(
+					DynamicTableSink.Context context,
+					DataType consumedDataType) {
+				final RowType physicalRowType = (RowType) consumedDataType.getLogicalType();
+				final LogicalType fieldType = validateAndExtractSingleField(physicalRowType);
+				return new RawFormatSerializationSchema(fieldType, charsetName, isBigEndian);
+			}
+
+			@Override
+			public ChangelogMode getChangelogMode() {
+				return ChangelogMode.insertOnly();
+			}
+		};
+	}
+
+	// ------------------------------------------------------------------------------------------
+
+	/**
+	 * Validates and extract the single field type from the given physical row schema.
+	 */
+	private static LogicalType validateAndExtractSingleField(RowType physicalRowType) {
+		if (physicalRowType.getFieldCount() != 1) {

Review comment:
       shall we perform real type validation here? e.g. TIMESTAMP or NULL would still pass

##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/raw/RawFormatDeserializationSchema.java
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.raw;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.HeapMemorySegment;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RawValueData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RawType;
+import org.apache.flink.types.DeserializationException;
+import org.apache.flink.util.function.FunctionUtils;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+import java.util.function.Consumer;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Deserialization schema from raw (byte based) value to Flink Table/SQL internal data structure {@link RowData}.
+ */
+@Internal
+public class RawFormatDeserializationSchema implements DeserializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType deserializedType;
+
+	private final DeserializationRuntimeConverter converter;
+
+	private final TypeInformation<RowData> producedTypeInfo;
+
+	private final String charsetName;
+
+	private final boolean isBigEndian;
+
+	private transient GenericRowData reuse;
+
+	public RawFormatDeserializationSchema(
+			LogicalType deserializedType,
+			TypeInformation<RowData> producedTypeInfo,
+			String charsetName,
+			boolean isBigEndian) {
+		this.deserializedType = checkNotNull(deserializedType);
+		this.producedTypeInfo = checkNotNull(producedTypeInfo);
+		this.converter = createConverter(deserializedType, charsetName, isBigEndian);
+		this.charsetName = charsetName;
+		this.isBigEndian = isBigEndian;
+	}
+
+	@Override
+	public void open(InitializationContext context) throws Exception {
+		reuse = new GenericRowData(1);
+		converter.open();
+	}
+
+	@Override
+	public RowData deserialize(byte[] message) throws IOException {
+		reuse.setField(0, converter.convert(message));
+		return reuse;
+	}
+
+	@Override
+	public boolean isEndOfStream(RowData nextElement) {
+		return false;
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return producedTypeInfo;
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		RawFormatDeserializationSchema that = (RawFormatDeserializationSchema) o;
+		return producedTypeInfo.equals(that.producedTypeInfo) &&
+			deserializedType.equals(that.deserializedType) &&
+			charsetName.equals(that.charsetName) &&
+			isBigEndian == that.isBigEndian;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(producedTypeInfo, deserializedType, charsetName, isBigEndian);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert byte[] to internal data structure object.
+	 */
+	@FunctionalInterface
+	private interface DeserializationRuntimeConverter extends Serializable {
+
+		default void open() {}
+
+		Object convert(byte[] data) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private static DeserializationRuntimeConverter createConverter(
+			LogicalType type, String charsetName, boolean isBigEndian) {
+		final DeserializationRuntimeConverter converter = createNotNullConverter(type, charsetName, isBigEndian);
+		final Consumer<byte[]> validator = createDataLengthValidator(type);
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+
+			@Override
+			public void open() {
+				converter.open();
+			}
+
+			@Override
+			public Object convert(byte[] data) throws IOException {
+				if (data == null) {

Review comment:
       I think data is never null at this location.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164",
       "triggerID" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175",
       "triggerID" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13cf67f4b76c0d67346e760d23d2f9b7c96be5f0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9211",
       "triggerID" : "13cf67f4b76c0d67346e760d23d2f9b7c96be5f0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fe7ef3f93e33dbe1eae8965707f83432e1e9859d UNKNOWN
   * 13cf67f4b76c0d67346e760d23d2f9b7c96be5f0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9211) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] twalthr commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518138892



##########
File path: docs/dev/table/connectors/formats/single-field.md
##########
@@ -0,0 +1,151 @@
+---
+title: "Single Field Format"
+nav-title: SingleField
+nav-parent_id: sql-formats
+nav-pos: 7
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<span class="label label-info">Format: Serialization Schema</span>
+<span class="label label-info">Format: Deserialization Schema</span>
+
+* This will be replaced by the TOC
+{:toc}
+
+The SingleField format allows to read and write data where the data contains only a single field, and that field is not wrapped within a JSON object, or an Avro record.
+
+Currently, the SingleField format supports `String`, `byte[]` and primitive type.
+
+Note: this format encodes `null` values as `null` `byte[]`. This may have limitation when used in `upsert-kafka`, because `upsert-kafka` treats `null` values as a tombstone message (DELETE on the key). Therefore, we recommend avoiding using `upsert-kafka` connector and `single-field` format if the field can have a `null` value.
+
+Example
+----------------
+
+For example, you may have following raw log data in Kafka and want to read and analyse such data using Flink SQL.
+
+```
+47.29.201.179 - - [28/Feb/2019:13:17:10 +0000] "GET /?p=1 HTTP/2.0" 200 5316 "https://domain.com/?p=1" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"
+```
+
+The following creates a table where it reads from (and writes to) the underlying Kafka topic as an anonymous string value by using `single-field` format:
+
+<div class="codetabs" markdown="1">
+<div data-lang="SQL" markdown="1">
+{% highlight sql %}
+CREATE TABLE nginx_log (
+  log STRING
+) WITH (
+  'connector' = 'kafka',
+  'topic' = 'nginx_log',
+  'properties.bootstrap.servers' = 'localhost:9092',
+  'properties.group.id' = 'testGroup',
+  'format' = 'single-field'
+)
+{% endhighlight %}
+</div>
+</div>
+
+Then you can read out the raw data as a pure string, and split it into multiple fields using user-defined-function for further analysing, e.g. `my_split` in the example.
+
+<div class="codetabs" markdown="1">
+<div data-lang="SQL" markdown="1">
+{% highlight sql %}
+SELECT t.hostname, t.datetime, t.url, t.browser, ...
+FROM(
+  SELECT my_split(log) as t FROM nginx_log
+);
+{% endhighlight %}
+</div>
+</div>
+
+In contrast, you can also write a single field of STRING type into Kafka topic as an anonymous string value.
+
+Format Options
+----------------
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left" style="width: 25%">Option</th>
+        <th class="text-center" style="width: 8%">Required</th>
+        <th class="text-center" style="width: 7%">Default</th>
+        <th class="text-center" style="width: 10%">Type</th>
+        <th class="text-center" style="width: 50%">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+    <tr>
+      <td><h5>format</h5></td>
+      <td>required</td>
+      <td style="word-wrap: break-word;">(none)</td>
+      <td>String</td>
+      <td>Specify what format to use, here should be 'single-field'.</td>
+    </tr>
+    </tbody>
+</table>
+
+Data Type Mapping
+----------------
+
+The table below details the SQL types the format supports, including details of the serializer and deserializer class for encoding and decoding.
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left">Flink SQL type</th>
+        <th class="text-left">Value</th>
+      </tr>
+    </thead>
+    <tbody>
+    <tr>
+      <td><code>CHAR / VARCHAR / STRING</code></td>
+      <td>A UTF-8 encoded text string</td>
+    </tr>
+    <tr>
+      <td><code>BOOLEAN</code></td>
+      <td>A single byte to indicate boolean value, 0 means false, 1 means true.</td>
+    </tr>
+    <tr>
+      <td><code>TINYINT</code></td>
+      <td>A 8-bit signed number</td>
+    </tr>
+    <tr>
+      <td><code>SMALLINT</code></td>
+      <td>A 16-bit signed number</td>
+    </tr>
+    <tr>
+      <td><code>INT</code></td>
+      <td>A 32-bit signed integer</td>
+    </tr>
+    <tr>
+      <td><code>BIGINT</code></td>
+      <td>A 64-bit signed integer</td>
+    </tr>
+    <tr>
+      <td><code>FLOAT</code></td>
+      <td>A 32-bit floating point number</td>
+    </tr>
+    <tr>

Review comment:
       True, it is difficult to declare the RAW type in DDL. But this is an orthogonal topic that might be solved soon. We should support all data types for our implementations otherwise we need to multiple iterations to get the connectors/formats into a usable state. IMHO adding the RAW here is not a big overhead.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] twalthr commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518851869



##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/raw/RawFormatDeserializationSchema.java
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.raw;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.HeapMemorySegment;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RawValueData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RawType;
+import org.apache.flink.types.DeserializationException;
+import org.apache.flink.util.function.FunctionUtils;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+import java.util.function.Consumer;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Deserialization schema from raw (byte based) value to Flink Table/SQL internal data structure {@link RowData}.
+ */
+@Internal
+public class RawFormatDeserializationSchema implements DeserializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType deserializedType;
+
+	private final DeserializationRuntimeConverter converter;
+
+	private final TypeInformation<RowData> producedTypeInfo;
+
+	private final String charsetName;
+
+	private final boolean isBigEndian;
+
+	private transient GenericRowData reuse;
+
+	public RawFormatDeserializationSchema(
+			LogicalType deserializedType,
+			TypeInformation<RowData> producedTypeInfo,
+			String charsetName,
+			boolean isBigEndian) {
+		this.deserializedType = checkNotNull(deserializedType);
+		this.producedTypeInfo = checkNotNull(producedTypeInfo);
+		this.converter = createConverter(deserializedType, charsetName, isBigEndian);
+		this.charsetName = charsetName;
+		this.isBigEndian = isBigEndian;
+	}
+
+	@Override
+	public void open(InitializationContext context) throws Exception {
+		reuse = new GenericRowData(1);
+		converter.open();
+	}
+
+	@Override
+	public RowData deserialize(byte[] message) throws IOException {
+		reuse.setField(0, converter.convert(message));
+		return reuse;
+	}
+
+	@Override
+	public boolean isEndOfStream(RowData nextElement) {
+		return false;
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return producedTypeInfo;
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		RawFormatDeserializationSchema that = (RawFormatDeserializationSchema) o;
+		return producedTypeInfo.equals(that.producedTypeInfo) &&
+			deserializedType.equals(that.deserializedType) &&
+			charsetName.equals(that.charsetName) &&
+			isBigEndian == that.isBigEndian;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(producedTypeInfo, deserializedType, charsetName, isBigEndian);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert byte[] to internal data structure object.
+	 */
+	@FunctionalInterface
+	private interface DeserializationRuntimeConverter extends Serializable {
+
+		default void open() {}
+
+		Object convert(byte[] data) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private static DeserializationRuntimeConverter createConverter(
+			LogicalType type, String charsetName, boolean isBigEndian) {
+		final DeserializationRuntimeConverter converter = createNotNullConverter(type, charsetName, isBigEndian);
+		final Consumer<byte[]> validator = createDataLengthValidator(type);
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+
+			@Override
+			public void open() {
+				converter.open();
+			}
+
+			@Override
+			public Object convert(byte[] data) throws IOException {
+				if (data == null) {

Review comment:
       at other locations we also don't check that:
   `JsonRowDataDeserializationSchema#deserialize` or `CsvRowDataDeserializationSchema#deserialize`
   
   the JavaDocs of `DeserializationSchema` don't mention that either
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wuchong commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518136335



##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/singlefield/SingleFieldSerializationSchema.java
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.singlefield;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.BooleanSerializer;
+import org.apache.flink.api.common.typeutils.base.ByteSerializer;
+import org.apache.flink.api.common.typeutils.base.DoubleSerializer;
+import org.apache.flink.api.common.typeutils.base.FloatSerializer;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeutils.base.ShortSerializer;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.RowData.FieldGetter;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Objects;
+
+/**
+ * Serialization schema that serializes an {@link RowData} object into a single field bytes.
+ */
+@Internal
+public class SingleFieldSerializationSchema implements SerializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType fieldType;
+
+	private final SerializationRuntimeConverter converter;
+
+	private final FieldGetter fieldGetter;
+
+	public SingleFieldSerializationSchema(LogicalType fieldType) {
+		this.fieldType = fieldType;
+		this.fieldGetter = RowData.createFieldGetter(fieldType, 0);
+		this.converter = createConverter(fieldType);
+	}
+
+	@Override
+	public byte[] serialize(RowData row) {
+		try {
+			return converter.convert(fieldGetter.getFieldOrNull(row));
+		} catch (IOException e) {
+			throw new RuntimeException("Could not serialize row '" + row + "'. ", e);
+		}
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		SingleFieldSerializationSchema that = (SingleFieldSerializationSchema) o;
+		return fieldType.equals(that.fieldType);
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(fieldType);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert an object of internal data structure to byte[].
+	 */
+	@FunctionalInterface
+	private interface SerializationRuntimeConverter extends Serializable {
+		byte[] convert(Object value) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private SerializationRuntimeConverter createConverter(LogicalType type) {
+		switch (type.getTypeRoot()) {
+			case CHAR:
+			case VARCHAR:
+				return value -> {
+					// put null check logic in the lambda instead of wrapping outside
+					// to avoid virtual method invoking.
+					if (value == null) {
+						return null;
+					}
+					return ((StringData) value).toBytes();

Review comment:
       I'm sure this is the same semantic of `"string".getBytes()`, becuase we are using `StringData.fromBytes(stringBytes)` to save decoding overhead (`stringBytes` to `String`) in the sources. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wuchong commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "single-field" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518136335



##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/singlefield/SingleFieldSerializationSchema.java
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.singlefield;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.base.BooleanSerializer;
+import org.apache.flink.api.common.typeutils.base.ByteSerializer;
+import org.apache.flink.api.common.typeutils.base.DoubleSerializer;
+import org.apache.flink.api.common.typeutils.base.FloatSerializer;
+import org.apache.flink.api.common.typeutils.base.IntSerializer;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeutils.base.ShortSerializer;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.RowData.FieldGetter;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Objects;
+
+/**
+ * Serialization schema that serializes an {@link RowData} object into a single field bytes.
+ */
+@Internal
+public class SingleFieldSerializationSchema implements SerializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType fieldType;
+
+	private final SerializationRuntimeConverter converter;
+
+	private final FieldGetter fieldGetter;
+
+	public SingleFieldSerializationSchema(LogicalType fieldType) {
+		this.fieldType = fieldType;
+		this.fieldGetter = RowData.createFieldGetter(fieldType, 0);
+		this.converter = createConverter(fieldType);
+	}
+
+	@Override
+	public byte[] serialize(RowData row) {
+		try {
+			return converter.convert(fieldGetter.getFieldOrNull(row));
+		} catch (IOException e) {
+			throw new RuntimeException("Could not serialize row '" + row + "'. ", e);
+		}
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		SingleFieldSerializationSchema that = (SingleFieldSerializationSchema) o;
+		return fieldType.equals(that.fieldType);
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(fieldType);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert an object of internal data structure to byte[].
+	 */
+	@FunctionalInterface
+	private interface SerializationRuntimeConverter extends Serializable {
+		byte[] convert(Object value) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private SerializationRuntimeConverter createConverter(LogicalType type) {
+		switch (type.getTypeRoot()) {
+			case CHAR:
+			case VARCHAR:
+				return value -> {
+					// put null check logic in the lambda instead of wrapping outside
+					// to avoid virtual method invoking.
+					if (value == null) {
+						return null;
+					}
+					return ((StringData) value).toBytes();

Review comment:
       I'm sure this is the same semantic of `"string".getBytes()`, becuase we are using `StringData.fromBytes(stringBytes)` to save decoding overhead (`stringBytes` to `String`) in the sources. 
   
   The string length is not included in the bytes. The length is appended by the `StringDataSerializer`. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wuchong commented on a change in pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #13909:
URL: https://github.com/apache/flink/pull/13909#discussion_r518856386



##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/formats/raw/RawFormatDeserializationSchema.java
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.formats.raw;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.HeapMemorySegment;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RawValueData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RawType;
+import org.apache.flink.types.DeserializationException;
+import org.apache.flink.util.function.FunctionUtils;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+import java.util.function.Consumer;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Deserialization schema from raw (byte based) value to Flink Table/SQL internal data structure {@link RowData}.
+ */
+@Internal
+public class RawFormatDeserializationSchema implements DeserializationSchema<RowData> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final LogicalType deserializedType;
+
+	private final DeserializationRuntimeConverter converter;
+
+	private final TypeInformation<RowData> producedTypeInfo;
+
+	private final String charsetName;
+
+	private final boolean isBigEndian;
+
+	private transient GenericRowData reuse;
+
+	public RawFormatDeserializationSchema(
+			LogicalType deserializedType,
+			TypeInformation<RowData> producedTypeInfo,
+			String charsetName,
+			boolean isBigEndian) {
+		this.deserializedType = checkNotNull(deserializedType);
+		this.producedTypeInfo = checkNotNull(producedTypeInfo);
+		this.converter = createConverter(deserializedType, charsetName, isBigEndian);
+		this.charsetName = charsetName;
+		this.isBigEndian = isBigEndian;
+	}
+
+	@Override
+	public void open(InitializationContext context) throws Exception {
+		reuse = new GenericRowData(1);
+		converter.open();
+	}
+
+	@Override
+	public RowData deserialize(byte[] message) throws IOException {
+		reuse.setField(0, converter.convert(message));
+		return reuse;
+	}
+
+	@Override
+	public boolean isEndOfStream(RowData nextElement) {
+		return false;
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return producedTypeInfo;
+	}
+
+	@Override
+	public boolean equals(Object o) {
+		if (this == o) {
+			return true;
+		}
+		if (o == null || getClass() != o.getClass()) {
+			return false;
+		}
+		RawFormatDeserializationSchema that = (RawFormatDeserializationSchema) o;
+		return producedTypeInfo.equals(that.producedTypeInfo) &&
+			deserializedType.equals(that.deserializedType) &&
+			charsetName.equals(that.charsetName) &&
+			isBigEndian == that.isBigEndian;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(producedTypeInfo, deserializedType, charsetName, isBigEndian);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * Runtime converter that convert byte[] to internal data structure object.
+	 */
+	@FunctionalInterface
+	private interface DeserializationRuntimeConverter extends Serializable {
+
+		default void open() {}
+
+		Object convert(byte[] data) throws IOException;
+	}
+
+	/**
+	 * Creates a runtime converter.
+	 */
+	private static DeserializationRuntimeConverter createConverter(
+			LogicalType type, String charsetName, boolean isBigEndian) {
+		final DeserializationRuntimeConverter converter = createNotNullConverter(type, charsetName, isBigEndian);
+		final Consumer<byte[]> validator = createDataLengthValidator(type);
+
+		return new DeserializationRuntimeConverter() {
+			private static final long serialVersionUID = 1L;
+
+			@Override
+			public void open() {
+				converter.open();
+			}
+
+			@Override
+			public Object convert(byte[] data) throws IOException {
+				if (data == null) {

Review comment:
       I think both JSON and CSV format didn't implement correctly. There will be NPE if encountering tombstone messages. That can be skipped when `ignoreParseErrors` is enabled, but I think the format should handle tombstone messages. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164",
       "triggerID" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175",
       "triggerID" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13cf67f4b76c0d67346e760d23d2f9b7c96be5f0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9211",
       "triggerID" : "13cf67f4b76c0d67346e760d23d2f9b7c96be5f0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fe7ef3f93e33dbe1eae8965707f83432e1e9859d UNKNOWN
   * d60a354989d3e3a3e6cea3fcce6903a35fa72965 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175) 
   * 13cf67f4b76c0d67346e760d23d2f9b7c96be5f0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9211) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164",
       "triggerID" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175",
       "triggerID" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13cf67f4b76c0d67346e760d23d2f9b7c96be5f0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "13cf67f4b76c0d67346e760d23d2f9b7c96be5f0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fe7ef3f93e33dbe1eae8965707f83432e1e9859d UNKNOWN
   * d60a354989d3e3a3e6cea3fcce6903a35fa72965 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9175) 
   * 13cf67f4b76c0d67346e760d23d2f9b7c96be5f0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] wuchong merged pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
wuchong merged pull request #13909:
URL: https://github.com/apache/flink/pull/13909


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13909: [FLINK-14356][table][formats] Introduce "raw" format to (de)serialize message to a single field

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13909:
URL: https://github.com/apache/flink/pull/13909#issuecomment-721281261


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912",
       "triggerID" : "8de5115aa0bbc4bc486dced3d2bc67d34823659f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164",
       "triggerID" : "6051bbecc6f91a54f72d7982b5390e8569189caa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fe7ef3f93e33dbe1eae8965707f83432e1e9859d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d60a354989d3e3a3e6cea3fcce6903a35fa72965",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de5115aa0bbc4bc486dced3d2bc67d34823659f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8912) 
   * 6051bbecc6f91a54f72d7982b5390e8569189caa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9164) 
   * fe7ef3f93e33dbe1eae8965707f83432e1e9859d UNKNOWN
   * d60a354989d3e3a3e6cea3fcce6903a35fa72965 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org