You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "RubyChou (via GitHub)" <gi...@apache.org> on 2024/03/04 04:25:18 UTC
[PR] [FLINK-16627][json]Support ignore null fields when serializing into JSON [flink]
RubyChou opened a new pull request, #24430:
URL: https://github.com/apache/flink/pull/24430
## What is the purpose of the change
*This pull request introduces `json.encode.ignore-null-fields` for JSON formats which indicates whether to ignore null fields when serializing into JSON.*
## Brief change log
- *Add this option in JsonOptions.*
- *Let JsonFormatFactorys and JsonRowDataSerializationSchemas be aware of this option.*
## Verifying this change
This change added tests and can be verified as follows:
- *Added test in `JsonRowDataSerDeSchemaTest` to check whether the result of serialization is expected.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-16627][json]Support ignore null fields when serializing into JSON [flink]
Posted by "RubyChou (via GitHub)" <gi...@apache.org>.
RubyChou commented on PR #24430:
URL: https://github.com/apache/flink/pull/24430#issuecomment-1978316740
hi @libenchao, I've resolved all comments above. Please help reviewing. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-16627][json]Support ignore null fields when serializing into JSON [flink]
Posted by "libenchao (via GitHub)" <gi...@apache.org>.
libenchao commented on code in PR #24430:
URL: https://github.com/apache/flink/pull/24430#discussion_r1512697745
##########
docs/content.zh/docs/connectors/table/formats/json.md:
##########
@@ -135,6 +135,13 @@ Format 参数
<td>Boolean</td>
<td>将所有 DECIMAL 类型的数据保持原状，不使用科学计数法表示。例：<code>0.000000027</code> 默认会表示为 <code>2.7E-8</code>。当此选项设为 true 时，则会表示为 <code>0.000000027</code>。</td>
</tr>
+ <tr>
+ <td><h5>json.encode.ignore-null-fields</h5></td>
+ <td>选填</td>
+ <td style="word-wrap: break-word;">false</td>
Review Comment:
I think this option can be forwarded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-16627][json]Support ignore null fields when serializing into JSON [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #24430:
URL: https://github.com/apache/flink/pull/24430#issuecomment-1975680308
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1df534772b13597af5d87922deb3d09a7cfad9e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1df534772b13597af5d87922deb3d09a7cfad9e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1df534772b13597af5d87922deb3d09a7cfad9e9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-16627][json]Support ignore null fields when serializing into JSON [flink]
Posted by "libenchao (via GitHub)" <gi...@apache.org>.
libenchao closed pull request #24430: [FLINK-16627][json]Support ignore null fields when serializing into JSON
URL: https://github.com/apache/flink/pull/24430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-16627][json]Support ignore null fields when serializing into JSON [flink]
Posted by "libenchao (via GitHub)" <gi...@apache.org>.
libenchao commented on code in PR #24430:
URL: https://github.com/apache/flink/pull/24430#discussion_r1512231078
##########
docs/content.zh/docs/connectors/table/formats/maxwell.md:
##########
@@ -251,6 +251,13 @@ Format Options
<td>Boolean</td>
<td>Encode all decimals as plain numbers instead of possible scientific notations. By default, decimals may be written using scientific notation. For example, <code>0.000000027</code> is encoded as <code>2.7E-8</code> by default, and will be written as <code>0.000000027</code> if set this option to true.</td>
</tr>
+ <tr>
+ <td><h5>maxwell-json.encode.ignore-null-fields</h5></td>
+ <td>optional</td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>Encode only non-null fields. By default, all fields will be written.</td>
Review Comment:
```suggestion
<td>Encode only non-null fields. By default, all fields will be included.</td>
```
##########
docs/content.zh/docs/connectors/table/formats/debezium.md:
##########
@@ -424,6 +424,13 @@ Flink 提供了 `debezium-avro-confluent` 和 `debezium-json` 两种 format 来
<td>Boolean</td>
<td>将所有 DECIMAL 类型的数据保持原状，不使用科学计数法表示。例：<code>0.000000027</code> 默认会表示为 <code>2.7E-8</code>。当此选项设为 true 时，则会表示为 <code>0.000000027</code>。</td>
</tr>
+ <tr>
+ <td><h5>debezium-json.encode.ignore-null-fields</h5></td>
+ <td>选填</td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>仅序列化非Null的列，默认情况下，会序列化所有列无论是否为Null。</td>
Review Comment:
In Chinese document, we usually add spaces between english words, see more here: https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications
```suggestion
<td>仅序列化非 Null 的列，默认情况下，会序列化所有列无论是否为 Null。</td>
```
##########
flink-formats/flink-json/src/test/java/org/apache/flink/formats/json/JsonRowDataSerDeSchemaTest.java:
##########
@@ -626,6 +635,62 @@ void testSerializationDecimalEncode() throws Exception {
assertThat(scientificDecimalResult).isEqualTo(scientificDecimalJson);
}
+ @TestTemplate
+ public void testSerDeMultiRowsWithNullValuesIgnored() throws Exception {
Review Comment:
Since we have migrated to Junit5, `public` is no longer needed, see more here: https://docs.google.com/document/d/1514Wa_aNB9bJUen4xm5uiuXOooOJTtXqS_Jqk9KJitU/edit#heading=h.4iilyfqpoiw2
```suggestion
void testSerDeMultiRowsWithNullValuesIgnored() throws Exception {
```
##########
flink-formats/flink-json/src/test/java/org/apache/flink/formats/json/canal/CanalJsonFormatFactoryTest.java:
##########
@@ -89,6 +90,7 @@ void testUserDefinedOptions() {
options.put("canal-json.map-null-key.mode", "LITERAL");
options.put("canal-json.map-null-key.literal", "nullKey");
options.put("canal-json.encode.decimal-as-plain-number", "true");
+ options.put("canal-json.encode.ignore-null-fields", "true");
Review Comment:
Why are you enable Canal json format to test "encode.ignore-null-fields" while leave others (debezium, maxwell, ogg) out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org