You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/01/12 13:27:17 UTC
[GitHub] [flink] wuchong commented on a change in pull request #14604: [FLINK-20861][format/json] Provide an option for serializing DECIMALs in JSON as plain number instead of scientific notation
wuchong commented on a change in pull request #14604:
URL: https://github.com/apache/flink/pull/14604#discussion_r555766824
##########
File path: docs/dev/table/connectors/formats/json.md
##########
@@ -136,6 +136,13 @@ Format Options
<td>String</td>
<td>Specify string literal to replace null key when <code>'json.map-null-key.mode'</code> is LITERAL.</td>
</tr>
+ <tr>
+ <td><h5>json.encode.decimal-as-plain-number</h5></td>
+ <td>optional</td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>Encode all decimals as plain numbers instead of possible scientific notations. e.g. <code>0.000000027</code> is encoded as <code>2.7E-8</code> when this option is set to false.</td>
Review comment:
```suggestion
<td>Encode all decimals as plain numbers instead of possible scientific notations. By default decimals are written using scientific notation. For example, <code>0.000000027</code> is encoded as <code>2.7E-8</code> by default, and will be written as <code>0.000000027</code> if set this option to true.</td>
```
##########
File path: flink-formats/flink-json/src/test/java/org/apache/flink/formats/json/JsonRowDataSerDeSchemaTest.java
##########
@@ -562,6 +579,56 @@ public void testSerializationMapNullKey() throws Exception {
assertEquals(expectResult3, new String(actual3));
}
+ @Test
+ public void testSerializationDecimalEncode() throws Exception {
+ BigDecimal decimal1 = new BigDecimal("123.456789");
+ BigDecimal decimal2 = new BigDecimal("454621864049246170");
+ BigDecimal decimal3 = new BigDecimal("0.000000027");
+
+ ObjectMapper objectMapper = new ObjectMapper();
+ objectMapper.configure(JsonGenerator.Feature.WRITE_BIGDECIMAL_AS_PLAIN, true);
+
+ ObjectNode root = objectMapper.createObjectNode();
+ root.put("decimal1", decimal1);
+ root.put("decimal2", decimal2);
+ root.put("decimal3", decimal3);
+
+ byte[] serializedJson = objectMapper.writeValueAsBytes(root);
+
+ DataType dataType =
+ ROW(
+ FIELD("decimal1", DECIMAL(9, 6)),
+ FIELD("decimal2", DECIMAL(20, 0)),
+ FIELD("decimal3", DECIMAL(11, 9)));
+
+ RowType schema = (RowType) dataType.getLogicalType();
+ TypeInformation<RowData> resultTypeInfo = InternalTypeInfo.of(schema);
+
+ JsonRowDataDeserializationSchema deserializationSchema =
+ new JsonRowDataDeserializationSchema(
+ schema, resultTypeInfo, false, false, TimestampFormat.ISO_8601);
+
+ Row expected = new Row(3);
+ expected.setField(0, decimal1);
+ expected.setField(1, decimal2);
+ expected.setField(2, decimal3);
+
+ RowData rowData = deserializationSchema.deserialize(serializedJson);
Review comment:
We can manually construct the input RowData by using `GenericRowData` and `DecimalData`.
##########
File path: flink-formats/flink-json/src/test/java/org/apache/flink/formats/json/JsonRowDataSerDeSchemaTest.java
##########
@@ -562,6 +579,56 @@ public void testSerializationMapNullKey() throws Exception {
assertEquals(expectResult3, new String(actual3));
}
+ @Test
+ public void testSerializationDecimalEncode() throws Exception {
+ BigDecimal decimal1 = new BigDecimal("123.456789");
+ BigDecimal decimal2 = new BigDecimal("454621864049246170");
+ BigDecimal decimal3 = new BigDecimal("0.000000027");
+
+ ObjectMapper objectMapper = new ObjectMapper();
+ objectMapper.configure(JsonGenerator.Feature.WRITE_BIGDECIMAL_AS_PLAIN, true);
+
+ ObjectNode root = objectMapper.createObjectNode();
+ root.put("decimal1", decimal1);
+ root.put("decimal2", decimal2);
+ root.put("decimal3", decimal3);
+
+ byte[] serializedJson = objectMapper.writeValueAsBytes(root);
Review comment:
I think we can manually construct the serialized JSON string.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org