You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "gnehil (via GitHub)" <gi...@apache.org> on 2023/07/21 09:06:43 UTC
[GitHub] [doris-spark-connector] gnehil opened a new pull request, #120: [improvement] json format support json by line mode
gnehil opened a new pull request, #120:
URL: https://github.com/apache/doris-spark-connector/pull/120
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
json format support json by line mode
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris-spark-connector] JNSimba merged pull request #120: [improvement] json format support json by line mode
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #120:
URL: https://github.com/apache/doris-spark-connector/pull/120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris-spark-connector] JNSimba commented on a diff in pull request #120: [improvement] json format support json by line mode
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #120:
URL: https://github.com/apache/doris-spark-connector/pull/120#discussion_r1272923188
##########
spark-doris-connector/src/main/java/org/apache/doris/spark/load/DorisStreamLoad.java:
##########
@@ -105,6 +107,12 @@ public DorisStreamLoad(SparkSettings settings) {
fileType = streamLoadProp.getOrDefault("format", "csv");
if ("csv".equals(fileType)){
FIELD_DELIMITER = EscapeHandler.escapeString(streamLoadProp.getOrDefault("column_separator", "\t"));
+ } else if ("json".equalsIgnoreCase(fileType)) {
+ readJsonByLine = Boolean.parseBoolean(streamLoadProp.getOrDefault("read_json_by_line", "false"));
+ boolean stripOuterArray = Boolean.parseBoolean(streamLoadProp.getOrDefault("strip_outer_array", "false"));
+ if (readJsonByLine == stripOuterArray) {
+ throw new IllegalArgumentException("Only one of options 'read_json_by_line' and 'strip_outer_array' can be set to true");
+ }
Review Comment:
Do need to give a default value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org