[GitHub] [incubator-doris] caiconghui commented on a change in pull request #5774: [Load] Support reading multi-line json objects for JsonScanner

[GitBox <gi...@apache.org>]

caiconghui commented on a change in pull request #5774:
URL: https://github.com/apache/incubator-doris/pull/5774#discussion_r632246796



##########
File path: be/src/exec/json_scanner.cpp
##########
@@ -270,27 +339,20 @@ void JsonReader::_close() {
     if (_closed) {
         return;
     }
-    if (typeid(*_file_reader) == typeid(doris::BrokerReader) ||
-        typeid(*_file_reader) == typeid(doris::LocalFileReader)) {
-        _file_reader->close();
-        delete _file_reader;
-    }
     _closed = true;
 }
 
-// read one json string from file read and parse it to json doc.
+// read one json string from line read and parse it to json doc.
 // return Status::DataQualityError() if data has quality error.
 // return other error if encounter other problemes.
 // return Status::OK() if parse succeed or reach EOF.
-Status JsonReader::_parse_json_doc(bool* eof) {
-    // read a whole message, must be delete json_str by `delete[]`
+Status JsonReader::_parse_json_doc(size_t* size, bool* eof) {
+    // read a whole message
     SCOPED_TIMER(_file_read_timer);
-    std::unique_ptr<uint8_t[]> json_str;
-    size_t length = 0;
-    RETURN_IF_ERROR(_file_reader->read_one_message(&json_str, &length));
-    _bytes_read_counter += length;
-    if (length == 0) {
-        *eof = true;
+    const uint8_t* json_str = nullptr;
+    RETURN_IF_ERROR(_line_reader->read_line(&json_str, size, eof));

Review comment:
       In fact, JSON usually serialized format does not contain line breaks. what you give above is JSON display format. If we encounter it, we will set the line_delimiter to other value




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org