You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "soumilshah1995 (via GitHub)" <gi...@apache.org> on 2023/04/08 12:54:22 UTC

[GitHub] [hudi] soumilshah1995 opened a new issue, #8409: [SUPPORT] Hudi Delta Streamer Ingesting JSON files Question

soumilshah1995 opened a new issue, #8409:
URL: https://github.com/apache/hudi/issues/8409

   "Does Apache Hudi Delta Streamer support ingestion of a JSON file with multiple JSON records in a single file, where each record is delimited by a new line character? For example, if I have a file named File1.json with several JSON records separated by new line characters, can I use Hudi Delta Streamer to ingest the data?"
   
   
   Input Format  org.apache.hadoop.mapred.TextInputFormat
   
   '''
   {}
   {}
   {}
   {}
   '''
   This is how records are in files
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] soumilshah1995 commented on issue #8409: [SUPPORT] Hudi Delta Streamer Ingesting JSON files Question

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #8409:
URL: https://github.com/apache/hudi/issues/8409#issuecomment-1503221895

   Thanks to our event-driven architecture, which includes consumer-driven data dumping into SQS and lambda batch uploading of JSON files, we dump a lot of data onto S3. This will enable us to use Hudi Delta Streamer for our use cases right out of the box. I appreciate you creating JIRA tickets.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codope commented on issue #8409: [SUPPORT] Hudi Delta Streamer Ingesting JSON files Question

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #8409:
URL: https://github.com/apache/hudi/issues/8409#issuecomment-1505003812

   Closing it as we have a jira to track the enhancement.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] soumilshah1995 commented on issue #8409: [SUPPORT] Hudi Delta Streamer Ingesting JSON files Question

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #8409:
URL: https://github.com/apache/hudi/issues/8409#issuecomment-1505798598

   > Closing it as we have a jira to track the enhancement.
   
   sounds good buddy thanks :D 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] ad1happy2go commented on issue #8409: [SUPPORT] Hudi Delta Streamer Ingesting JSON files Question

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8409:
URL: https://github.com/apache/hudi/issues/8409#issuecomment-1502918948

   Just like CSV, delta streamer doesn't  yet support the file JSON sources using normal spark reader like spark.read.format("csv")
   
   This can be good enhancement and we can add the normal JSON source also by overriding org.apache.hudi.utilities.sources.
   
   This type of source doesn't need to specify schema as it will directly infer the same.
   
   Created JIRA for the same - https://issues.apache.org/jira/browse/HUDI-6059
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codope closed issue #8409: [SUPPORT] Hudi Delta Streamer Ingesting JSON files Question

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #8409: [SUPPORT] Hudi Delta Streamer Ingesting JSON files Question
URL: https://github.com/apache/hudi/issues/8409


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org