You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by resonance <ma...@live.com> on 2016/04/20 04:07:57 UTC
How does .jsonFile() work?
Hi, this is more of a theoretical question and I'm asking it here because I
have no idea where to find documentation for this stuff.
I am currently working with Spark SQL and am considering using data
contained within JSON datasets. I am aware of the .jsonFile() method in
Spark SQL.
What is the general strategy used by Spark SQL .jsonFile() to parse/decode a
JSON dataset?
(For example, an answer I might be looking for is that the JSON file is read
into an ETL pipeline and transformed into a predefined data structure.)
I am deeply appreciative of any help provided.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-does-jsonFile-work-tp26802.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How does .jsonFile() work?
Posted by Aliaksandr Bedrytski <me...@gmail.com>.
If your question is about how the schema is inferred for JSON,
the paragraph 5.1 from this paper
https://amplab.cs.berkeley.edu/wp-content/uploads/2015/03/SparkSQLSigmod2015.pdf
explains it quite well (long story short, Spark tries to find
the most specific type for the field, otherwise it is a string)
On Thu, Apr 28, 2016 at 5:53 PM harjitdotsingh <si...@gmail.com>
wrote:
> From what I know and what I have played with, jsonFile reads JsonRecords
> which are defined as one record per line. Its not always the case that you
> can supply the data that way. If you have custom data json data where you
> cannot define a record per line, you will have to write your own
> customReceiver to receive the data and then parse it. I hope it makes
> sense.
> I wrote my own handler to read directory and that directory contained json
> files, I read until I have hit the EOF and then later call the store method
> which then sends the data to your driver.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-does-jsonFile-work-tp26802p26844.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: How does .jsonFile() work?
Posted by harjitdotsingh <si...@gmail.com>.
>From what I know and what I have played with, jsonFile reads JsonRecords
which are defined as one record per line. Its not always the case that you
can supply the data that way. If you have custom data json data where you
cannot define a record per line, you will have to write your own
customReceiver to receive the data and then parse it. I hope it makes sense.
I wrote my own handler to read directory and that directory contained json
files, I read until I have hit the EOF and then later call the store method
which then sends the data to your driver.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-does-jsonFile-work-tp26802p26844.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How does .jsonFile() work?
Posted by Hyukjin Kwon <gu...@gmail.com>.
Hi,
I hope I understood correctly. This is a simplified procedures.
Precondition
- JSON file is written line by line. Each is each JSON document.
- Root array is supported,
eg.
[{...},
{...}
{...}]
Procedures
- Schema inference (If user schema is not given)
1. Read line JSON document.
2. Read token by token by Jackson parser (recursively if needed)
3. Find (or infer) ,from each value , types appropriate with Spark. (eg.
DateType, StringType and DecimalType)
4. 3. returns a StructType for underlying a single JSON document.
5. 1.-4. becomes an RDD, RDD[DataType]
6. Aggregates each DataType (StructType normally) into a single
StructType compatible across all the StructTypes.
7. Use the aggregated StructType as a schema.
- Parse JSON data.
1. Read line JSON document.
2. Read token by token by Jackson parser (recursively if needed)
3. Convert each value to the given type (above).
4. 3. returns a Row for underlying a single JSON document.
Thanks!
2016-04-20 11:07 GMT+09:00 resonance <ma...@live.com>:
> Hi, this is more of a theoretical question and I'm asking it here because I
> have no idea where to find documentation for this stuff.
>
> I am currently working with Spark SQL and am considering using data
> contained within JSON datasets. I am aware of the .jsonFile() method in
> Spark SQL.
>
> What is the general strategy used by Spark SQL .jsonFile() to parse/decode
> a
> JSON dataset?
>
> (For example, an answer I might be looking for is that the JSON file is
> read
> into an ETL pipeline and transformed into a predefined data structure.)
>
> I am deeply appreciative of any help provided.
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-does-jsonFile-work-tp26802.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>