reading data from s3

[Hitesh Goyal <hi...@nlpcaptcha.com>]

Hi team,
I want to read the text file from s3. I am doing it using DataFrame. Like below:-
DataFrame d=sql.read().text("s3://my_first_text_file.txt");
              d.registerTempTable("table1");
              DataFrame d1=sql.sql("Select * from table1");
              d1.printSchema();
              d1.show();

But it is not registering the text file as a temp table so that I  can make SQL  queries on that. Can't I do this on a text file ?? Or if I can, suggest any way to do.
Like if I try to do it by JSON file, it is successful.

Regards,
Hitesh Goyal
Simpli5d Technologies
Cont No.: 9996588220

Re: reading data from s3

[Sudev A C <su...@goibibo.com>]

Hi Hitesh,

Schema of the table is inferred automatically if you are reading from JSON
file, wherein when you are reading from a text file you will have to
provide a schema for the table you want to create (JSON has schema within
it).

You can create a data frames and register them as tables.
1. Inferring schema using reflection
<https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection>
2. Programmatically Specifying the Schema
<https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema>

Also you may use packages like spark-csv to infer the schema from CSV.
https://github.com/databricks/spark-csv#sql-api

Thanks
Sudev

On Fri, Dec 9, 2016 at 11:13 AM Hitesh Goyal <hi...@nlpcaptcha.com>
wrote:

Hi team,

I want to read the text file from s3. I am doing it using DataFrame. Like
below:-

DataFrame d=sql.read().text("s3://my_first_text_file.txt");

              d.registerTempTable("table1");

              DataFrame d1=sql.sql("Select * from table1");

              d1.printSchema();

              d1.show();



But it is not registering the text file as a temp table so that I  can make
SQL  queries on that. Can’t I do this on a text file ?? Or if I can,
suggest any way to do.

Like if I try to do it by JSON file, it is successful.



Regards,

*Hitesh Goyal*

Simpli5d Technologies

Cont No.: 9996588220