[jira] [Created] (NIFI-4236) Exception in the program whose purpose is to deserialize/read content of an AVRO file with Spark SQL

["Ella (JIRA)" <ji...@apache.org>]

Ella created NIFI-4236:
--------------------------

             Summary: Exception in the program whose purpose is to deserialize/read content of an AVRO file with Spark SQL
                 Key: NIFI-4236
                 URL: https://issues.apache.org/jira/browse/NIFI-4236
             Project: Apache NiFi
          Issue Type: Bug
    Affects Versions: 1.3.0
         Environment: Windows 10- IntelliJIDEA
            Reporter: Ella


Hi Guys,

I have written a Java program so as to read content of an AVRO file by Spark Sql in IntelliJ; however, I get the following Exception--I have checked the correctness of type of file as well as its path for many times. I really got confused regarding this issue.
I would greatly appreciate if anyone guided me accordingly.

Thanks a lot in advance.
Regards,
Ella

//here is the Java code



import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.SparkContext$;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.SQLContext;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.catalyst.encoders.RowEncoder$;


/**
 * Created by asus on 7/24/2017.
 */
public class AvroTestFinal {

    public static void main(String[] args) {

        System.setProperty("hadoop.home.dir","c:\\winutil\\");
       SparkConf sparkConf = new SparkConf().setMaster("local").setAppName("TEST");
        JavaSparkContext sparkw = new JavaSparkContext(sparkConf);
        sparkw.hadoopConfiguration().setBoolean("avro.mapred.ignore.inputs.without.extension",false);

        SQLContext sqlContext = new SQLContext(sparkw);
        sqlContext.sparkContext().hadoopConfiguration().setBoolean("avro.mapred.ignore.inputs.without.extension", false);

        SparkContext spark = JavaSparkContext.toSparkContext(sparkw);
        // Creates a DataFrame from a specified file
        SparkSession sparkSession = new SparkSession(spark);
     
        sqlContext.read().format("com.databricks.spark.avro").load("C:/Users/asus/IdeaProjects/comavro/avro/509048290108207.avro");

Dataset<Row> tableRowsDataFrame = sparkSession.sql("select count(*) as total from avroTable");
        tableRowsDataFrame.show();
    }
}

//Here is the aforementioned Exception

tion in thread "main" java.io.IOException: Not an Avro data file
	at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:63)
	at com.databricks.spark.avro.DefaultSource$$anonfun$5.apply(DefaultSource.scala:80)
	at com.databricks.spark.avro.DefaultSource$$anonfun$5.apply(DefaultSource.scala:77)
	at scala.Option.getOrElse(Option.scala:121)
	at com.databricks.spark.avro.DefaultSource.inferSchema(DefaultSource.scala:77)
	at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:184)
	at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:184)
	at scala.Option.orElse(Option.scala:289)
	at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$getOrInferFileFormatSchema(DataSource.scala:183)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:387)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:135)
	at AvroTestFinal.main(AvroTestFinal.java:40)
17/07/28 21:10:31 INFO SparkContext: Invoking stop() from shutdown hook
17/07/28 21:10:31 INFO SparkUI: Stopped Spark web UI at http://192.168.209.1:4040
17/07/28 21:10:31 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/07/28 21:10:31 INFO MemoryStore: MemoryStore cleared
17/07/28 21:10:31 INFO BlockManager: BlockManager stopped
17/07/28 21:10:31 INFO BlockManagerMaster: BlockManagerMaster stopped
17/07/28 21:10:31 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/07/28 21:10:31 INFO SparkContext: Successfully stopped SparkContext
17/07/28 21:10:31 INFO ShutdownHookManager: Shutdown hook called
17/07/28 21:10:31 INFO ShutdownHookManager: Deleting directory C:\Users\asus\AppData\Local\Temp\spark-83a8dd26-f9a7-4f76-8806-12e3a9dc94f2

Process finished with exit code 1




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)