skip.header.line.count is ignored in HiveContext

["sunerhan1992@sina.com" <su...@sina.com>]

hello,
       I've got a table in Hive(path located to csv formatted files) which
is configured to skip the header row using
TBLPROPERTIES("skip.header.line.count"="1").
When querying from Hive the header row is not included in the data, but when
running the same query via HiveContext I get the header row.
"show create table " via the HiveContext confirms that it is aware of the
setting.
 
I found a related jira https://issues.apache.org/jira/browse/SPARK-11374
 
about this problem, it is for spark 1.5.1 and it is closed without fixing.
I want to whether this problem will be fixed in spark 2.1?



sunerhan1992@sina.com

Re: skip.header.line.count is ignored in HiveContext

[Dongjoon Hyun <do...@gmail.com>]

Hi,

For 2.1.X, 2.1.2 was already out and I don't think 2.1.3 will be out.

At that time, I made a PR for SPARK-11374 based on Spark 2.0. I believe the
patch works in 2.1, too. If you need, you can try.

https://github.com/apache/spark/pull/14638

At that time, it was not accepted as you know, 'Won't Fix' was the
community decision at Spark 2.X. So, I guess 2.2.1 or 2.3 will not have a
fix for that.

Bests,
Dongjoon.


On Thu, Nov 9, 2017 at 5:59 PM, sunerhan1992@sina.com <sunerhan1992@sina.com
> wrote:

> hello,
>        I've got a table in Hive(path located to csv formatted files) which
> is configured to skip the header row using
> TBLPROPERTIES("skip.header.line.count"="1").
> When querying from Hive the header row is not included in the data, but
> when
> running the same query via HiveContext I get the header row.
> "show create table " via the HiveContext confirms that it is aware of the
> setting.
>
> I found a related jira https://issues.apache.org/jira/browse/SPARK-11374
>
> about this problem, it is for spark 1.5.1 and it is closed without fixing.
> I want to whether this problem will be fixed in spark 2.1?
>
> ------------------------------
> sunerhan1992@sina.com
>