Inquery about Spark's behaviour for configurations in Hadoop configuration instance via read/write.options()

[Hyukjin Kwon <gu...@gmail.com>]

Hi all,


If my understanding is correct, now Spark supports to set some options to
Hadoop configuration instance via read/write.option(..) API.

However, I recently saw some comments and opinion about this. If I
understood them correctly, it was as below:

   -

   Respecting all the configurations in Hadoop configuration instance
   including what Spark configurations/options cover and Spark
   configurations/options will override it if the equivalent ones are
   duplicated
   -

   Not respecting the configurations in the instance that Spark
   configurations/options cover, meaning using the default value in Spark
   configuration regardless of the values set in this instance for the
   equivalent ones.

For example, now, Spark is supporting compression for ORC as an option but
currently we are not respecting orc.compress. This is being ignored
(meaning following the latter case).

Maybe I understood those comments and opinions wrongly and might be a dump
question from my misunderstanding but I would really appreciate that if
anyone helps me to know which one is correct.
​

Thanks !