[GitHub] [iceberg] singhpk234 commented on pull request #6655: Spark: Handle ResolvingFileIO while determining LocalityPreference

["singhpk234 (via GitHub)" <gi...@apache.org>]

singhpk234 commented on PR #6655:
URL: https://github.com/apache/iceberg/pull/6655#issuecomment-1416552184

   > I am not sure I understood why not simply use Util$mayHaveBlockLocations in Spark.
   > boolean defaultValue = Util.mayHaveBlockLocations(table.io(), table.location());
   return PropertyUtil.propertyAsBoolean(readOptions, SparkReadOptions.LOCALITY, defaultValue);
   
   previous behviour :
   | SparkReadOptions.LOCALITY      | FileIO | SparkReadConf#localityEnabled
   | ----------- | ----------- |---|
   | true      |  S3FileIO       | false |
   
   after the change : 
   
   | SparkReadOptions.LOCALITY      | FileIO | SparkReadConf#localityEnabled
   | ----------- | ----------- |---|
   | true      |  S3FileIO       | true |
   
   This means since localityEnabled is true now spark will call Utils#blockLocation for each SparkInputPartition even for S3FileIO if the conf is set which was not the case earlier. (Apologies for my prev correspondance.)
   
   But it makes sense to make flink and spark locality enabled identical, made the changes suggested above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org