[GitHub] [iceberg] manishmalhotrawork edited a comment on pull request #1466: Warning about RemoveOrphanFiles

[GitBox <gi...@apache.org>]

manishmalhotrawork edited a comment on pull request #1466:
URL: https://github.com/apache/iceberg/pull/1466#issuecomment-693752556


   > Having a UDF that accepts columns from two relations does not eliminate the cross join.
   > 
   > I guess we have two options:
   > 
   > * keep the join by file name and replace `contains` condition with another UDF that would ignore authority
   > * replace the existing UDF that produces file names with another UDF that would produce a scheme and a relative path and then use DataFrame operations. That way, we will have only one UDF.
   > 
   > ```
   > Column pathCond = actualFileDF.col("relative_path").equalTo(validDataFileDF.col("relative_path"));
   > Column schemeEquality = actualFileDF.col("scheme").equalTo(validDataFileDF.col("scheme")); 
   > Column schemeCond = validDataFileDF.col("scheme").isNull().or(schemeEquality);
   > Column joinCond = pathCond.and(schemeCond);
   > ```
   
   @aokolnychyi thanks for adding dtails.
   
   yeah as our internal discussion and internal PR I have is similar to this.
   Though while testing, realized checking name of the `scheme` can also be troublesome as name of the scheme is configurable like instead of `hdfs` we can `myhdfs` and same for S3.
   
   I believe not considering `scheme` should also be ok, as there should not be case where table location is in HDFS but data its pointing to S3 which needs to be removed?
   
   So, either can avoid checking scheme as well, or have a flag to consider that or not.
   
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org