[GitHub] [incubator-iceberg] danielcweeks opened a new pull request #593: Scope pig properties specific to a load using signatures

[GitBox <gi...@apache.org>]

danielcweeks opened a new pull request #593: Scope pig properties specific to a load using signatures
URL: https://github.com/apache/incubator-iceberg/pull/593
 
 
   In cases where pig loads multiple Iceberg tables in the same map stage, it's possible that values we set in the job configuration can collide (e.g. filter expression).
   
   This scopes those properties to the signature set in the LoadFunc to prevent collisions.
   
   We observed cases where the a filter was being applied to the wrong table during `getSplits` and while this could be resolved by clearing out properties between loads (e.g. `setLocation` calls), it seems best to ensure everything is scoped to the signature.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org