You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Filip <fi...@gmail.com> on 2019/06/24 14:46:13 UTC
Iceberg API incompatibility - written file paths (to human
representation) breaks partition discovery based on generated path
For time-based related partition transforms Iceberg makes the file
partition path human readable
(i.e. for a day transform we get /date=2019-01-15 even though the actual
value of the partition is of type Integer, namely the diff in number of
days since epoch time).
So when we rely on Iceberg to retrieve the partition from the file path we
get an error - so today we need to convert its value to the Iceberg Integer
equivalent (we basically re-apply the function of determining the diff in
number of days since epoch time).
Reading a path `protocol:/internet/date=2019-01-01` should actually
translate to extracting date as 17907 (days since epoch till 2019-01-01) -
assuming that `date` is a `day` function backed partition transform. Makes
sense?
Is it a known limitation or is it an issue? If it's an issue, do you know
if it's already tracked?
/Filip