Hive Partition Names for numbers (SPOT-19 and SPOT-239)

[Jeremy Nelson <je...@digitalminion.com>]

Greetings,

I noticed that SPOT-19 and SPOT-239 describe two aspects of the same
problem – that is – HIVE partition names are stored as numbers (ie,
y=2020/m=3/d=4) but some software (like the SPOT-ML) expects these
partition names to be stored as two-digit zero-padded strings (ie,
y=2020/m=03/d=04).

We should identify which approach is preferable (break the schema, or break
the software?) and then harmonize all affected systems to do the same
thing, one way or the other.

Thanks,

Jeremy

Re: Hive Partition Names for numbers (SPOT-19 and SPOT-239)

[Vivienne Pustell <vi...@yellingviv.com>]

Hi Jeremy,

My personal vote would be to align around the two-digit zero-padded
strings. It's more standardized, and the consistency in format makes it
easier to run scripts as needed. In that sense, it should be viable to run
a script to change the numbers that are currently stored, and then stay
consistent going forward. We've been running into issues with things like
this in $DAY_JOB and it's definitely worth getting consistent sooner rather
than later, IMO.

Cheers,
-Vivienne

On Mon, Mar 30, 2020 at 12:47 PM Jeremy Nelson <je...@digitalminion.com>
wrote:

> Greetings,
>
> I noticed that SPOT-19 and SPOT-239 describe two aspects of the same
> problem – that is – HIVE partition names are stored as numbers (ie,
> y=2020/m=3/d=4) but some software (like the SPOT-ML) expects these
> partition names to be stored as two-digit zero-padded strings (ie,
> y=2020/m=03/d=04).
>
> We should identify which approach is preferable (break the schema, or break
> the software?) and then harmonize all affected systems to do the same
> thing, one way or the other.
>
> Thanks,
>
> Jeremy
>