You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Olaf Collider <ol...@gmail.com> on 2016/08/27 00:18:11 UTC
Pig: is it possible to use pytz or dateutils for Python udfs?
I am using `datetime` in some Python udfs that I use in my `pig`
script. So far so good. I use pig 12.0 on cloudera.
However, I also need to use the `pytz` or `dateutils` packages as well
and they dont seem to be part of a vanilla python install.
Can I use them in my `Pig` udfs in some ways? If so, how?
Many thanks!
Re: Pig: is it possible to use pytz or dateutils for Python udfs?
Posted by Art Kho <be...@gmail.com>.
Most likely dateutil is not deployed in the host where your code is executed.
On a different matter, have you considered writing your code in PySpark instead of Pig?
Art
許冠俊
🌴🐚🐋☀️🏊📚😄
> On Aug 29, 2016, at 08:40, Olaf Collider <ol...@gmail.com> wrote:
>
> thanks Art
>
> i have installed python-dateutils in all the nodes but I still get
>
> from dateutil import tz
> importError: no module named dateutil
>
> what could cause that issue?
>
> thanks again
>
>> On Aug 26, 2016 22:13, "Art Kho" <be...@gmail.com> wrote:
>>
>> Olaf,
>>
>> You need to install python-dateutil in all of your data nodes. Cloudera's
>> pig 0.12 comes with jython 2.5, so make sure to install an older version of
>> Python-dateutil because the latest version uses functions that are not
>> supported by jython 2.5. For example, dateutil's relativedelta imports
>> math's copysign function; jython's math library doesn't support copysign. I
>> deployed Python-dateutil 1.4.1.
>>
>>
>>
>> Art
>>
>>
>>>> On Aug 26, 2016, at 17:18, Olaf Collider <ol...@gmail.com>
>>> wrote:
>>>
>>> I am using `datetime` in some Python udfs that I use in my `pig`
>>> script. So far so good. I use pig 12.0 on cloudera.
>>>
>>> However, I also need to use the `pytz` or `dateutils` packages as well
>>> and they dont seem to be part of a vanilla python install.
>>>
>>> Can I use them in my `Pig` udfs in some ways? If so, how?
>>>
>>> Many thanks!
>>
Re: Pig: is it possible to use pytz or dateutils for Python udfs?
Posted by Olaf Collider <ol...@gmail.com>.
thanks Art
i have installed python-dateutils in all the nodes but I still get
from dateutil import tz
importError: no module named dateutil
what could cause that issue?
thanks again
On Aug 26, 2016 22:13, "Art Kho" <be...@gmail.com> wrote:
> Olaf,
>
> You need to install python-dateutil in all of your data nodes. Cloudera's
> pig 0.12 comes with jython 2.5, so make sure to install an older version of
> Python-dateutil because the latest version uses functions that are not
> supported by jython 2.5. For example, dateutil's relativedelta imports
> math's copysign function; jython's math library doesn't support copysign. I
> deployed Python-dateutil 1.4.1.
>
>
>
> Art
>
>
> > On Aug 26, 2016, at 17:18, Olaf Collider <ol...@gmail.com>
> wrote:
> >
> > I am using `datetime` in some Python udfs that I use in my `pig`
> > script. So far so good. I use pig 12.0 on cloudera.
> >
> > However, I also need to use the `pytz` or `dateutils` packages as well
> > and they dont seem to be part of a vanilla python install.
> >
> > Can I use them in my `Pig` udfs in some ways? If so, how?
> >
> > Many thanks!
>
Re: Pig: is it possible to use pytz or dateutils for Python udfs?
Posted by Art Kho <be...@gmail.com>.
Olaf,
You need to install python-dateutil in all of your data nodes. Cloudera's pig 0.12 comes with jython 2.5, so make sure to install an older version of Python-dateutil because the latest version uses functions that are not supported by jython 2.5. For example, dateutil's relativedelta imports math's copysign function; jython's math library doesn't support copysign. I deployed Python-dateutil 1.4.1.
Art
> On Aug 26, 2016, at 17:18, Olaf Collider <ol...@gmail.com> wrote:
>
> I am using `datetime` in some Python udfs that I use in my `pig`
> script. So far so good. I use pig 12.0 on cloudera.
>
> However, I also need to use the `pytz` or `dateutils` packages as well
> and they dont seem to be part of a vanilla python install.
>
> Can I use them in my `Pig` udfs in some ways? If so, how?
>
> Many thanks!