You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Iván de Prado <iv...@gmail.com> on 2008/09/05 10:52:10 UTC
Load Function to load data directly from a database
Hello,
I would like to write a Load Function to load data in Pig directly from
tables in a database. The definition of the function would be
LoadFromDb(selectQuery, connectionParameters).
In order to work properly one requirements must be fit: The function
must be called only by one map. Otherwise the data would be duplicated.
Other difficulty: Each LoadFunction MUST be associated with a file in
the HDFS. But in this case that have not sense, because data comes from
the database.
I was thinking in use a fake empty file as file for the load function.
My questions is: If the fake file is empty, will that assure me that
only one Map will be launched?
Any other ideas are welcome.
Thanks!
Iván de Prado
www.ivanprado.es
Re: Load Function to load data directly from a database
Posted by Iván de Prado <iv...@gmail.com>.
Ok, thanks.
I think speculative execution is not a problem because the queries are
select queries. No problem if more that one task is executing the same
query, if only the data from one of them is used.
Iván de Prado Alonso
www.ivanprado.es
El lun, 08-09-2008 a las 09:18 -0700, Alan Gates escribió:
> In general hadoop does not start more than one map per split item unless
> you have speculative execution set.
>
> Alan.
>
> Iván de Prado wrote:
> > Thanks. It is one possibility, but I prefer to provide those parameters
> > from the script. Otherwise I would have to add a step before to write
> > the file.
> >
> > I have tried with an empty file, and nothing was executed. I guess that
> > Hadoop filter the file if it doesn't exists. I have tried with a small
> > file as well and it works! Only one mapper was used.
> >
> > I wonder if the replication factor of the file could affect the number
> > of mappers in the case of small files.
> >
> > Thanks,
> > Iván de Prado Alonso
> > www.ivanprado.es
> >
> > El vie, 05-09-2008 a las 19:15 +0800, paradisehit escribió:
> >
> >> I think you can write you selectQuery, connectionParameters in the configure file, then the loadFunc can read the configure file, and load the data.
> >>
> >>
> >>
> >>
> >>
> >>
> >> 在2008-09-05，"Iván de Prado" <iv...@gmail.com> 写道：
> >>
> >>> Hello,
> >>>
> >>> I would like to write a Load Function to load data in Pig directly from
> >>> tables in a database. The definition of the function would be
> >>> LoadFromDb(selectQuery, connectionParameters).
> >>>
> >>> In order to work properly one requirements must be fit: The function
> >>> must be called only by one map. Otherwise the data would be duplicated.
> >>>
> >>> Other difficulty: Each LoadFunction MUST be associated with a file in
> >>> the HDFS. But in this case that have not sense, because data comes from
> >>> the database.
> >>>
> >>> I was thinking in use a fake empty file as file for the load function.
> >>> My questions is: If the fake file is empty, will that assure me that
> >>> only one Map will be launched?
> >>>
> >>> Any other ideas are welcome.
> >>>
> >>> Thanks!
> >>> Iván de Prado
> >>> www.ivanprado.es
> >>>
> >>>
> >
> >
Re: Load Function to load data directly from a database
Posted by Alan Gates <ga...@yahoo-inc.com>.
In general hadoop does not start more than one map per split item unless
you have speculative execution set.
Alan.
Iván de Prado wrote:
> Thanks. It is one possibility, but I prefer to provide those parameters
> from the script. Otherwise I would have to add a step before to write
> the file.
>
> I have tried with an empty file, and nothing was executed. I guess that
> Hadoop filter the file if it doesn't exists. I have tried with a small
> file as well and it works! Only one mapper was used.
>
> I wonder if the replication factor of the file could affect the number
> of mappers in the case of small files.
>
> Thanks,
> Iván de Prado Alonso
> www.ivanprado.es
>
> El vie, 05-09-2008 a las 19:15 +0800, paradisehit escribió:
>
>> I think you can write you selectQuery, connectionParameters in the configure file, then the loadFunc can read the configure file, and load the data.
>>
>>
>>
>>
>>
>>
>> 在2008-09-05，"Iván de Prado" <iv...@gmail.com> 写道：
>>
>>> Hello,
>>>
>>> I would like to write a Load Function to load data in Pig directly from
>>> tables in a database. The definition of the function would be
>>> LoadFromDb(selectQuery, connectionParameters).
>>>
>>> In order to work properly one requirements must be fit: The function
>>> must be called only by one map. Otherwise the data would be duplicated.
>>>
>>> Other difficulty: Each LoadFunction MUST be associated with a file in
>>> the HDFS. But in this case that have not sense, because data comes from
>>> the database.
>>>
>>> I was thinking in use a fake empty file as file for the load function.
>>> My questions is: If the fake file is empty, will that assure me that
>>> only one Map will be launched?
>>>
>>> Any other ideas are welcome.
>>>
>>> Thanks!
>>> Iván de Prado
>>> www.ivanprado.es
>>>
>>>
>
>
Re: Re:Load Function to load data directly from a database
Posted by Iván de Prado <iv...@gmail.com>.
Thanks. It is one possibility, but I prefer to provide those parameters
from the script. Otherwise I would have to add a step before to write
the file.
I have tried with an empty file, and nothing was executed. I guess that
Hadoop filter the file if it doesn't exists. I have tried with a small
file as well and it works! Only one mapper was used.
I wonder if the replication factor of the file could affect the number
of mappers in the case of small files.
Thanks,
Iván de Prado Alonso
www.ivanprado.es
El vie, 05-09-2008 a las 19:15 +0800, paradisehit escribió:
> I think you can write you selectQuery, connectionParameters in the configure file, then the loadFunc can read the configure file, and load the data.
>
>
>
>
>
>
> 在2008-09-05，"Iván de Prado" <iv...@gmail.com> 写道：
> >Hello,
> >
> >I would like to write a Load Function to load data in Pig directly from
> >tables in a database. The definition of the function would be
> >LoadFromDb(selectQuery, connectionParameters).
> >
> >In order to work properly one requirements must be fit: The function
> >must be called only by one map. Otherwise the data would be duplicated.
> >
> >Other difficulty: Each LoadFunction MUST be associated with a file in
> >the HDFS. But in this case that have not sense, because data comes from
> >the database.
> >
> >I was thinking in use a fake empty file as file for the load function.
> >My questions is: If the fake file is empty, will that assure me that
> >only one Map will be launched?
> >
> >Any other ideas are welcome.
> >
> >Thanks!
> >Iván de Prado
> >www.ivanprado.es
> >
Re:Load Function to load data directly from a database
Posted by paradisehit <pa...@163.com>.
I think you can write you selectQuery, connectionParameters in the configure file, then the loadFunc can read the configure file, and load the data.
在2008-09-05，"Iván de Prado" <iv...@gmail.com> 写道：
>Hello,
>
>I would like to write a Load Function to load data in Pig directly from
>tables in a database. The definition of the function would be
>LoadFromDb(selectQuery, connectionParameters).
>
>In order to work properly one requirements must be fit: The function
>must be called only by one map. Otherwise the data would be duplicated.
>
>Other difficulty: Each LoadFunction MUST be associated with a file in
>the HDFS. But in this case that have not sense, because data comes from
>the database.
>
>I was thinking in use a fake empty file as file for the load function.
>My questions is: If the fake file is empty, will that assure me that
>only one Map will be launched?
>
>Any other ideas are welcome.
>
>Thanks!
>Iván de Prado
>www.ivanprado.es
>