You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Marcel Holle <ma...@googlemail.com> on 2012/01/11 02:10:44 UTC
How to use CombineFileInputFormat in Pig
How could I use the CombineFileInputFormat in Pig? I have a performance
issue with lots of small files which I want to get rid of. I think by
default the FileInputFormat is used.
Re: How to use CombineFileInputFormat in Pig
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
It just works. Pig does this automatically.
On Tue, Jan 10, 2012 at 5:10 PM, Marcel Holle
<ma...@googlemail.com> wrote:
> How could I use the CombineFileInputFormat in Pig? I have a performance
> issue with lots of small files which I want to get rid of. I think by
> default the FileInputFormat is used.
Re: How to use CombineFileInputFormat in Pig
Posted by Thejas Nair <th...@hortonworks.com>.
What load function are you using ? if it implements some of the
interfaces specified here, it turns off split combination -
http://pig.apache.org/docs/r0.9.1/perf.html#combine-files
-Thejas
On 1/11/12 11:07 PM, Marcel Holle wrote:
> In my pig.properties are only these parameters specified: log4jconf,
> fs.default.name, mapred.job.tracker. So it should use the
> CombineFileInputFormat by default. I have 100.000 files of around 16K.
>
> 2012/1/11 Prashant Kommireddi<pr...@gmail.com>
>
>> Hi Marcel,
>>
>> You might not find "pig.splitCombination" in your configuration if not
>> manually set. Pig internally defaults it to true.
>>
>> What is the value of "pig.maxCombinedSplitSize", if you are not setting it
>> manually this should be equal to your block size. What is the individual
>> filesize of the small files?
>>
>> Thanks,
>> Prashant
>>
>>
>> On Wed, Jan 11, 2012 at 3:18 PM, Marcel Holle
>> <ma...@googlemail.com>wrote:
>>
>>> If I got it right I should see an output like "Total input paths
>> (combined)
>>> to process : 7" when I run a pig script, but I'm missing the "(combined)"
>>> part, so CombineFileInputFormat is not used? Where could I find the pig
>>> configuration? I think I have to check the "pig.splitCombination" value.
>>>
>>> 2012/1/11 Daniel Dai<da...@hortonworks.com>
>>>
>>>> Check PIG-1518.
>>>>
>>>> Daniel
>>>>
>>>> On Wed, Jan 11, 2012 at 11:01 AM, Marcel Holle
>>>> <ma...@googlemail.com>wrote:
>>>>
>>>>> How could I verify this information? Could you point me to a config
>> or
>>>> the
>>>>> source code?
>>>>>
>>>>> 2012/1/11 Daniel Dai<da...@hortonworks.com>
>>>>>
>>>>>> It is default in 0.8 as well.
>>>>>>
>>>>>> Daniel
>>>>>>
>>>>>> On Wed, Jan 11, 2012 at 10:43 AM, Marcel Holle
>>>>>> <ma...@googlemail.com>wrote:
>>>>>>
>>>>>>> Is there also a way to activate the CombineFileInputFormat in Pig
>>>>> 0.8.1?
>>>>>>>
>>>>>>> 2012/1/10 Alex Rovner<al...@gmail.com>
>>>>>>>
>>>>>>>> In versions 9+ default is CombineFileInputFormat
>>>>>>>>
>>>>>>>> On Tue, Jan 10, 2012 at 8:10 PM, Marcel Holle
>>>>>>>> <ma...@googlemail.com>wrote:
>>>>>>>>
>>>>>>>>> How could I use the CombineFileInputFormat in Pig? I have a
>>>>>> performance
>>>>>>>>> issue with lots of small files which I want to get rid of. I
>>>> think
>>>>> by
>>>>>>>>> default the FileInputFormat is used.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: How to use CombineFileInputFormat in Pig
Posted by Marcel Holle <ma...@googlemail.com>.
In my pig.properties are only these parameters specified: log4jconf,
fs.default.name, mapred.job.tracker. So it should use the
CombineFileInputFormat by default. I have 100.000 files of around 16K.
2012/1/11 Prashant Kommireddi <pr...@gmail.com>
> Hi Marcel,
>
> You might not find "pig.splitCombination" in your configuration if not
> manually set. Pig internally defaults it to true.
>
> What is the value of "pig.maxCombinedSplitSize", if you are not setting it
> manually this should be equal to your block size. What is the individual
> filesize of the small files?
>
> Thanks,
> Prashant
>
>
> On Wed, Jan 11, 2012 at 3:18 PM, Marcel Holle
> <ma...@googlemail.com>wrote:
>
> > If I got it right I should see an output like "Total input paths
> (combined)
> > to process : 7" when I run a pig script, but I'm missing the "(combined)"
> > part, so CombineFileInputFormat is not used? Where could I find the pig
> > configuration? I think I have to check the "pig.splitCombination" value.
> >
> > 2012/1/11 Daniel Dai <da...@hortonworks.com>
> >
> > > Check PIG-1518.
> > >
> > > Daniel
> > >
> > > On Wed, Jan 11, 2012 at 11:01 AM, Marcel Holle
> > > <ma...@googlemail.com>wrote:
> > >
> > > > How could I verify this information? Could you point me to a config
> or
> > > the
> > > > source code?
> > > >
> > > > 2012/1/11 Daniel Dai <da...@hortonworks.com>
> > > >
> > > > > It is default in 0.8 as well.
> > > > >
> > > > > Daniel
> > > > >
> > > > > On Wed, Jan 11, 2012 at 10:43 AM, Marcel Holle
> > > > > <ma...@googlemail.com>wrote:
> > > > >
> > > > > > Is there also a way to activate the CombineFileInputFormat in Pig
> > > > 0.8.1?
> > > > > >
> > > > > > 2012/1/10 Alex Rovner <al...@gmail.com>
> > > > > >
> > > > > > > In versions 9+ default is CombineFileInputFormat
> > > > > > >
> > > > > > > On Tue, Jan 10, 2012 at 8:10 PM, Marcel Holle
> > > > > > > <ma...@googlemail.com>wrote:
> > > > > > >
> > > > > > > > How could I use the CombineFileInputFormat in Pig? I have a
> > > > > performance
> > > > > > > > issue with lots of small files which I want to get rid of. I
> > > think
> > > > by
> > > > > > > > default the FileInputFormat is used.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: How to use CombineFileInputFormat in Pig
Posted by Prashant Kommireddi <pr...@gmail.com>.
Hi Marcel,
You might not find "pig.splitCombination" in your configuration if not
manually set. Pig internally defaults it to true.
What is the value of "pig.maxCombinedSplitSize", if you are not setting it
manually this should be equal to your block size. What is the individual
filesize of the small files?
Thanks,
Prashant
On Wed, Jan 11, 2012 at 3:18 PM, Marcel Holle
<ma...@googlemail.com>wrote:
> If I got it right I should see an output like "Total input paths (combined)
> to process : 7" when I run a pig script, but I'm missing the "(combined)"
> part, so CombineFileInputFormat is not used? Where could I find the pig
> configuration? I think I have to check the "pig.splitCombination" value.
>
> 2012/1/11 Daniel Dai <da...@hortonworks.com>
>
> > Check PIG-1518.
> >
> > Daniel
> >
> > On Wed, Jan 11, 2012 at 11:01 AM, Marcel Holle
> > <ma...@googlemail.com>wrote:
> >
> > > How could I verify this information? Could you point me to a config or
> > the
> > > source code?
> > >
> > > 2012/1/11 Daniel Dai <da...@hortonworks.com>
> > >
> > > > It is default in 0.8 as well.
> > > >
> > > > Daniel
> > > >
> > > > On Wed, Jan 11, 2012 at 10:43 AM, Marcel Holle
> > > > <ma...@googlemail.com>wrote:
> > > >
> > > > > Is there also a way to activate the CombineFileInputFormat in Pig
> > > 0.8.1?
> > > > >
> > > > > 2012/1/10 Alex Rovner <al...@gmail.com>
> > > > >
> > > > > > In versions 9+ default is CombineFileInputFormat
> > > > > >
> > > > > > On Tue, Jan 10, 2012 at 8:10 PM, Marcel Holle
> > > > > > <ma...@googlemail.com>wrote:
> > > > > >
> > > > > > > How could I use the CombineFileInputFormat in Pig? I have a
> > > > performance
> > > > > > > issue with lots of small files which I want to get rid of. I
> > think
> > > by
> > > > > > > default the FileInputFormat is used.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: How to use CombineFileInputFormat in Pig
Posted by Marcel Holle <ma...@googlemail.com>.
If I got it right I should see an output like "Total input paths (combined)
to process : 7" when I run a pig script, but I'm missing the "(combined)"
part, so CombineFileInputFormat is not used? Where could I find the pig
configuration? I think I have to check the "pig.splitCombination" value.
2012/1/11 Daniel Dai <da...@hortonworks.com>
> Check PIG-1518.
>
> Daniel
>
> On Wed, Jan 11, 2012 at 11:01 AM, Marcel Holle
> <ma...@googlemail.com>wrote:
>
> > How could I verify this information? Could you point me to a config or
> the
> > source code?
> >
> > 2012/1/11 Daniel Dai <da...@hortonworks.com>
> >
> > > It is default in 0.8 as well.
> > >
> > > Daniel
> > >
> > > On Wed, Jan 11, 2012 at 10:43 AM, Marcel Holle
> > > <ma...@googlemail.com>wrote:
> > >
> > > > Is there also a way to activate the CombineFileInputFormat in Pig
> > 0.8.1?
> > > >
> > > > 2012/1/10 Alex Rovner <al...@gmail.com>
> > > >
> > > > > In versions 9+ default is CombineFileInputFormat
> > > > >
> > > > > On Tue, Jan 10, 2012 at 8:10 PM, Marcel Holle
> > > > > <ma...@googlemail.com>wrote:
> > > > >
> > > > > > How could I use the CombineFileInputFormat in Pig? I have a
> > > performance
> > > > > > issue with lots of small files which I want to get rid of. I
> think
> > by
> > > > > > default the FileInputFormat is used.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: How to use CombineFileInputFormat in Pig
Posted by Daniel Dai <da...@hortonworks.com>.
Check PIG-1518.
Daniel
On Wed, Jan 11, 2012 at 11:01 AM, Marcel Holle
<ma...@googlemail.com>wrote:
> How could I verify this information? Could you point me to a config or the
> source code?
>
> 2012/1/11 Daniel Dai <da...@hortonworks.com>
>
> > It is default in 0.8 as well.
> >
> > Daniel
> >
> > On Wed, Jan 11, 2012 at 10:43 AM, Marcel Holle
> > <ma...@googlemail.com>wrote:
> >
> > > Is there also a way to activate the CombineFileInputFormat in Pig
> 0.8.1?
> > >
> > > 2012/1/10 Alex Rovner <al...@gmail.com>
> > >
> > > > In versions 9+ default is CombineFileInputFormat
> > > >
> > > > On Tue, Jan 10, 2012 at 8:10 PM, Marcel Holle
> > > > <ma...@googlemail.com>wrote:
> > > >
> > > > > How could I use the CombineFileInputFormat in Pig? I have a
> > performance
> > > > > issue with lots of small files which I want to get rid of. I think
> by
> > > > > default the FileInputFormat is used.
> > > > >
> > > >
> > >
> >
>
Re: How to use CombineFileInputFormat in Pig
Posted by Marcel Holle <ma...@googlemail.com>.
How could I verify this information? Could you point me to a config or the
source code?
2012/1/11 Daniel Dai <da...@hortonworks.com>
> It is default in 0.8 as well.
>
> Daniel
>
> On Wed, Jan 11, 2012 at 10:43 AM, Marcel Holle
> <ma...@googlemail.com>wrote:
>
> > Is there also a way to activate the CombineFileInputFormat in Pig 0.8.1?
> >
> > 2012/1/10 Alex Rovner <al...@gmail.com>
> >
> > > In versions 9+ default is CombineFileInputFormat
> > >
> > > On Tue, Jan 10, 2012 at 8:10 PM, Marcel Holle
> > > <ma...@googlemail.com>wrote:
> > >
> > > > How could I use the CombineFileInputFormat in Pig? I have a
> performance
> > > > issue with lots of small files which I want to get rid of. I think by
> > > > default the FileInputFormat is used.
> > > >
> > >
> >
>
Re: How to use CombineFileInputFormat in Pig
Posted by Daniel Dai <da...@hortonworks.com>.
It is default in 0.8 as well.
Daniel
On Wed, Jan 11, 2012 at 10:43 AM, Marcel Holle
<ma...@googlemail.com>wrote:
> Is there also a way to activate the CombineFileInputFormat in Pig 0.8.1?
>
> 2012/1/10 Alex Rovner <al...@gmail.com>
>
> > In versions 9+ default is CombineFileInputFormat
> >
> > On Tue, Jan 10, 2012 at 8:10 PM, Marcel Holle
> > <ma...@googlemail.com>wrote:
> >
> > > How could I use the CombineFileInputFormat in Pig? I have a performance
> > > issue with lots of small files which I want to get rid of. I think by
> > > default the FileInputFormat is used.
> > >
> >
>
Re: How to use CombineFileInputFormat in Pig
Posted by Marcel Holle <ma...@googlemail.com>.
Is there also a way to activate the CombineFileInputFormat in Pig 0.8.1?
2012/1/10 Alex Rovner <al...@gmail.com>
> In versions 9+ default is CombineFileInputFormat
>
> On Tue, Jan 10, 2012 at 8:10 PM, Marcel Holle
> <ma...@googlemail.com>wrote:
>
> > How could I use the CombineFileInputFormat in Pig? I have a performance
> > issue with lots of small files which I want to get rid of. I think by
> > default the FileInputFormat is used.
> >
>
Re: How to use CombineFileInputFormat in Pig
Posted by Alex Rovner <al...@gmail.com>.
In versions 9+ default is CombineFileInputFormat
On Tue, Jan 10, 2012 at 8:10 PM, Marcel Holle
<ma...@googlemail.com>wrote:
> How could I use the CombineFileInputFormat in Pig? I have a performance
> issue with lots of small files which I want to get rid of. I think by
> default the FileInputFormat is used.
>