You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mahender Sarangam <Ma...@outlook.com> on 2017/01/23 22:17:59 UTC
Hive Tez on External Table running on Single Mapper
Hi,
We have 20 GB txt File, When we have created external table on top of 20
Gb file, we see Tez is creating only one mapper. We have applied setting
like reducing distribution block size less 128 MB. Manually Set Mappers
and Reducer but of no use. We are using Tez 0.7 version, is there any
setting which makes application to run in more than one mapper. Input
file is not Compressed, it's a Txt File with 100 columns.
Re: Hive Tez on External Table running on Single Mapper
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> > 'skip.header.line.count'='1',
Trying removing that config option.
I've definitely seen footer markers disabling file splitting, possibly header also does.
Cheers,
Gopal
Re: Hive Tez on External Table running on Single Mapper
Posted by Mahender Sarangam <Ma...@outlook.com>.
ping..
On 1/24/2017 10:37 AM, Mahender Sarangam wrote:
> Here are the table properties
>
> TBLPROPERTIES (
> 'numFiles'='1',
> 'serialization.null.format'='',
> 'skip.header.line.count'='1',
> 'totalSize'='20971513935',
> 'transient_lastDdlTime'='1485091440');
>
>
> On 1/24/2017 10:27 AM, Mahender Sarangam wrote:
>> When i see properties of external table, STORED AS INPUTFORMAT
>> 'org.apache.hadoop.mapred.TextInputFormat'
>>
>>
>> On 1/23/2017 2:25 PM, Gopal Vijayaraghavan wrote:
>>>> We have 20 GB txt File, When we have created external table on top of 20
>>>> Gb file, we see Tez is creating only one mapper.
>>> For an uncompressed file, that is very strange. Is this created as "STORED AS TEXTFILE" or some other strange format?
>>>
>>> Cheers,
>>> Gopal
>>>
>>>
>>>
Re: Hive Tez on External Table running on Single Mapper
Posted by Mahender Sarangam <Ma...@outlook.com>.
Here are the table properties
TBLPROPERTIES (
'numFiles'='1',
'serialization.null.format'='',
'skip.header.line.count'='1',
'totalSize'='20971513935',
'transient_lastDdlTime'='1485091440');
On 1/24/2017 10:27 AM, Mahender Sarangam wrote:
> When i see properties of external table, STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
>
>
> On 1/23/2017 2:25 PM, Gopal Vijayaraghavan wrote:
>>> We have 20 GB txt File, When we have created external table on top of 20
>>> Gb file, we see Tez is creating only one mapper.
>> For an uncompressed file, that is very strange. Is this created as "STORED AS TEXTFILE" or some other strange format?
>>
>> Cheers,
>> Gopal
>>
>>
>>
Re: Hive Tez on External Table running on Single Mapper
Posted by Mahender Sarangam <Ma...@outlook.com>.
Here is the screenshot
[cid:part1.CDBA25BE.CDE6CB4A@outlook.com]
On 1/24/2017 10:27 AM, Mahender Sarangam wrote:
When i see properties of external table, STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
On 1/23/2017 2:25 PM, Gopal Vijayaraghavan wrote:
We have 20 GB txt File, When we have created external table on top of 20
Gb file, we see Tez is creating only one mapper.
For an uncompressed file, that is very strange. Is this created as "STORED AS TEXTFILE" or some other strange format?
Cheers,
Gopal
Re: Hive Tez on External Table running on Single Mapper
Posted by Mahender Sarangam <Ma...@outlook.com>.
When i see properties of external table, STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
On 1/23/2017 2:25 PM, Gopal Vijayaraghavan wrote:
>> We have 20 GB txt File, When we have created external table on top of 20
>> Gb file, we see Tez is creating only one mapper.
> For an uncompressed file, that is very strange. Is this created as "STORED AS TEXTFILE" or some other strange format?
>
> Cheers,
> Gopal
>
>
>
Re: Hive Tez on External Table running on Single Mapper
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> We have 20 GB txt File, When we have created external table on top of 20
> Gb file, we see Tez is creating only one mapper.
For an uncompressed file, that is very strange. Is this created as "STORED AS TEXTFILE" or some other strange format?
Cheers,
Gopal