You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "YoungYik (JIRA)" <ji...@apache.org> on 2011/03/23 08:48:05 UTC

[jira] [Created] (HIVE-2072) set hive.exec.reducers.max= is not working to view

set hive.exec.reducers.max=<number> is not working to view
----------------------------------------------------------

                 Key: HIVE-2072
                 URL: https://issues.apache.org/jira/browse/HIVE-2072
             Project: Hive
          Issue Type: Bug
          Components: CLI
    Affects Versions: 0.6.0
         Environment: Linux 2.6.18-128.el5 x86_64 GNU/Linux, JRE 1.6.0_14
            Reporter: YoungYik


I create two views from one table of different partitions, and join these two views in my query, it just use 1 reducer and the tasks stay in 82% for a long time, and then failed.

So, I use set hive.exec.reducers.max=28 before the execution of query, when select on joined tables, it works, but still keep saying "Number of reduce tasks determined at compile time: 1" when on the two joined views:


1)hive -e "create view view_1(uname, login) as select uname,'this30d' from userlist where domain='$domain' and year=$YYYY and month=$MM and day=$DD and type='all' and period_days=30;"

2)hive -e "create view view_0(uname, login) as select uname,'last30d' from userlist where domain='$domain' and year=$YYYY and month=$MM and day=$DD and type='all' and period_days=30;"

3)hive -e "set mapred.reduce.tasks=28; set; select v0.login,v1.login,count(*) from view_0 v0 full outer join view_1 v1 group by v0.login,v1.login; set;"

then the output:

Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>

and at last, it just still keep using 1 reducer in this query.
but if I use "set;" to print the environment, it tells me:

mapred.reduce.tasks=28

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HIVE-2072) test

Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Carl Steinbach resolved HIVE-2072.
----------------------------------

    Resolution: Incomplete

> test
> ----
>
>                 Key: HIVE-2072
>                 URL: https://issues.apache.org/jira/browse/HIVE-2072
>             Project: Hive
>          Issue Type: Test
>            Reporter: YoungYik
>            Priority: Trivial
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2072) set hive.exec.reducers.max= is not working to view

Posted by "YoungYik (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010538#comment-13010538 ] 

YoungYik commented on HIVE-2072:
--------------------------------

I have made a mistake in my query SQL, just forget this issue, sorry.

> set hive.exec.reducers.max=<number> is not working to view
> ----------------------------------------------------------
>
>                 Key: HIVE-2072
>                 URL: https://issues.apache.org/jira/browse/HIVE-2072
>             Project: Hive
>          Issue Type: Bug
>          Components: CLI
>    Affects Versions: 0.6.0
>         Environment: Linux 2.6.18-128.el5 x86_64 GNU/Linux, JRE 1.6.0_14
>            Reporter: YoungYik
>
> I create two views from one table of different partitions, and join these two views in my query, it just use 1 reducer and the tasks stay in 82% for a long time, and then failed.
> So, I use set hive.exec.reducers.max=28 before the execution of query, when select on joined tables, it works, but still keep saying "Number of reduce tasks determined at compile time: 1" when on the two joined views:
> 1)hive -e "create view view_1(uname, login) as select uname,'this30d' from userlist where domain='$domain' and year=$YYYY and month=$MM and day=$DD and type='all' and period_days=30;"
> 2)hive -e "create view view_0(uname, login) as select uname,'last30d' from userlist where domain='$domain' and year=$YYYY and month=$MM and day=$DD and type='all' and period_days=30;"
> 3)hive -e "set mapred.reduce.tasks=28; set; select v0.login,v1.login,count(*) from view_0 v0 full outer join view_1 v1 group by v0.login,v1.login; set;"
> then the output:
> Total MapReduce jobs = 2
> Launching Job 1 out of 2
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
>   set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
>   set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
>   set mapred.reduce.tasks=<number>
> and at last, it just still keep using 1 reducer in this query.
> but if I use "set;" to print the environment, it tells me:
> mapred.reduce.tasks=28

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2072) test

Posted by "YoungYik (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

YoungYik updated HIVE-2072:
---------------------------

          Component/s:     (was: CLI)
          Description:     (was: I create two views from one table of different partitions, and join these two views in my query, it just use 1 reducer and the tasks stay in 82% for a long time, and then failed.

So, I use set hive.exec.reducers.max=28 before the execution of query, when select on joined tables, it works, but still keep saying "Number of reduce tasks determined at compile time: 1" when on the two joined views:


1)hive -e "create view view_1(uname, login) as select uname,'this30d' from userlist where domain='$domain' and year=$YYYY and month=$MM and day=$DD and type='all' and period_days=30;"

2)hive -e "create view view_0(uname, login) as select uname,'last30d' from userlist where domain='$domain' and year=$YYYY and month=$MM and day=$DD and type='all' and period_days=30;"

3)hive -e "set mapred.reduce.tasks=28; set; select v0.login,v1.login,count(*) from view_0 v0 full outer join view_1 v1 group by v0.login,v1.login; set;"

then the output:

Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>

and at last, it just still keep using 1 reducer in this query.
but if I use "set;" to print the environment, it tells me:

mapred.reduce.tasks=28)
             Priority: Trivial  (was: Major)
          Environment:     (was: Linux 2.6.18-128.el5 x86_64 GNU/Linux, JRE 1.6.0_14)
    Affects Version/s:     (was: 0.6.0)
           Issue Type: Test  (was: Bug)
              Summary: test  (was: set hive.exec.reducers.max=<number> is not working to view)

> test
> ----
>
>                 Key: HIVE-2072
>                 URL: https://issues.apache.org/jira/browse/HIVE-2072
>             Project: Hive
>          Issue Type: Test
>            Reporter: YoungYik
>            Priority: Trivial
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira