You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Carl Steinbach (JIRA)" <ji...@apache.org> on 2011/07/07 05:03:18 UTC
[jira] [Created] (HIVE-2269) Hive --auxpath option can't handle
multiple colon separated values
Hive --auxpath option can't handle multiple colon separated values
------------------------------------------------------------------
Key: HIVE-2269
URL: https://issues.apache.org/jira/browse/HIVE-2269
Project: Hive
Issue Type: Bug
Components: CLI
Affects Versions: 0.7.1, 0.7.0
Reporter: Carl Steinbach
Assignee: Carl Steinbach
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle
multiple colon separated values
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061295#comment-13061295 ]
Edward Capriolo commented on HIVE-2269:
---------------------------------------
I am someone confused by the title, auxpath is 'comma' separate but the title says 'colon'.
> Hive --auxpath option can't handle multiple colon separated values
> ------------------------------------------------------------------
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.7.0, 0.7.1
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-2269-auxpath.1.patch.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle
multiple colon separated values
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061018#comment-13061018 ]
Carl Steinbach commented on HIVE-2269:
--------------------------------------
{code}
% hive --auxpath $HIVE_HOME/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar:$HIVE_HOME/lib/hive-contrib-0.8.0-SNAPSHOT.jar
Logging initialized using configuration in jar:file:/Users/carl/Work/repos/hive3/build/dist/lib/hive-common-0.8.0-SNAPSHOT.jar!/hive-log4j.properties
Hive history file=/tmp/carl/hive_job_log_carl_201107062017_907171289.txt
hive> SELECT * FROM tmp WHERE x > 1;
SELECT * FROM tmp WHERE x > 1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Execution log at: /tmp/carl/carl_20110706201717_5294c2b7-1cdc-40ab-9aeb-d1811eb8c486.log
java.io.FileNotFoundException: File /Users/carl/Work/repos/hive3/build/dist/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar:/Users/carl/Work/repos/hive3/build/dist/lib/hive-contrib-0.8.0-SNAPSHOT.jar does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:509)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:629)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:409)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:663)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Job Submission failed with exception 'java.io.FileNotFoundException(File /Users/carl/Work/repos/hive3/build/dist/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar:/Users/carl/Work/repos/hive3/build/dist/lib/hive-contrib-0.8.0-SNAPSHOT.jar does not exist.)'
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
hive>
{code}
> Hive --auxpath option can't handle multiple colon separated values
> ------------------------------------------------------------------
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.7.0, 0.7.1
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2269) Hive --auxpath option can't handle
multiple colon separated values
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2269:
---------------------------------
Status: Patch Available (was: Open)
With the patch in the place this works:
{code}
% hive --auxpath "::;;,;,$HIVE_HOME/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar:,:$HIVE_HOME/lib/hive-contrib-0.8.0-SNAPSHOT.jar,::"
Logging initialized using configuration in jar:file:/Users/carl/Work/repos/hive3/build/dist/lib/hive-common-0.8.0-SNAPSHOT.jar!/hive-log4j.properties
Hive history file=/tmp/carl/hive_job_log_carl_201107062104_2080798370.txt
hive> show tables;
show tables;
OK
tmp
Time taken: 3.573 seconds
hive> select * from tmp where x > 1;
select * from tmp where x > 1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Execution log at: /tmp/carl/carl_20110706210404_ad7de6b9-1b72-4477-bb97-08f3baf1b43f.log
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 0; number of reducers: 0
2011-07-06 21:04:19,185 null map = 100%, reduce = 0%
Ended Job = job_local_0001
OK
Time taken: 5.292 seconds
{code}
> Hive --auxpath option can't handle multiple colon separated values
> ------------------------------------------------------------------
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.7.1, 0.7.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-2269-auxpath.1.patch.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2269) Hive --auxpath option can't handle
multiple colon separated values
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HIVE-2269:
------------------------------
Status: Open (was: Patch Available)
Canceling patch until questions about what a valid auxpath is are answered.
> Hive --auxpath option can't handle multiple colon separated values
> ------------------------------------------------------------------
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.7.1, 0.7.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-2269-auxpath.1.patch.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle
multiple colon separated values
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061513#comment-13061513 ]
Edward Capriolo commented on HIVE-2269:
---------------------------------------
I am not sure this is documented anywhere. As a hadoop historian, I am guessing ',' was chosen to avoid the separator/path separator differences on different OS's. I do not see a problem supporting columns other then how it might effect the dedicated cygwin/hive users.
> Hive --auxpath option can't handle multiple colon separated values
> ------------------------------------------------------------------
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.7.0, 0.7.1
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-2269-auxpath.1.patch.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle
multiple colon separated values
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061636#comment-13061636 ]
Patrick Hunt commented on HIVE-2269:
------------------------------------
Hi. It looks like HIVE_AUX_JARS_PATH (hive --auxpath ...) can be either a directory (if so it considers all *.jar in the directory) or a list. In the case it's a list, can the entries be only jars or mixed jars & directories?
I can't find any documentation on this, perhaps a JIRA should be opened to document the semantics of this parameter?
> Hive --auxpath option can't handle multiple colon separated values
> ------------------------------------------------------------------
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.7.0, 0.7.1
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-2269-auxpath.1.patch.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle
multiple colon separated values
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061455#comment-13061455 ]
Carl Steinbach commented on HIVE-2269:
--------------------------------------
bq. auxpath is 'comma' separate but the title says 'colon'
@Ed: Where is this documented? Regardless, with this patch the value supplied to --auxpath can be either comma or colon separated.
> Hive --auxpath option can't handle multiple colon separated values
> ------------------------------------------------------------------
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.7.0, 0.7.1
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-2269-auxpath.1.patch.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HIVE-2269) Hive --auxpath option
can't handle multiple colon separated values
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061031#comment-13061031 ]
Carl Steinbach edited comment on HIVE-2269 at 7/7/11 4:07 AM:
--------------------------------------------------------------
With the patch in place this works:
{code}
% hive --auxpath "::;;,;,$HIVE_HOME/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar:,:$HIVE_HOME/lib/hive-contrib-0.8.0-SNAPSHOT.jar,::"
Logging initialized using configuration in jar:file:/Users/carl/Work/repos/hive3/build/dist/lib/hive-common-0.8.0-SNAPSHOT.jar!/hive-log4j.properties
Hive history file=/tmp/carl/hive_job_log_carl_201107062104_2080798370.txt
hive> show tables;
show tables;
OK
tmp
Time taken: 3.573 seconds
hive> select * from tmp where x > 1;
select * from tmp where x > 1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Execution log at: /tmp/carl/carl_20110706210404_ad7de6b9-1b72-4477-bb97-08f3baf1b43f.log
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 0; number of reducers: 0
2011-07-06 21:04:19,185 null map = 100%, reduce = 0%
Ended Job = job_local_0001
OK
Time taken: 5.292 seconds
{code}
was (Author: cwsteinbach):
With the patch in the place this works:
{code}
% hive --auxpath "::;;,;,$HIVE_HOME/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar:,:$HIVE_HOME/lib/hive-contrib-0.8.0-SNAPSHOT.jar,::"
Logging initialized using configuration in jar:file:/Users/carl/Work/repos/hive3/build/dist/lib/hive-common-0.8.0-SNAPSHOT.jar!/hive-log4j.properties
Hive history file=/tmp/carl/hive_job_log_carl_201107062104_2080798370.txt
hive> show tables;
show tables;
OK
tmp
Time taken: 3.573 seconds
hive> select * from tmp where x > 1;
select * from tmp where x > 1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Execution log at: /tmp/carl/carl_20110706210404_ad7de6b9-1b72-4477-bb97-08f3baf1b43f.log
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 0; number of reducers: 0
2011-07-06 21:04:19,185 null map = 100%, reduce = 0%
Ended Job = job_local_0001
OK
Time taken: 5.292 seconds
{code}
> Hive --auxpath option can't handle multiple colon separated values
> ------------------------------------------------------------------
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.7.0, 0.7.1
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-2269-auxpath.1.patch.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2269) Hive --auxpath option can't handle
multiple colon separated values
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2269:
---------------------------------
Attachment: HIVE-2269-auxpath.1.patch.txt
> Hive --auxpath option can't handle multiple colon separated values
> ------------------------------------------------------------------
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.7.0, 0.7.1
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-2269-auxpath.1.patch.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2269) Hive --auxpath option can't handle
multiple colon separated values
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061520#comment-13061520 ]
Edward Capriolo commented on HIVE-2269:
---------------------------------------
Also it seems that the AUX_PATH is being specified as URI's.
{noformat}
https://github.com/riptano/hive/wiki/Cassandra-Handler-usage-in-Hive-0.7-with-Cassandra-0.7
{noformat}
{noformat}
<property> <name>hive.aux.jars.path</name> <value>file:///path/to/guava-r05.jar,file:///path/to/commons-lang-2.4.jar,file:///path/to/apache-cassandra-0.7.6.jar,file:///path/to/hive-cassandra-handler-0.7.0-beta1.jar</value> </property>
{noformat}
So splitting on ':' could break URI's?
> Hive --auxpath option can't handle multiple colon separated values
> ------------------------------------------------------------------
>
> Key: HIVE-2269
> URL: https://issues.apache.org/jira/browse/HIVE-2269
> Project: Hive
> Issue Type: Bug
> Components: CLI
> Affects Versions: 0.7.0, 0.7.1
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-2269-auxpath.1.patch.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira