You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Kevin Wilfong (JIRA)" <ji...@apache.org> on 2011/09/07 01:53:10 UTC
[jira] [Created] (HIVE-2428) ADD ARCHIVE has no effect when query
runs in local mode
ADD ARCHIVE has no effect when query runs in local mode
-------------------------------------------------------
Key: HIVE-2428
URL: https://issues.apache.org/jira/browse/HIVE-2428
Project: Hive
Issue Type: Bug
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
For example running the commands
echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
when the query is allowed to run in local mode, only information for foo.sh is returned.
However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2428) ADD ARCHIVE has no effect when query
runs in local mode
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-2428:
--------------------------------
Attachment: HIVE-2428.1.patch.txt
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2428.1.patch.txt
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2428) ADD ARCHIVE has no effect when query
runs in local mode
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2428:
------------------------------
Attachment: HIVE-2428.D2121.2.patch
kevinwilfong updated the revision "HIVE-2428 [jira] ADD ARCHIVE has no effect when query runs in local mode".
Reviewers: JIRA, njain, ashutoshc
Modified add_archive_local_script to make it deterministic.
REVISION DETAIL
https://reviews.facebook.net/D2121
AFFECTED FILES
data/files/add_archive_local.zip
data/scripts/add_archive_local_script
ql/src/test/results/clientpositive/add_archive_local.q.out
ql/src/test/queries/clientpositive/add_archive_local.q
ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java
To: JIRA, njain, ashutoshc, kevinwilfong
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: add_archive_local.zip, ASF.LICENSE.NOT.GRANTED--HIVE-2428.D2121.1.patch, HIVE-2428.1.patch.txt, HIVE-2428.D2121.2.patch
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2428) ADD ARCHIVE has no effect when query
runs in local mode
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2428:
------------------------------
Attachment: HIVE-2428.D2121.1.patch
kevinwilfong requested code review of "HIVE-2428 [jira] ADD ARCHIVE has no effect when query runs in local mode".
Reviewers: JIRA
https://issues.apache.org/jira/browse/HIVE-2428
Added the -archives flag to the command run when a map reduce task is run locally. It runs essentially exactly as it does for the -files flag. This fixes the issue, I have added a test to verify that.
When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
For example running the commands
echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM USING "sh foo.sh" FROM src'
when the query is allowed to run in local mode, only information for foo.sh is returned.
However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D2121
AFFECTED FILES
data/files/add_archive_local.zip
data/scripts/add_archive_local_script
ql/src/test/results/clientpositive/add_archive_local.q.out
ql/src/test/queries/clientpositive/add_archive_local.q
ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/4599/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2428.1.patch.txt, HIVE-2428.D2121.1.patch
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2428) ADD ARCHIVE has no effect when query
runs in local mode
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen reassigned HIVE-2428:
--------------------------------
Assignee: Kevin Wilfong (was: Scott Chen)
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2428.1.patch.txt
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2428) ADD ARCHIVE has no effect when query
runs in local mode
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen reassigned HIVE-2428:
--------------------------------
Assignee: Scott Chen (was: Kevin Wilfong)
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Scott Chen
> Attachments: HIVE-2428.1.patch.txt
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HIVE-2428) ADD ARCHIVE has no effect when
query runs in local mode
Posted by "Kevin Wilfong (Work started) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HIVE-2428 started by Kevin Wilfong.
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2428.1.patch.txt
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2428) ADD ARCHIVE has no effect when query
runs in local mode
Posted by "Kevin Wilfong (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-2428:
--------------------------------
Status: Patch Available (was: Open)
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2428.1.patch.txt, HIVE-2428.D2121.1.patch, add_archive_local.zip
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2428) ADD ARCHIVE has no effect when query
runs in local mode
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222852#comment-13222852 ]
Phabricator commented on HIVE-2428:
-----------------------------------
kevinwilfong has commented on the revision "HIVE-2428 [jira] ADD ARCHIVE has no effect when query runs in local mode".
Attached the zip file used by the test to the JIRA.
Before this fix was committed, add_archive_local.zip would not appear in the output when the test q file is run.
REVISION DETAIL
https://reviews.facebook.net/D2121
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2428.1.patch.txt, HIVE-2428.D2121.1.patch, add_archive_local.zip
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work stopped] (HIVE-2428) ADD ARCHIVE has no effect when
query runs in local mode
Posted by "Kevin Wilfong (Work stopped) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HIVE-2428 stopped by Kevin Wilfong.
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2428.1.patch.txt
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2428) ADD ARCHIVE has no effect when query
runs in local mode
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098488#comment-13098488 ]
jiraposter@reviews.apache.org commented on HIVE-2428:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1731/
-----------------------------------------------------------
Review request for hive, Scott Chen and Ning Zhang.
Summary
-------
Added the -archives flag to the command run when a map reduce task is run locally. It runs essentially exactly as it does for the -files flag.
This addresses bug HIVE-2428.
https://issues.apache.org/jira/browse/HIVE-2428
Diffs
-----
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 1164711
Diff: https://reviews.apache.org/r/1731/diff
Testing
-------
I run the unit test queries to verify nothing was broken.
I also ran the query specified in the issues description and verified that information was returned for both files and archives in both local and non-local mode.
Thanks,
Kevin
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2428.1.patch.txt
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2428) ADD ARCHIVE has no effect when query
runs in local mode
Posted by "Kevin Wilfong (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-2428:
--------------------------------
Attachment: add_archive_local.zip
> ADD ARCHIVE has no effect when query runs in local mode
> -------------------------------------------------------
>
> Key: HIVE-2428
> URL: https://issues.apache.org/jira/browse/HIVE-2428
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2428.1.patch.txt, HIVE-2428.D2121.1.patch, add_archive_local.zip
>
>
> When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all.
> For example running the commands
> echo '#!/bin/bash\nls -l 1>&2;read' > foo.sh
> hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING "sh foo.sh" FROM src'
> when the query is allowed to run in local mode, only information for foo.sh is returned.
> However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira