You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (Created) (JIRA)" <ji...@apache.org> on 2012/01/18 21:00:40 UTC
[jira] [Created] (PIG-2482) Integrate HCat DDL command into Pig
Integrate HCat DDL command into Pig
-----------------------------------
Key: PIG-2482
URL: https://issues.apache.org/jira/browse/PIG-2482
Project: Pig
Issue Type: New Feature
Components: impl
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.11
We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
Grunt> hcat create table .....
Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2482) Integrate HCat DDL command into Pig
Posted by "Daniel Dai (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai resolved PIG-2482.
-----------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Unit test pass. test-patch:
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 10 new or modified tests.
[exec]
[exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 release audit. The applied patch generated 526 release audit warnings (more than the trunk's current 517 warnings).
javadoc warning seems unrelated. All new file has proper head, ignore release audit warning.
> Integrate HCat DDL command into Pig
> -----------------------------------
>
> Key: PIG-2482
> URL: https://issues.apache.org/jira/browse/PIG-2482
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.11
>
> Attachments: PIG-2482-1.patch, PIG-2482-2.patch, PIG-2482-3.patch
>
>
> We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
> Grunt> hcat create table .....
> Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2482) Integrate HCat DDL command into Pig
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195289#comment-13195289 ]
Daniel Dai commented on PIG-2482:
---------------------------------
Sure I can improve the error message.
For HADOOP_CLASSPATH, when we run pig, bin/pig script will put pig-withouthadoop.jar into HADOOP_CLASSPATH, which contains antlr.jar. hcat script will take HADOOP_CLASSPATH and use its antlr.jar instead of Hive's version. That's why we need to unset HADOOP_CLASSPATH before launching hcat.
> Integrate HCat DDL command into Pig
> -----------------------------------
>
> Key: PIG-2482
> URL: https://issues.apache.org/jira/browse/PIG-2482
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.11
>
> Attachments: PIG-2482-1.patch
>
>
> We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
> Grunt> hcat create table .....
> Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2482) Integrate HCat DDL command into Pig
Posted by "Alan Gates (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189189#comment-13189189 ]
Alan Gates commented on PIG-2482:
---------------------------------
I think it's fine not to add commands for it to PigServer. But what about o.a.p.scripting.Pig object? It has a fs() call. It seems like it should have an hcat() call as well.
Also, should we say "hcat" or "sql"? The latter would leave us open to extend this to exporting full SQL statements later. It is also more generic and in keeping with our goal of keeping Hadoop specifics out of Pig Latin.
> Integrate HCat DDL command into Pig
> -----------------------------------
>
> Key: PIG-2482
> URL: https://issues.apache.org/jira/browse/PIG-2482
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.11
>
>
> We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
> Grunt> hcat create table .....
> Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2482) Integrate HCat DDL command into Pig
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189386#comment-13189386 ]
Daniel Dai commented on PIG-2482:
---------------------------------
bq. I think it's fine not to add commands for it to PigServer. But what about o.a.p.scripting.Pig object? It has a fs() call. It seems like it should have an hcat() call as well.
Yes, I can include it in scription.Pig
bq. Also, should we say "hcat" or "sql"? The latter would leave us open to extend this to exporting full SQL statements later. It is also more generic and in keeping with our goal of keeping Hadoop specifics out of Pig Latin.
That's better for future extension. I will name it as "sql".
> Integrate HCat DDL command into Pig
> -----------------------------------
>
> Key: PIG-2482
> URL: https://issues.apache.org/jira/browse/PIG-2482
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.11
>
>
> We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
> Grunt> hcat create table .....
> Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2482) Integrate HCat DDL command into Pig
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2482:
----------------------------
Attachment: PIG-2482-2.patch
PIG-2482-2.patch fix the error message.
> Integrate HCat DDL command into Pig
> -----------------------------------
>
> Key: PIG-2482
> URL: https://issues.apache.org/jira/browse/PIG-2482
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.11
>
> Attachments: PIG-2482-1.patch, PIG-2482-2.patch
>
>
> We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
> Grunt> hcat create table .....
> Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2482) Integrate HCat DDL command into Pig
Posted by "Alan Gates (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195319#comment-13195319 ]
Alan Gates commented on PIG-2482:
---------------------------------
+1, looks good.
> Integrate HCat DDL command into Pig
> -----------------------------------
>
> Key: PIG-2482
> URL: https://issues.apache.org/jira/browse/PIG-2482
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.11
>
> Attachments: PIG-2482-1.patch, PIG-2482-2.patch
>
>
> We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
> Grunt> hcat create table .....
> Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2482) Integrate HCat DDL command into Pig
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2482:
----------------------------
Attachment: PIG-2482-1.patch
> Integrate HCat DDL command into Pig
> -----------------------------------
>
> Key: PIG-2482
> URL: https://issues.apache.org/jira/browse/PIG-2482
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.11
>
> Attachments: PIG-2482-1.patch
>
>
> We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
> Grunt> hcat create table .....
> Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2482) Integrate HCat DDL command into Pig
Posted by "Alan Gates (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195271#comment-13195271 ]
Alan Gates commented on PIG-2482:
---------------------------------
In general looks good. A couple of comments/questions:
In GruntParser.processSQLCommand(), the error message in the following code looks incomplete:
{code}
if (new File("hcat.bin").exists()) {
throw new IOException(hcatBin + " does not ");
}
{code}
How does removing HADOOP_CLASSPATH from the environment variable solve the problem of antlr version clashes? Doesn't hcat need HADOOP_CLASSPATH set in order to work properly?
> Integrate HCat DDL command into Pig
> -----------------------------------
>
> Key: PIG-2482
> URL: https://issues.apache.org/jira/browse/PIG-2482
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.11
>
> Attachments: PIG-2482-1.patch
>
>
> We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
> Grunt> hcat create table .....
> Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2482) Integrate HCat DDL command into Pig
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2482:
----------------------------
Attachment: PIG-2482-3.patch
Add scripting.Pig part in PIG-2482-3.patch.
> Integrate HCat DDL command into Pig
> -----------------------------------
>
> Key: PIG-2482
> URL: https://issues.apache.org/jira/browse/PIG-2482
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.11
>
> Attachments: PIG-2482-1.patch, PIG-2482-2.patch, PIG-2482-3.patch
>
>
> We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
> Grunt> hcat create table .....
> Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2482) Integrate HCat DDL command into Pig
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2482:
----------------------------
Release Note:
Pig support sql command in script/grunt/embedding.
Script/grunt:
sql create table ......;
Note the sql statement end with a ";"
embedding:
from org.apache.pig.scripting import Pig
ret = Pig.sql("""drop table if exists table_1;""")
if ret==0:
#success
Configuration:
pig.sql.type=hcat (backend of sql, hcat is the only sql backend now)
hcat.bin=/usr/local/hcat/bin/hcat (binary location for hcat)
> Integrate HCat DDL command into Pig
> -----------------------------------
>
> Key: PIG-2482
> URL: https://issues.apache.org/jira/browse/PIG-2482
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.11
>
> Attachments: PIG-2482-1.patch, PIG-2482-2.patch, PIG-2482-3.patch
>
>
> We would like to run hcat DDL command inside Pig script or Grunt. We can use a similar approach as "fs" or "sh".
> Grunt> hcat create table .....
> Similar to "fs" and "sh", we don't plan to add Java API in PigServer for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira