You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Aaron Kimball (JIRA)" <ji...@apache.org> on 2009/12/18 02:34:18 UTC
[jira] Created: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
CREATE TABLE statements for Hive do not correctly specify delimiters
--------------------------------------------------------------------
Key: MAPREDUCE-1310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/sqoop
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795731#action_12795731 ]
Hadoop QA commented on MAPREDUCE-1310:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429236/MAPREDUCE-1310.2.patch
against trunk revision 894962.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/352/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/352/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/352/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/352/console
This message is automatically generated.
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1310.2.patch, MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1310:
-------------------------------------
Status: Patch Available (was: Open)
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1310.2.patch, MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1310:
-------------------------------------
Status: Patch Available (was: Open)
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795800#action_12795800 ]
Aaron Kimball commented on MAPREDUCE-1310:
------------------------------------------
Test failures are unrelated -- it looks like Hudson's classpath is broken.
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1310.2.patch, MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797154#action_12797154 ]
Hudson commented on MAPREDUCE-1310:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #197 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/197/])
. CREATE TABLE statements for Hive do not correctly specify delimiters. Contributed by Aaron Kimball.
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1310.2.patch, MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792282#action_12792282 ]
Aaron Kimball commented on MAPREDUCE-1310:
------------------------------------------
Note that the method where octal values are parsed (tracing through the parsing of CREATE TABLE statements in Hive) is {{org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.unescapeSQLString()}}.
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792651#action_12792651 ]
Hadoop QA commented on MAPREDUCE-1310:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428386/MAPREDUCE-1310.patch
against trunk revision 892178.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/219/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/219/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/219/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/219/console
This message is automatically generated.
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1310:
-------------------------------------
Attachment: MAPREDUCE-1310.2.patch
good catch. new patch.
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1310.2.patch, MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1310:
-------------------------------------
Status: Open (was: Patch Available)
{noformat}
+ String octalStr = Integer.toOctalString(charNum);
+ while (octalStr.length() < 3) {
+ octalStr = "0" + octalStr;
+ }
+
+ return "\\" + octalStr;
{noformat}
This can be replaced with {noformat}return String.format("\\%03o", charNum);{noformat} or (more efficiently) reuse a static, threadlocal Formatter.
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1310:
-------------------------------------
Attachment: MAPREDUCE-1310.patch
Hive expects delimiter characters expressed as octal values to be exactly three digits, and do not require an additional '0' prefix character. As a result, octal strings such as {{'\01'}} generated by Sqoop do not get parsed correctly by Hive.
This patch adds a new test case file; committing this requires running:
{code}
svn add src/contrib/sqoop/src/test/org/apache/hadoop/sqoop/hive/TestTableDefWriter.java
{code}
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1310:
---------------------------------
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Aaron!
bq. Test failures are unrelated - it looks like Hudson's classpath is broken.
Subsequent patch builds on this machine have not had any NoClassDefFoundErrors, so I'm not sure if the problem is still there.
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1310.2.patch, MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1310) CREATE TABLE statements for Hive
do not correctly specify delimiters
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792685#action_12792685 ]
Aaron Kimball commented on MAPREDUCE-1310:
------------------------------------------
These test failures are unrelated
> CREATE TABLE statements for Hive do not correctly specify delimiters
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-1310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1310.patch
>
>
> Imports to HDFS via Sqoop that also inject metadata into Hive do not correctly specify delimiters; using Hive to access the data results in rows being parsed as NULL characters. See http://getsatisfaction.com/cloudera/topics/sqoop_hive_import_giving_null_query_values for an example bug report
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.