You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Lars Francke (Created) (JIRA)" <ji...@apache.org> on 2011/12/14 15:51:30 UTC
[jira] [Created] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
hive-drop-import-delims doesn't work with MySQL direct import
-------------------------------------------------------------
Key: SQOOP-409
URL: https://issues.apache.org/jira/browse/SQOOP-409
Project: Sqoop
Issue Type: Bug
Components: hive-integration
Affects Versions: 1.4.0-incubating
Reporter: Lars Francke
When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
I tried looking at the code but it wasn't easy to follow.
There seem to be at least two problems:
* The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
* This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
Posted by "Bilung Lee (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bilung Lee updated SQOOP-409:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks, Jarcec! Patch committed.
> hive-drop-import-delims doesn't work with MySQL direct import
> -------------------------------------------------------------
>
> Key: SQOOP-409
> URL: https://issues.apache.org/jira/browse/SQOOP-409
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.0-incubating
> Reporter: Lars Francke
> Assignee: Jarek Jarcec Cecho
> Fix For: 1.4.1-incubating
>
> Attachments: SQOOP-409.patch, SQOOP-409.patch
>
>
> When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
> I tried looking at the code but it wasn't easy to follow.
> There seem to be at least two problems:
> * The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
> * This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
Posted by "Jarek Jarcec Cecho (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186221#comment-13186221 ]
Jarek Jarcec Cecho commented on SQOOP-409:
------------------------------------------
Hi Lars,
this is actually general problem of the direct connectors that are "kind of" different than normal connectors. We're going to fix this discrepancy between different connectors in Sqoop 2. Please try to avoid using --direct option in case that you need to drop hive delimiters if possible for Sqoop 1.x.
Jarcec
> hive-drop-import-delims doesn't work with MySQL direct import
> -------------------------------------------------------------
>
> Key: SQOOP-409
> URL: https://issues.apache.org/jira/browse/SQOOP-409
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.0-incubating
> Reporter: Lars Francke
>
> When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
> I tried looking at the code but it wasn't easy to follow.
> There seem to be at least two problems:
> * The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
> * This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
Posted by "Jarek Jarcec Cecho (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho updated SQOOP-409:
-------------------------------------
Attachment: SQOOP-409.patch
> hive-drop-import-delims doesn't work with MySQL direct import
> -------------------------------------------------------------
>
> Key: SQOOP-409
> URL: https://issues.apache.org/jira/browse/SQOOP-409
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.0-incubating
> Reporter: Lars Francke
> Assignee: Jarek Jarcec Cecho
> Fix For: 1.4.1-incubating
>
> Attachments: SQOOP-409.patch, SQOOP-409.patch
>
>
> When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
> I tried looking at the code but it wasn't easy to follow.
> There seem to be at least two problems:
> * The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
> * This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188159#comment-13188159 ]
Hudson commented on SQOOP-409:
------------------------------
Integrated in Sqoop-ant-jdk-1.6 #78 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6/78/])
SQOOP-409 hive-drop-import-delims doesn't work with MySQL direct import
blee : http://svn.apache.org/viewvc/?view=rev&rev=1232653
Files :
* /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/ImportTool.java
> hive-drop-import-delims doesn't work with MySQL direct import
> -------------------------------------------------------------
>
> Key: SQOOP-409
> URL: https://issues.apache.org/jira/browse/SQOOP-409
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.0-incubating
> Reporter: Lars Francke
> Assignee: Jarek Jarcec Cecho
> Fix For: 1.4.1-incubating
>
> Attachments: SQOOP-409.patch, SQOOP-409.patch
>
>
> When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
> I tried looking at the code but it wasn't easy to follow.
> There seem to be at least two problems:
> * The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
> * This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186229#comment-13186229 ]
jiraposter@reviews.apache.org commented on SQOOP-409:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3506/
-----------------------------------------------------------
Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee.
Summary
-------
I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.
This addresses bug SQOOP-409.
https://issues.apache.org/jira/browse/SQOOP-409
Diffs
-----
/src/java/org/apache/sqoop/tool/ImportTool.java 1231506
Diff: https://reviews.apache.org/r/3506/diff
Testing
-------
Thanks,
Jarek
> hive-drop-import-delims doesn't work with MySQL direct import
> -------------------------------------------------------------
>
> Key: SQOOP-409
> URL: https://issues.apache.org/jira/browse/SQOOP-409
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.0-incubating
> Reporter: Lars Francke
>
> When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
> I tried looking at the code but it wasn't easy to follow.
> There seem to be at least two problems:
> * The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
> * This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
Posted by "Jarek Jarcec Cecho (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho updated SQOOP-409:
-------------------------------------
Attachment: SQOOP-409.patch
> hive-drop-import-delims doesn't work with MySQL direct import
> -------------------------------------------------------------
>
> Key: SQOOP-409
> URL: https://issues.apache.org/jira/browse/SQOOP-409
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.0-incubating
> Reporter: Lars Francke
> Assignee: Jarek Jarcec Cecho
> Fix For: 1.4.1-incubating
>
> Attachments: SQOOP-409.patch
>
>
> When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
> I tried looking at the code but it wasn't easy to follow.
> There seem to be at least two problems:
> * The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
> * This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
Posted by "Jarek Jarcec Cecho (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho updated SQOOP-409:
-------------------------------------
Fix Version/s: 1.4.1-incubating
Assignee: Jarek Jarcec Cecho
Status: Patch Available (was: Open)
I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.
> hive-drop-import-delims doesn't work with MySQL direct import
> -------------------------------------------------------------
>
> Key: SQOOP-409
> URL: https://issues.apache.org/jira/browse/SQOOP-409
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.0-incubating
> Reporter: Lars Francke
> Assignee: Jarek Jarcec Cecho
> Fix For: 1.4.1-incubating
>
>
> When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
> I tried looking at the code but it wasn't easy to follow.
> There seem to be at least two problems:
> * The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
> * This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188131#comment-13188131 ]
jiraposter@reviews.apache.org commented on SQOOP-409:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3506/#review4431
-----------------------------------------------------------
Ship it!
- Bilung
On 2012-01-17 18:59:07, Jarek Cecho wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3506/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-01-17 18:59:07)
bq.
bq.
bq. Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.
bq.
bq.
bq. This addresses bug SQOOP-409.
bq. https://issues.apache.org/jira/browse/SQOOP-409
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /src/java/org/apache/sqoop/tool/ImportTool.java 1231506
bq.
bq. Diff: https://reviews.apache.org/r/3506/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Jarek
bq.
bq.
> hive-drop-import-delims doesn't work with MySQL direct import
> -------------------------------------------------------------
>
> Key: SQOOP-409
> URL: https://issues.apache.org/jira/browse/SQOOP-409
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.0-incubating
> Reporter: Lars Francke
> Assignee: Jarek Jarcec Cecho
> Fix For: 1.4.1-incubating
>
> Attachments: SQOOP-409.patch, SQOOP-409.patch
>
>
> When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
> I tried looking at the code but it wasn't easy to follow.
> There seem to be at least two problems:
> * The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
> * This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187886#comment-13187886 ]
jiraposter@reviews.apache.org commented on SQOOP-409:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3506/
-----------------------------------------------------------
(Updated 2012-01-17 18:59:07.118101)
Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee.
Changes
-------
Thank you for the comments sir, I've completely missed check style step before uploading this patch :-( Sorry about that.
Summary
-------
I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.
This addresses bug SQOOP-409.
https://issues.apache.org/jira/browse/SQOOP-409
Diffs (updated)
-----
/src/java/org/apache/sqoop/tool/ImportTool.java 1231506
Diff: https://reviews.apache.org/r/3506/diff
Testing
-------
Thanks,
Jarek
> hive-drop-import-delims doesn't work with MySQL direct import
> -------------------------------------------------------------
>
> Key: SQOOP-409
> URL: https://issues.apache.org/jira/browse/SQOOP-409
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.0-incubating
> Reporter: Lars Francke
> Assignee: Jarek Jarcec Cecho
> Fix For: 1.4.1-incubating
>
> Attachments: SQOOP-409.patch, SQOOP-409.patch
>
>
> When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
> I tried looking at the code but it wasn't easy to follow.
> There seem to be at least two problems:
> * The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
> * This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-409) hive-drop-import-delims doesn't work
with MySQL direct import
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187181#comment-13187181 ]
jiraposter@reviews.apache.org commented on SQOOP-409:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3506/#review4407
-----------------------------------------------------------
Look good overall. Two nits below.
/src/java/org/apache/sqoop/tool/ImportTool.java
<https://reviews.apache.org/r/3506/#comment9922>
Add a space after "if".
/src/java/org/apache/sqoop/tool/ImportTool.java
<https://reviews.apache.org/r/3506/#comment9921>
"+" should be on a new line (from checkstyle).
- Bilung
On 2012-01-14 15:09:02, Jarek Cecho wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3506/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-01-14 15:09:02)
bq.
bq.
bq. Review request for Sqoop, Arvind Prabhakar, Alex Newman, and Bilung Lee.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. I've tweaked parameter handling to rise an exception in case that both --direct and --hive-drop-import-delims are present on the command line.
bq.
bq.
bq. This addresses bug SQOOP-409.
bq. https://issues.apache.org/jira/browse/SQOOP-409
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /src/java/org/apache/sqoop/tool/ImportTool.java 1231506
bq.
bq. Diff: https://reviews.apache.org/r/3506/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Jarek
bq.
bq.
> hive-drop-import-delims doesn't work with MySQL direct import
> -------------------------------------------------------------
>
> Key: SQOOP-409
> URL: https://issues.apache.org/jira/browse/SQOOP-409
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.0-incubating
> Reporter: Lars Francke
> Assignee: Jarek Jarcec Cecho
> Fix For: 1.4.1-incubating
>
> Attachments: SQOOP-409.patch
>
>
> When using {{hive-drop-import-delims}} together with {{direct}} on a MySQL database the delimiters are not dropped. I have not tested PostgreSQL yet.
> I tried looking at the code but it wasn't easy to follow.
> There seem to be at least two problems:
> * The logic in {{MySQLDumpMapper.map}} only looks for {{MySQLUtils.outputDelimsAreMySQL(conf)}} to decide if it needs to be reparsed or not, this should also include the hive options
> * This is just a guess at the real problem: The {{ReparsingStreamThread}} calls a {{RecordParser}} directly without using the created ORM classes and thus circumventing the calls to {{hiveStringDropDelims}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira