You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Jhovanny (JIRA)" <ji...@apache.org> on 2012/08/06 03:54:02 UTC
[jira] [Created] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Jhovanny created SQOOP-539:
------------------------------
Summary: getPrimaryKeyQuery is very slow in mysql 5.0.77
Key: SQOOP-539
URL: https://issues.apache.org/jira/browse/SQOOP-539
Project: Sqoop
Issue Type: Bug
Components: connectors/mysql
Affects Versions: 1.4.2, 2.0.0
Environment: mysql 5.0.77-percona-highperfb0805
Reporter: Jhovanny
Priority: Minor
Fix For: 1.4.2, 2.0.0
The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Jhovanny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jhovanny updated SQOOP-539:
---------------------------
Attachment: patch
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch, patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Abhijeet Gaikwad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429690#comment-13429690 ]
Abhijeet Gaikwad commented on SQOOP-539:
----------------------------------------
Patch looks good to me now.
As this is just a query change at one place, I feel we can bypass review board.
For me,
ant test -Dhadoopversion=100 [success]
ant checkstyle [no errors]
In all -
+1 from my side
But we need to wait till one of the committers approves this.
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch, patch, patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429983#comment-13429983 ]
Hudson commented on SQOOP-539:
------------------------------
Integrated in Sqoop-ant-jdk-1.6-hadoop100 #5 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/5/])
SQOOP-539. getPrimaryKeyQuery is very slow in mysql 5.0.77.
(Jhovanny via Jarek Jarcec Cecho) (Revision 1370124)
Result = FAILURE
jarcec :
Files :
* /sqoop/trunk/src/java/org/apache/sqoop/manager/MySQLManager.java
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Assignee: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch, patch, patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho resolved SQOOP-539.
--------------------------------------
Resolution: Fixed
Assignee: Jhovanny
Committed revision 1370124.
Thanks Jhovanny for your contribution!
Jarcec
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Assignee: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch, patch, patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Jhovanny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430301#comment-13430301 ]
Jhovanny commented on SQOOP-539:
--------------------------------
Thanks to Abhijeet and Jarek for such a prompt response. Happy to help.
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Assignee: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch, patch, patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Abhijeet Gaikwad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429229#comment-13429229 ]
Abhijeet Gaikwad commented on SQOOP-539:
----------------------------------------
In Sqoop InformationSchemaManager class is a super of two managers namely - SQLServerManager and MySqlManager.
MySql contains column COLUMN_KEY in INFORMATION_SCHEMA.COLUMNS (http://dev.mysql.com/doc/refman/5.1/en/columns-table.html), but it seems SqlServer does not support this column (http://msdn.microsoft.com/en-us/library/ms188348%28v=sql.105%29.aspx). So this change will fail the SqlServer flow.
What I propose is to make this change specific to Mysql by overriding getPrimaryKeyQuery() in MySqlManger class. Let me know.
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429957#comment-13429957 ]
Jarek Jarcec Cecho commented on SQOOP-539:
------------------------------------------
Thanks Abhijeet for testing and Jhovany for the patch. I do not have any issues with it, so I'm giving +1 and I'll commit it shortly.
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch, patch, patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Jhovanny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jhovanny updated SQOOP-539:
---------------------------
Attachment: patch
Thanks Abhijeet. Per your suggestion, I moved the code down to MySqlManager and tested it both with --direct and without. It still works properly. I'm attaching the updated patch file.
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch, patch, patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Jhovanny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jhovanny updated SQOOP-539:
---------------------------
Attachment: patch
Patch for query change
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.2, 2.0.0
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2, 2.0.0
>
> Attachments: patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429988#comment-13429988 ]
Hudson commented on SQOOP-539:
------------------------------
Integrated in Sqoop-ant-jdk-1.6-hadoop23 #137 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/137/])
SQOOP-539. getPrimaryKeyQuery is very slow in mysql 5.0.77.
(Jhovanny via Jarek Jarcec Cecho) (Revision 1370124)
Result = SUCCESS
jarcec :
Files :
* /sqoop/trunk/src/java/org/apache/sqoop/manager/MySQLManager.java
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Assignee: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch, patch, patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Abhijeet Gaikwad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429724#comment-13429724 ]
Abhijeet Gaikwad commented on SQOOP-539:
----------------------------------------
Just for committers info:
Also verified that the query works successfully on mysql command-line and via Sqoop.
Thanks Jhovanny for the patch.
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch, patch, patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Abhijeet Gaikwad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhijeet Gaikwad updated SQOOP-539:
-----------------------------------
Affects Version/s: (was: 1.4.2)
(was: 2.0.0)
1.4.1-incubating
Fix Version/s: (was: 2.0.0)
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-539) getPrimaryKeyQuery is very slow in
mysql 5.0.77
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429981#comment-13429981 ]
Hudson commented on SQOOP-539:
------------------------------
Integrated in Sqoop-ant-jdk-1.6-hadoop20 #6 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/6/])
SQOOP-539. getPrimaryKeyQuery is very slow in mysql 5.0.77.
(Jhovanny via Jarek Jarcec Cecho) (Revision 1370124)
Result = FAILURE
jarcec :
Files :
* /sqoop/trunk/src/java/org/apache/sqoop/manager/MySQLManager.java
> getPrimaryKeyQuery is very slow in mysql 5.0.77
> -----------------------------------------------
>
> Key: SQOOP-539
> URL: https://issues.apache.org/jira/browse/SQOOP-539
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.1-incubating
> Environment: mysql 5.0.77-percona-highperfb0805
> Reporter: Jhovanny
> Assignee: Jhovanny
> Priority: Minor
> Labels: performance
> Fix For: 1.4.2
>
> Attachments: patch, patch, patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The current query used to get the primary key for a table takes 20 seconds in my production environment and 11 minutes in my QA environment. I simplified it and it now runs very fast (0.08 seconds in my production environment). I'm submitting the patch I applied locally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira