You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Sid Wagle <sw...@hortonworks.com> on 2016/04/08 00:43:39 UTC
Review Request 45904: Ambari upgrade fails with MySQL DB and enforce
gtid enabled
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/
-----------------------------------------------------------
Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
Bugs: AMBARI-15774
https://issues.apache.org/jira/browse/AMBARI-15774
Repository: ambari
Description
-------
Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
{code}
ambari-server upgrade
Using python /usr/bin/python
Upgrading ambari-server
Updating properties in ambari.properties ...
WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings
WARNING: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings
Fixing database objects owner
Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (y)? y
Upgrading database schema
Error output from schema upgrade command:
Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
Error Code: 1146
{code}
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertStateChangedListener.java 08563e3
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
ambari-server/src/test/java/org/apache/ambari/server/state/cluster/AlertDataManagerTest.java e8ecd09
Diff: https://reviews.apache.org/r/45904/diff/
Testing
-------
All alert unit test passed.
mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
Thanks,
Sid Wagle
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sid Wagle <sw...@hortonworks.com>.
> On April 8, 2016, 12:26 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java, lines 687-692
> > <https://reviews.apache.org/r/45904/diff/2/?file=1331569#file1331569line687>
> >
> > This is a very dangerous query. This can literally return 10,000 entries.
Thanks for pointing that out, I will explore an explicit join query as a workaround to offload it back to the DB.
For the issue regarding reccurence of this situation I suggested adding gtid enforce policy in the system tests for the use case of delete cluster, etc.
- Sid
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review127772
-----------------------------------------------------------
On April 7, 2016, 11:34 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated April 7, 2016, 11:34 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sid Wagle <sw...@hortonworks.com>.
> On April 8, 2016, 12:26 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java, lines 687-692
> > <https://reviews.apache.org/r/45904/diff/2/?file=1331569#file1331569line687>
> >
> > This is a very dangerous query. This can literally return 10,000 entries.
>
> Sid Wagle wrote:
> Thanks for pointing that out, I will explore an explicit join query as a workaround to offload it back to the DB.
> For the issue regarding reccurence of this situation I suggested adding gtid enforce policy in the system tests for the use case of delete cluster, etc.
Modified to reading ids only.
- Sid
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review127772
-----------------------------------------------------------
On April 8, 2016, 8:56 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated April 8, 2016, 8:56 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review127772
-----------------------------------------------------------
I know that this is to support GTID on MySQL and preventing temporary table usage. But my issue with this is twofold:
- Alert History tables are huge and the queries that are being made are going to be problematic.
- This doesn't prevent this type of query from sneaking into the product again and causing issues only caught in deployments
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java (lines 687 - 692)
<https://reviews.apache.org/r/45904/#comment191157>
This is a very dangerous query. This can literally return 10,000 entries.
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java (line 1491)
<https://reviews.apache.org/r/45904/#comment191158>
This can be a lot of entries and cause OOM problems with JPA; this is why we use a direct JPQL query.
- Jonathan Hurley
On April 7, 2016, 7:34 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated April 7, 2016, 7:34 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sid Wagle <sw...@hortonworks.com>.
> On April 8, 2016, 3:36 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java, line 745
> > <https://reviews.apache.org/r/45904/diff/2/?file=1331570#file1331570line745>
> >
> > Why this change? For compiler optimization???
Leftover debuggin artifact, I will edit before commitng or posting next update if needed.
> On April 8, 2016, 3:36 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java, line 938
> > <https://reviews.apache.org/r/45904/diff/2/?file=1331570#file1331570line938>
> >
> > Similar - why we are keeping count?
Mostly debugging reasons.
> On April 8, 2016, 3:36 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java, line 71
> > <https://reviews.apache.org/r/45904/diff/2/?file=1331571#file1331571line71>
> >
> > Oh! these existed - just not mapped to Entity obj
Yes.
- Sid
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review127728
-----------------------------------------------------------
On April 7, 2016, 11:34 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated April 7, 2016, 11:34 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review127728
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java (line 745)
<https://reviews.apache.org/r/45904/#comment191115>
Why this change? For compiler optimization???
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java (line 938)
<https://reviews.apache.org/r/45904/#comment191116>
Similar - why we are keeping count?
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java (line 70)
<https://reviews.apache.org/r/45904/#comment191117>
Oh! these existed - just not mapped to Entity obj
- Sumit Mohanty
On April 7, 2016, 11:34 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated April 7, 2016, 11:34 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Myroslav Papirkovskyy <mp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review127749
-----------------------------------------------------------
Ship it!
Ship It!
- Myroslav Papirkovskyy
On Квітень 8, 2016, 2:34 до полудня, Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated Квітень 8, 2016, 2:34 до полудня)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review127729
-----------------------------------------------------------
Ship it!
LGTM, but lets get some more eyes on the code.
- Sumit Mohanty
On April 7, 2016, 11:34 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated April 7, 2016, 11:34 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review128212
-----------------------------------------------------------
Ship it!
Ship It!
- Jonathan Hurley
On April 11, 2016, 2:41 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated April 11, 2016, 2:41 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/
-----------------------------------------------------------
(Updated April 11, 2016, 6:41 p.m.)
Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
Changes
-------
- Added helpful logging messages
- Added comment right where the queries were changed
Bugs: AMBARI-15774
https://issues.apache.org/jira/browse/AMBARI-15774
Repository: ambari
Description
-------
Approach:
- Added member varaibles only when underlying DB columns exist
- Changed DELETE via implcit join query to find and delete
- MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
- Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
Error output from schema upgrade command:
Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
Error Code: 1146
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
Diff: https://reviews.apache.org/r/45904/diff/
Testing
-------
Manually verified on 1 node repro cluster.
All alert unit test passed.
mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
Thanks,
Sid Wagle
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Myroslav Papirkovskyy <mp...@hortonworks.com>.
> On Квітень 11, 2016, 3:36 після полудня, Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java, lines 701-708
> > <https://reviews.apache.org/r/45904/diff/3/?file=1337530#file1337530line701>
> >
> > This solution is still far from ideal. You'll be making something like 20,000 database queries and if even 1 fails, then the entire transaction is rolled back.
> >
> > This is why we went for a pure JPQL approach here; it was clean and didn't cause any weird problems.
> >
> > This also doesn't address the fact that nothing prevents future JPQL queries from causing a similar problem.
> >
> > Perhaps we need to consider a different approach. The issue was that MySQL was using temporary tables outside of a transaction. If we can solve that problem, we can leave the existing JPQL.
This is not quite correct. Issue was that EclipseLink uses MySQL temporary tables INSIDE of transactions.
Temp tables outside of transaction are supported even with GTID enabled.
However disabling temp tables simply doesn't work. EclipseLink starts to generate invalid queries for MySQL. And after quick look at sources it seems that major modifications are required to fix this.
For MySQL temp tables are used to bypass limitations of subqueries in delete/update statements.
- Myroslav
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review128100
-----------------------------------------------------------
On Квітень 8, 2016, 11:56 після полудня, Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated Квітень 8, 2016, 11:56 після полудня)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On April 11, 2016, 8:36 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java, lines 701-708
> > <https://reviews.apache.org/r/45904/diff/3/?file=1337530#file1337530line701>
> >
> > This solution is still far from ideal. You'll be making something like 20,000 database queries and if even 1 fails, then the entire transaction is rolled back.
> >
> > This is why we went for a pure JPQL approach here; it was clean and didn't cause any weird problems.
> >
> > This also doesn't address the fact that nothing prevents future JPQL queries from causing a similar problem.
> >
> > Perhaps we need to consider a different approach. The issue was that MySQL was using temporary tables outside of a transaction. If we can solve that problem, we can leave the existing JPQL.
>
> Myroslav Papirkovskyy wrote:
> This is not quite correct. Issue was that EclipseLink uses MySQL temporary tables INSIDE of transactions.
> Temp tables outside of transaction are supported even with GTID enabled.
>
> However disabling temp tables simply doesn't work. EclipseLink starts to generate invalid queries for MySQL. And after quick look at sources it seems that major modifications are required to fix this.
> For MySQL temp tables are used to bypass limitations of subqueries in delete/update statements.
Ah, that stinks. Yeah, my hope was that telling MySQL to not use temp tables would be the ticket. But if it starts forming bad queries, that's no good either.
This is quite bad in that we really can't leverage JPQL anymore if we say we support GTID. I'm curious if running in the WARN mode would be a good balances. This way, GTID will be enabled, but some Ambari queries which violated GTID constraints can still be allowed to execute.
The current approach seems very fragile and extremely hard to enforce moving forward.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review128100
-----------------------------------------------------------
On April 8, 2016, 4:56 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated April 8, 2016, 4:56 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sid Wagle <sw...@hortonworks.com>.
> On April 11, 2016, 12:36 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java, lines 701-708
> > <https://reviews.apache.org/r/45904/diff/3/?file=1337530#file1337530line701>
> >
> > This solution is still far from ideal. You'll be making something like 20,000 database queries and if even 1 fails, then the entire transaction is rolled back.
> >
> > This is why we went for a pure JPQL approach here; it was clean and didn't cause any weird problems.
> >
> > This also doesn't address the fact that nothing prevents future JPQL queries from causing a similar problem.
> >
> > Perhaps we need to consider a different approach. The issue was that MySQL was using temporary tables outside of a transaction. If we can solve that problem, we can leave the existing JPQL.
>
> Myroslav Papirkovskyy wrote:
> This is not quite correct. Issue was that EclipseLink uses MySQL temporary tables INSIDE of transactions.
> Temp tables outside of transaction are supported even with GTID enabled.
>
> However disabling temp tables simply doesn't work. EclipseLink starts to generate invalid queries for MySQL. And after quick look at sources it seems that major modifications are required to fix this.
> For MySQL temp tables are used to bypass limitations of subqueries in delete/update statements.
>
> Jonathan Hurley wrote:
> Ah, that stinks. Yeah, my hope was that telling MySQL to not use temp tables would be the ticket. But if it starts forming bad queries, that's no good either.
>
> This is quite bad in that we really can't leverage JPQL anymore if we say we support GTID. I'm curious if running in the WARN mode would be a good balances. This way, GTID will be enabled, but some Ambari queries which violated GTID constraints can still be allowed to execute.
>
> The current approach seems very fragile and extremely hard to enforce moving forward.
Agree that we would entirely rely on system test to catch any queries in the future.
Unclear why you thing the implementation is fragile.
- Sid
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review128100
-----------------------------------------------------------
On April 8, 2016, 8:56 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated April 8, 2016, 8:56 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/#review128100
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java (lines 693 - 700)
<https://reviews.apache.org/r/45904/#comment191484>
This solution is still far from ideal. You'll be making something like 20,000 database queries and if even 1 fails, then the entire transaction is rolled back.
This is why we went for a pure JPQL approach here; it was clean and didn't cause any weird problems.
This also doesn't address the fact that nothing prevents future JPQL queries from causing a similar problem.
Perhaps we need to consider a different approach. The issue was that MySQL was using temporary tables outside of a transaction. If we can solve that problem, we can leave the existing JPQL.
- Jonathan Hurley
On April 8, 2016, 4:56 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45904/
> -----------------------------------------------------------
>
> (Updated April 8, 2016, 4:56 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15774
> https://issues.apache.org/jira/browse/AMBARI-15774
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Approach:
>
> - Added member varaibles only when underlying DB columns exist
> - Changed DELETE via implcit join query to find and delete
> - MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
> - Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
>
> Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
>
> Error output from schema upgrade command:
> Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
> Error Code: 1146
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
>
> Diff: https://reviews.apache.org/r/45904/diff/
>
>
> Testing
> -------
>
> Manually verified on 1 node repro cluster.
>
> All alert unit test passed.
>
> mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/
-----------------------------------------------------------
(Updated April 8, 2016, 8:56 p.m.)
Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
Changes
-------
- Instead of retrieving entity objects, changed to read only the ids
- Adding batch delete for AleertCurrent and AlertNotice entities with defaut batch size 1 < Oracle IN clause limit
- Added a unit test
- All uni tests passed
Bugs: AMBARI-15774
https://issues.apache.org/jira/browse/AMBARI-15774
Repository: ambari
Description
-------
Approach:
- Added member varaibles only when underlying DB columns exist
- Changed DELETE via implcit join query to find and delete
- MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
- Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
Error output from schema upgrade command:
Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
Error Code: 1146
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
Diff: https://reviews.apache.org/r/45904/diff/
Testing
-------
Manually verified on 1 node repro cluster.
All alert unit test passed.
mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
Thanks,
Sid Wagle
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/
-----------------------------------------------------------
(Updated April 7, 2016, 11:34 p.m.)
Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
Changes
-------
Remvoed some unnecessary setter calls.
Bugs: AMBARI-15774
https://issues.apache.org/jira/browse/AMBARI-15774
Repository: ambari
Description
-------
Approach:
- Added member varaibles only when underlying DB columns exist
- Changed DELETE via implcit join query to find and delete
- MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
- Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
Error output from schema upgrade command:
Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
Error Code: 1146
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
Diff: https://reviews.apache.org/r/45904/diff/
Testing
-------
Manually verified on 1 node repro cluster.
All alert unit test passed.
mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
Thanks,
Sid Wagle
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/
-----------------------------------------------------------
(Updated April 7, 2016, 11:01 p.m.)
Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
Bugs: AMBARI-15774
https://issues.apache.org/jira/browse/AMBARI-15774
Repository: ambari
Description
-------
Approach:
- Added member varaibles only when underlying DB columns exist
- Changed DELETE via implcit join query to find and delete
- MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
- Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
Error output from schema upgrade command:
Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
Error Code: 1146
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertStateChangedListener.java 08563e3
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
ambari-server/src/test/java/org/apache/ambari/server/state/cluster/AlertDataManagerTest.java e8ecd09
Diff: https://reviews.apache.org/r/45904/diff/
Testing (updated)
-------
Manually verified on 1 node repro cluster.
All alert unit test passed.
mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
Thanks,
Sid Wagle
Re: Review Request 45904: Ambari upgrade fails with MySQL DB and
enforce gtid enabled
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45904/
-----------------------------------------------------------
(Updated April 7, 2016, 10:54 p.m.)
Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Sumit Mohanty.
Bugs: AMBARI-15774
https://issues.apache.org/jira/browse/AMBARI-15774
Repository: ambari
Description (updated)
-------
Approach:
- Added member varaibles only when underlying DB columns exist
- Changed DELETE via implcit join query to find and delete
- MySQL temp tables are only used for UPDATE ALL and DELETE ALL, so SELECT and DELETE is a safer approach
- Visually inspected all Entities under org.apache.ambari.server.orm.entities.* for absence of this pattern
Issue: Ambari upgrade is failing from 2.2.0 to 2.2.1.1 with below error:
Error output from schema upgrade command:
Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.TL_alert_notice' doesn't exist
Error Code: 1146
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java a3befa6
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertStateChangedListener.java 08563e3
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java 82fa48a
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDispatchDAO.java 1f1aa45
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertsDAO.java 781d4cf
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertCurrentEntity.java 604b00e
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertHistoryEntity.java 03ffcde
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertNoticeEntity.java ae7495d
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertResourceProviderTest.java d611fe8
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAOTest.java 36e75e7
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertsDAOTest.java 10f099e
ambari-server/src/test/java/org/apache/ambari/server/state/cluster/AlertDataManagerTest.java e8ecd09
Diff: https://reviews.apache.org/r/45904/diff/
Testing
-------
All alert unit test passed.
mvn clean test -Dtest=Alert* -Drat.ignoreErrors -DfailIfNoTests=false
Thanks,
Sid Wagle