You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "Samarth Jain (JIRA)" <ji...@apache.org> on 2016/09/08 19:20:20 UTC
[jira] [Updated] (PHOENIX-3230) SYSTEM.CATALOG get restored from
snapshot with multi-client connection
[ https://issues.apache.org/jira/browse/PHOENIX-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Samarth Jain updated PHOENIX-3230:
----------------------------------
Attachment: PHOENIX-3230_nowhitespacediff.patch
Patch to make sure only client JVM is able to run the upgrade. Clients that lose the race are thrown an UpgradeInProgressException. The UpgradeInProgressException isn't marked as initException so that the clients can keep retrying on their end to establish the first connection to the cluster. I verified this by writing a simple test that keeps on requesting connection in a loop when another client is concurrently running the upgrade. Once the upgrade is done, the test was able to acquire connection successfully.
The UpgradeInProgressException stacktrace looks like this:
{code}
org.apache.phoenix.query.ConnectionQueryServicesImpl$UpgradeInProgressException: Cluster is being upgraded from 4.7.x to 4.8.x. Please retry establishing connection
at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.acquireUpgradeMutex(ConnectionQueryServicesImpl.java:2768)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2341)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:1)
at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2278)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:232)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:147)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at org.apache.phoenix.end2end.PhoenixRuntimeIT.testConnection(PhoenixRuntimeIT.java:151)
{code}
I also had to tweak the way we were looking up version string for system catalog timestamps. It turns out when doing concurrent upgrades, we could be at an intermediate timestamp. Using a navigable map to help store the timestamp->version combo helps provide a range based lookup.
I haven't added the part to store the upgrade state in a STATUS column in SYSTEM.CATALOG as I believe this mechanism ends up providing enough info to the user.
[~jamestaylor] - please review.
> SYSTEM.CATALOG get restored from snapshot with multi-client connection
> ----------------------------------------------------------------------
>
> Key: PHOENIX-3230
> URL: https://issues.apache.org/jira/browse/PHOENIX-3230
> Project: Phoenix
> Issue Type: Bug
> Reporter: Mujtaba Chohan
> Assignee: Samarth Jain
> Fix For: 4.8.1
>
> Attachments: PHOENIX-3230_nowhitespacediff.patch
>
>
> If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1 then second connection fails with the following exception. This happens even if second connection is couple of seconds apart but within upgrade window. This is likely to happen in situation where pool of client machines all get upgraded to latest Phoenix version. After this exception, all clients will cease to work with undefined column exception due to restore/aborted upgrade.
> {noformat}
> WARN query.ConnectionQueryServicesImpl: Table already modified at this timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Table already modified at this timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ VARCHAR
> WARN query.ConnectionQueryServicesImpl: Table already modified at this timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA BOOLEAN
> WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700 because upgrade failed
> 16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully restored SYSTEM.CATALOG using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> 16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully restored and enabled SYSTEM.CATALOG using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
> Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED (state=42703,code=504)
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
> at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
> at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
> at org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
> at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
> at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
> at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
> at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
> at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
> at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
> at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
> at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
> at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
> at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
> at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
> at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2282)
> at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:231)
> at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:144)
> at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)