You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by GitBox <gi...@apache.org> on 2020/02/26 23:44:32 UTC

[GitHub] [phoenix] sguggilam opened a new pull request #714: PHOENIX-5607 Client-server backward compatibility tests

sguggilam opened a new pull request #714: PHOENIX-5607 Client-server backward compatibility tests
URL: https://github.com/apache/phoenix/pull/714
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] ChinmaySKulkarni commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests

Posted by GitBox <gi...@apache.org>.
ChinmaySKulkarni commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests
URL: https://github.com/apache/phoenix/pull/714#discussion_r386662704
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java
 ##########
 @@ -145,6 +146,8 @@
     }
     
     public static final String CURRENT_CLIENT_VERSION = PHOENIX_MAJOR_VERSION + "." + PHOENIX_MINOR_VERSION + "." + PHOENIX_PATCH_NUMBER;
+    public static final List<String> COMPATIBLE_CLIENT_VERSIONS = 
 
 Review comment:
   Is there no way to get this info programatically? We will have to keep updating this version with each release right? If not, let's document it in the release webpage.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] sguggilam commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests

Posted by GitBox <gi...@apache.org>.
sguggilam commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests
URL: https://github.com/apache/phoenix/pull/714#discussion_r387305047
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java
 ##########
 @@ -145,6 +146,8 @@
     }
     
     public static final String CURRENT_CLIENT_VERSION = PHOENIX_MAJOR_VERSION + "." + PHOENIX_MINOR_VERSION + "." + PHOENIX_PATCH_NUMBER;
+    public static final List<String> COMPATIBLE_CLIENT_VERSIONS = 
 
 Review comment:
   There is no easy way to get this programatically as the number of backward compatible versions can depend on the current versions. Sometimes the first code cut of major version might not have any backward compatible clients as per the doc. I have filed a JIRA to add it to the release web page : https://issues.apache.org/jira/browse/PHOENIX-5746

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] sguggilam commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests

Posted by GitBox <gi...@apache.org>.
sguggilam commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests
URL: https://github.com/apache/phoenix/pull/714#discussion_r387305740
 
 

 ##########
 File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/BackwardCompatibilityIT.java
 ##########
 @@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import static org.apache.phoenix.query.BaseTest.setUpConfigForMiniCluster;
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assume.assumeFalse;
+
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileReader;
+import java.io.FileWriter;
+import java.io.InputStreamReader;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.curator.shaded.com.google.common.collect.Lists;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixDriver;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.query.QueryServicesOptions;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+/**
+ * This class is meant for testing all compatible client versions 
+ * against the current server version. It runs SQL queries with given 
+ * client versions and compares the output against gold files
+ */
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class BackwardCompatibilityIT {
+
+    private static final String SQL_DIR = "src/it/resources/sql_files/";
+    private static final String RESULT_DIR = "src/it/resources/gold_files/";
+    private static final String RESULT_PREFIX = "result_";
+    private static final String SQL_EXTENSION = ".sql";
+    private static final String TEXT_EXTENSION = ".txt";
+    private static final String CREATE_ADD = "create_add";
+    private static final String ADD_DATA = "add_data";
+    private static final String ADD_DELETE = "add_delete";
+    private static final String QUERY = "query";
+    private static final String QUERY_ADD_DELETE = "query_add_delete";
+
+    private final String compatibleClientVersion;
+    private static Configuration conf;
+    private static HBaseTestingUtility hbaseTestUtil;
+    private static String zkQuorum;
+    private static String url;
+
+    public BackwardCompatibilityIT(String compatibleClientVersion) {
+        this.compatibleClientVersion = compatibleClientVersion;
+    }
+
+    @Parameters(name = "BackwardCompatibilityIT_compatibleClientVersion={0}")
+    public static synchronized Collection<String> data() {
+        return MetaDataProtocol.COMPATIBLE_CLIENT_VERSIONS;
+    }
+
+    @Before
+    public synchronized void doSetup() throws Exception {
+        conf = HBaseConfiguration.create();
+        hbaseTestUtil = new HBaseTestingUtility(conf);
+        setUpConfigForMiniCluster(conf);
+        conf.set(QueryServices.EXTRA_JDBC_ARGUMENTS_ATTRIB, QueryServicesOptions.DEFAULT_EXTRA_JDBC_ARGUMENTS);
+        hbaseTestUtil.startMiniCluster();
+        zkQuorum = "localhost:" + hbaseTestUtil.getZkCluster().getClientPort();
+        url = PhoenixRuntime.JDBC_PROTOCOL + PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR + zkQuorum;
+        DriverManager.registerDriver(PhoenixDriver.INSTANCE);
+    }
+    
+    @After
+    public void cleanUpAfterTest() throws Exception {
+        try {
+            DriverManager.deregisterDriver(PhoenixDriver.INSTANCE);
+        } finally {
+            hbaseTestUtil.shutdownMiniCluster();
+        }
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with new client and read with old client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client 
+     * 4. New Client inserts more data into the tables created by old client 
+     * 5. Old Client reads the data inserted by new client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectUpsertWithNewClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Insert more data with new client and read with old client
+        executeQueriesWithCurrentVersion(ADD_DATA);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(ADD_DATA, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client 
+     * 4. Old Client inserts more data into the tables created by old client 
+     * 5. New Client reads the data inserted by new client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectUpsertWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with new client and read with old client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Insert more data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, ADD_DATA);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(ADD_DATA, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client 
+     * 4. Old Client creates and deletes the data
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertDeleteWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Deletes with the old client
+        executeQueryWithClientVersion(compatibleClientVersion, ADD_DELETE);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY_ADD_DELETE);
+        assertTrue(compareOutput(ADD_DELETE, QUERY_ADD_DELETE));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client 
+     * 4. New Client creates and deletes the data
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertDeleteWithNewClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Deletes with the new client
+        executeQueriesWithCurrentVersion(ADD_DELETE);
+        executeQueriesWithCurrentVersion(QUERY_ADD_DELETE);
+        assertTrue(compareOutput(ADD_DELETE, QUERY_ADD_DELETE));
+    }
+    
 
 Review comment:
   I think what you meant is creating views etc with a older client once the new client connects to the updated server. Yes, that makes sense and I have added more DDL to the add_data to have all these scenarios covered 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] ChinmaySKulkarni commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests

Posted by GitBox <gi...@apache.org>.
ChinmaySKulkarni commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests
URL: https://github.com/apache/phoenix/pull/714#discussion_r386665060
 
 

 ##########
 File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/BackwardCompatibilityIT.java
 ##########
 @@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import static org.apache.phoenix.query.BaseTest.setUpConfigForMiniCluster;
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assume.assumeFalse;
+
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileReader;
+import java.io.FileWriter;
+import java.io.InputStreamReader;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.curator.shaded.com.google.common.collect.Lists;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixDriver;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.query.QueryServicesOptions;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+/**
+ * This class is meant for testing all compatible client versions 
+ * against the current server version. It runs SQL queries with given 
+ * client versions and compares the output against gold files
+ */
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class BackwardCompatibilityIT {
+
+    private static final String SQL_DIR = "src/it/resources/sql_files/";
+    private static final String RESULT_DIR = "src/it/resources/gold_files/";
+    private static final String RESULT_PREFIX = "result_";
+    private static final String SQL_EXTENSION = ".sql";
+    private static final String TEXT_EXTENSION = ".txt";
+    private static final String CREATE_ADD = "create_add";
+    private static final String ADD_DATA = "add_data";
+    private static final String ADD_DELETE = "add_delete";
+    private static final String QUERY = "query";
+    private static final String QUERY_ADD_DELETE = "query_add_delete";
+
+    private final String compatibleClientVersion;
+    private static Configuration conf;
+    private static HBaseTestingUtility hbaseTestUtil;
+    private static String zkQuorum;
+    private static String url;
+
+    public BackwardCompatibilityIT(String compatibleClientVersion) {
+        this.compatibleClientVersion = compatibleClientVersion;
+    }
+
+    @Parameters(name = "BackwardCompatibilityIT_compatibleClientVersion={0}")
+    public static synchronized Collection<String> data() {
+        return MetaDataProtocol.COMPATIBLE_CLIENT_VERSIONS;
+    }
+
+    @Before
+    public synchronized void doSetup() throws Exception {
+        conf = HBaseConfiguration.create();
+        hbaseTestUtil = new HBaseTestingUtility(conf);
+        setUpConfigForMiniCluster(conf);
+        conf.set(QueryServices.EXTRA_JDBC_ARGUMENTS_ATTRIB, QueryServicesOptions.DEFAULT_EXTRA_JDBC_ARGUMENTS);
+        hbaseTestUtil.startMiniCluster();
+        zkQuorum = "localhost:" + hbaseTestUtil.getZkCluster().getClientPort();
+        url = PhoenixRuntime.JDBC_PROTOCOL + PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR + zkQuorum;
+        DriverManager.registerDriver(PhoenixDriver.INSTANCE);
+    }
+    
+    @After
+    public void cleanUpAfterTest() throws Exception {
+        try {
+            DriverManager.deregisterDriver(PhoenixDriver.INSTANCE);
+        } finally {
+            hbaseTestUtil.shutdownMiniCluster();
+        }
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with new client and read with old client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client 
+     * 4. New Client inserts more data into the tables created by old client 
+     * 5. Old Client reads the data inserted by new client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectUpsertWithNewClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Insert more data with new client and read with old client
+        executeQueriesWithCurrentVersion(ADD_DATA);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(ADD_DATA, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client 
+     * 4. Old Client inserts more data into the tables created by old client 
+     * 5. New Client reads the data inserted by new client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectUpsertWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with new client and read with old client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Insert more data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, ADD_DATA);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(ADD_DATA, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client 
+     * 4. Old Client creates and deletes the data
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertDeleteWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Deletes with the old client
+        executeQueryWithClientVersion(compatibleClientVersion, ADD_DELETE);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY_ADD_DELETE);
+        assertTrue(compareOutput(ADD_DELETE, QUERY_ADD_DELETE));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client 
+     * 4. New Client creates and deletes the data
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertDeleteWithNewClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Deletes with the new client
+        executeQueriesWithCurrentVersion(ADD_DELETE);
+        executeQueriesWithCurrentVersion(QUERY_ADD_DELETE);
+        assertTrue(compareOutput(ADD_DELETE, QUERY_ADD_DELETE));
+    }
+    
 
 Review comment:
   Most backward compatibility issues we found during the 4.15 release were related to old clients unable to perform DDL operations once the cluster was upgraded or old clients unable to query views/indexes/view indexes created by a new clients. Can we add more tests like that?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] ChinmaySKulkarni commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests

Posted by GitBox <gi...@apache.org>.
ChinmaySKulkarni commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests
URL: https://github.com/apache/phoenix/pull/714#discussion_r386663641
 
 

 ##########
 File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/BackwardCompatibilityIT.java
 ##########
 @@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import static org.apache.phoenix.query.BaseTest.setUpConfigForMiniCluster;
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assume.assumeFalse;
+
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileReader;
+import java.io.FileWriter;
+import java.io.InputStreamReader;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.curator.shaded.com.google.common.collect.Lists;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixDriver;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.query.QueryServicesOptions;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+/**
+ * This class is meant for testing all compatible client versions 
+ * against the current server version. It runs SQL queries with given 
+ * client versions and compares the output against gold files
+ */
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class BackwardCompatibilityIT {
+
+    private static final String SQL_DIR = "src/it/resources/sql_files/";
+    private static final String RESULT_DIR = "src/it/resources/gold_files/";
+    private static final String RESULT_PREFIX = "result_";
+    private static final String SQL_EXTENSION = ".sql";
+    private static final String TEXT_EXTENSION = ".txt";
+    private static final String CREATE_ADD = "create_add";
+    private static final String ADD_DATA = "add_data";
+    private static final String ADD_DELETE = "add_delete";
+    private static final String QUERY = "query";
+    private static final String QUERY_ADD_DELETE = "query_add_delete";
+
+    private final String compatibleClientVersion;
+    private static Configuration conf;
+    private static HBaseTestingUtility hbaseTestUtil;
+    private static String zkQuorum;
+    private static String url;
+
+    public BackwardCompatibilityIT(String compatibleClientVersion) {
+        this.compatibleClientVersion = compatibleClientVersion;
+    }
+
+    @Parameters(name = "BackwardCompatibilityIT_compatibleClientVersion={0}")
+    public static synchronized Collection<String> data() {
+        return MetaDataProtocol.COMPATIBLE_CLIENT_VERSIONS;
+    }
+
+    @Before
+    public synchronized void doSetup() throws Exception {
+        conf = HBaseConfiguration.create();
+        hbaseTestUtil = new HBaseTestingUtility(conf);
+        setUpConfigForMiniCluster(conf);
+        conf.set(QueryServices.EXTRA_JDBC_ARGUMENTS_ATTRIB, QueryServicesOptions.DEFAULT_EXTRA_JDBC_ARGUMENTS);
+        hbaseTestUtil.startMiniCluster();
+        zkQuorum = "localhost:" + hbaseTestUtil.getZkCluster().getClientPort();
+        url = PhoenixRuntime.JDBC_PROTOCOL + PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR + zkQuorum;
+        DriverManager.registerDriver(PhoenixDriver.INSTANCE);
+    }
+    
+    @After
+    public void cleanUpAfterTest() throws Exception {
+        try {
+            DriverManager.deregisterDriver(PhoenixDriver.INSTANCE);
+        } finally {
+            hbaseTestUtil.shutdownMiniCluster();
+        }
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with new client and read with old client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client 
+     * 4. New Client inserts more data into the tables created by old client 
+     * 5. Old Client reads the data inserted by new client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectUpsertWithNewClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Insert more data with new client and read with old client
+        executeQueriesWithCurrentVersion(ADD_DATA);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(ADD_DATA, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client 
+     * 4. Old Client inserts more data into the tables created by old client 
+     * 5. New Client reads the data inserted by new client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectUpsertWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with new client and read with old client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Insert more data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, ADD_DATA);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(ADD_DATA, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client 
+     * 4. Old Client creates and deletes the data
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertDeleteWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Deletes with the old client
+        executeQueryWithClientVersion(compatibleClientVersion, ADD_DELETE);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY_ADD_DELETE);
+        assertTrue(compareOutput(ADD_DELETE, QUERY_ADD_DELETE));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client 
+     * 4. New Client creates and deletes the data
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertDeleteWithNewClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Deletes with the new client
+        executeQueriesWithCurrentVersion(ADD_DELETE);
+        executeQueriesWithCurrentVersion(QUERY_ADD_DELETE);
+        assertTrue(compareOutput(ADD_DELETE, QUERY_ADD_DELETE));
+    }
+    
+    private void checkForPreConditions() throws Exception {
+        // For the first code cut of any major version, there wouldn't be any backward compatible
+        // clients. Hence the test wouldn't run and just return true when the client  
+        // version to be tested is same as current version
+        assumeFalse(compatibleClientVersion.contains(MetaDataProtocol.CURRENT_CLIENT_VERSION));
+        // Make sure that cluster is clean before test execution with no system tables
+        org.apache.hadoop.hbase.client.Connection conn = ConnectionFactory.createConnection(conf);
 
 Review comment:
   Put inside try-with-resources

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] sguggilam commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests

Posted by GitBox <gi...@apache.org>.
sguggilam commented on a change in pull request #714: PHOENIX-5607 Client-server backward compatibility tests
URL: https://github.com/apache/phoenix/pull/714#discussion_r387305169
 
 

 ##########
 File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/BackwardCompatibilityIT.java
 ##########
 @@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import static org.apache.phoenix.query.BaseTest.setUpConfigForMiniCluster;
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assume.assumeFalse;
+
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileReader;
+import java.io.FileWriter;
+import java.io.InputStreamReader;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.curator.shaded.com.google.common.collect.Lists;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.phoenix.coprocessor.MetaDataProtocol;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.jdbc.PhoenixDriver;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.query.QueryServicesOptions;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+/**
+ * This class is meant for testing all compatible client versions 
+ * against the current server version. It runs SQL queries with given 
+ * client versions and compares the output against gold files
+ */
+
+@RunWith(Parameterized.class)
+@Category(NeedsOwnMiniClusterTest.class)
+public class BackwardCompatibilityIT {
+
+    private static final String SQL_DIR = "src/it/resources/sql_files/";
+    private static final String RESULT_DIR = "src/it/resources/gold_files/";
+    private static final String RESULT_PREFIX = "result_";
+    private static final String SQL_EXTENSION = ".sql";
+    private static final String TEXT_EXTENSION = ".txt";
+    private static final String CREATE_ADD = "create_add";
+    private static final String ADD_DATA = "add_data";
+    private static final String ADD_DELETE = "add_delete";
+    private static final String QUERY = "query";
+    private static final String QUERY_ADD_DELETE = "query_add_delete";
+
+    private final String compatibleClientVersion;
+    private static Configuration conf;
+    private static HBaseTestingUtility hbaseTestUtil;
+    private static String zkQuorum;
+    private static String url;
+
+    public BackwardCompatibilityIT(String compatibleClientVersion) {
+        this.compatibleClientVersion = compatibleClientVersion;
+    }
+
+    @Parameters(name = "BackwardCompatibilityIT_compatibleClientVersion={0}")
+    public static synchronized Collection<String> data() {
+        return MetaDataProtocol.COMPATIBLE_CLIENT_VERSIONS;
+    }
+
+    @Before
+    public synchronized void doSetup() throws Exception {
+        conf = HBaseConfiguration.create();
+        hbaseTestUtil = new HBaseTestingUtility(conf);
+        setUpConfigForMiniCluster(conf);
+        conf.set(QueryServices.EXTRA_JDBC_ARGUMENTS_ATTRIB, QueryServicesOptions.DEFAULT_EXTRA_JDBC_ARGUMENTS);
+        hbaseTestUtil.startMiniCluster();
+        zkQuorum = "localhost:" + hbaseTestUtil.getZkCluster().getClientPort();
+        url = PhoenixRuntime.JDBC_PROTOCOL + PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR + zkQuorum;
+        DriverManager.registerDriver(PhoenixDriver.INSTANCE);
+    }
+    
+    @After
+    public void cleanUpAfterTest() throws Exception {
+        try {
+            DriverManager.deregisterDriver(PhoenixDriver.INSTANCE);
+        } finally {
+            hbaseTestUtil.shutdownMiniCluster();
+        }
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with new client and read with old client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client 
+     * 4. New Client inserts more data into the tables created by old client 
+     * 5. Old Client reads the data inserted by new client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectUpsertWithNewClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Insert more data with new client and read with old client
+        executeQueriesWithCurrentVersion(ADD_DATA);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(ADD_DATA, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client 
+     * 4. Old Client inserts more data into the tables created by old client 
+     * 5. New Client reads the data inserted by new client
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testSelectUpsertWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with new client and read with old client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Insert more data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, ADD_DATA);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(ADD_DATA, QUERY));
+    }
+
+    /**
+     * Scenario: 
+     * 1. Old Client connects to the updated server 
+     * 2. Old Client creates tables and inserts data 
+     * 3. New Client reads the data inserted by the old client 
+     * 4. Old Client creates and deletes the data
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertDeleteWithOldClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueryWithClientVersion(compatibleClientVersion, CREATE_ADD);
+        executeQueriesWithCurrentVersion(QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Deletes with the old client
+        executeQueryWithClientVersion(compatibleClientVersion, ADD_DELETE);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY_ADD_DELETE);
+        assertTrue(compareOutput(ADD_DELETE, QUERY_ADD_DELETE));
+    }
+
+    /**
+     * Scenario: 
+     * 1. New Client connects to the updated server 
+     * 2. New Client creates tables and inserts data 
+     * 3. Old Client reads the data inserted by the old client 
+     * 4. New Client creates and deletes the data
+     * 
+     * @throws Exception
+     */
+    @Test
+    public void testUpsertDeleteWithNewClient() throws Exception {
+        checkForPreConditions();
+        // Insert data with old client and read with new client
+        executeQueriesWithCurrentVersion(CREATE_ADD);
+        executeQueryWithClientVersion(compatibleClientVersion, QUERY);
+        assertTrue(compareOutput(CREATE_ADD, QUERY));
+
+        // Deletes with the new client
+        executeQueriesWithCurrentVersion(ADD_DELETE);
+        executeQueriesWithCurrentVersion(QUERY_ADD_DELETE);
+        assertTrue(compareOutput(ADD_DELETE, QUERY_ADD_DELETE));
+    }
+    
+    private void checkForPreConditions() throws Exception {
+        // For the first code cut of any major version, there wouldn't be any backward compatible
+        // clients. Hence the test wouldn't run and just return true when the client  
+        // version to be tested is same as current version
+        assumeFalse(compatibleClientVersion.contains(MetaDataProtocol.CURRENT_CLIENT_VERSION));
+        // Make sure that cluster is clean before test execution with no system tables
+        org.apache.hadoop.hbase.client.Connection conn = ConnectionFactory.createConnection(conf);
 
 Review comment:
   Done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services