You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by nico-pappagianis <gi...@git.apache.org> on 2016/12/14 20:41:40 UTC
[GitHub] phoenix pull request #226: Pass tenantId parameter to PhoenixRDD when readin...
GitHub user nico-pappagianis opened a pull request:
https://github.com/apache/phoenix/pull/226
Pass tenantId parameter to PhoenixRDD when reading tables
- Added corresponding tests
- Cleaned up some test code
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nico-pappagianis/phoenix PHOENIX-3532
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/phoenix/pull/226.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #226
----
commit b7e6192a1b927bd20db7c46a34b9b07847e032d3
Author: Nico <ni...@salesforce.com>
Date: 2016-12-14T20:40:13Z
Pass tenantId parameter to PhoenixRDD when reading tables, add corresponding tests, code cleanup
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #226: Pass tenantId parameter to PhoenixRDD when readin...
Posted by nico-pappagianis <gi...@git.apache.org>.
Github user nico-pappagianis commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/226#discussion_r92484983
--- Diff: phoenix-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkITTenantSpecific.scala ---
@@ -19,18 +19,35 @@ import org.apache.spark.sql.SQLContext
import scala.collection.mutable.ListBuffer
--- End diff --
Added two tests under "Read Tests"
General code cleanup
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #226: Pass tenantId parameter to PhoenixRDD when readin...
Posted by nico-pappagianis <gi...@git.apache.org>.
Github user nico-pappagianis commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/226#discussion_r92485510
--- Diff: phoenix-spark/src/main/scala/org/apache/phoenix/spark/SparkSqlContextFunctions.scala ---
@@ -29,11 +29,13 @@ class SparkSqlContextFunctions(@transient val sqlContext: SQLContext) extends Se
property will be used
*/
def phoenixTableAsDataFrame(table: String, columns: Seq[String],
- predicate: Option[String] = None, zkUrl: Option[String] = None,
+ predicate: Option[String] = None,
+ zkUrl: Option[String] = None,
+ tenantId: Option[String] = None,
conf: Configuration = new Configuration): DataFrame = {
// Create the PhoenixRDD and convert it to a DataFrame
- new PhoenixRDD(sqlContext.sparkContext, table, columns, predicate, zkUrl, conf)
+ new PhoenixRDD(sqlContext.sparkContext, table, columns, predicate, zkUrl, tenantId, conf)
--- End diff --
Now we pass tenantId to PhoenixRDD so we can read tenant-specific tables as DataFrames
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #226: Pass tenantId parameter to PhoenixRDD when readin...
Posted by nico-pappagianis <gi...@git.apache.org>.
Github user nico-pappagianis commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/226#discussion_r92485493
--- Diff: phoenix-spark/src/main/scala/org/apache/phoenix/spark/SparkContextFunctions.scala ---
@@ -32,10 +32,10 @@ class SparkContextFunctions(@transient val sc: SparkContext) extends Serializabl
*/
def phoenixTableAsRDD(table: String, columns: Seq[String], predicate: Option[String] = None,
- zkUrl: Option[String] = None, conf: Configuration = new Configuration())
+ zkUrl: Option[String] = None, tenantId: Option[String] = None, conf: Configuration = new Configuration())
: RDD[Map[String, AnyRef]] = {
// Create a PhoenixRDD, but only return the serializable 'result' map
- new PhoenixRDD(sc, table, columns, predicate, zkUrl, conf).map(_.result)
+ new PhoenixRDD(sc, table, columns, predicate, zkUrl, tenantId, conf).map(_.result)
--- End diff --
Now we pass tenantId to PhoenixRDD so we can read tenant-specific tables
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #226: Pass tenantId parameter to PhoenixRDD when readin...
Posted by nico-pappagianis <gi...@git.apache.org>.
Github user nico-pappagianis commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/226#discussion_r92484804
--- Diff: phoenix-spark/src/it/scala/org/apache/phoenix/spark/AbstractPhoenixSparkIT.scala ---
@@ -46,11 +46,6 @@ class AbstractPhoenixSparkIT extends FunSuite with Matchers with BeforeAndAfterA
// A global tenantId we can use across tests
final val TenantId = "theTenant"
- // TENANT_VIEW schema
--- End diff --
Moved these to the PhoenixSparkITTenantSpecific test class
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #226: Pass tenantId parameter to PhoenixRDD when readin...
Posted by nico-pappagianis <gi...@git.apache.org>.
Github user nico-pappagianis commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/226#discussion_r92484705
--- Diff: phoenix-spark/src/it/resources/tenantSetup.sql ---
@@ -15,3 +15,4 @@
-- limitations under the License.
CREATE VIEW IF NOT EXISTS TENANT_VIEW(TENANT_ONLY_COL VARCHAR) AS SELECT * FROM MULTITENANT_TEST_TABLE
+UPSERT INTO TENANT_VIEW (ORGANIZATION_ID, TENANT_ONLY_COL) VALUES ('defaultOrg', 'defaultData')
--- End diff --
Adding some default data so we can know something was actually read in the "read" tests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request #226: Pass tenantId parameter to PhoenixRDD when readin...
Posted by nico-pappagianis <gi...@git.apache.org>.
Github user nico-pappagianis commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/226#discussion_r92484889
--- Diff: phoenix-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkITTenantSpecific.scala ---
@@ -39,62 +56,78 @@ class PhoenixSparkITTenantSpecific extends AbstractPhoenixSparkIT {
results.append((rs.getString(1), rs.getString(2)))
}
stmt.close()
-
- results.toList shouldEqual DataSet
+ results.toList shouldEqual VerificationDataSet
}
- test("Can persist a dataframe using 'DataFrame.saveToPhoenix' on tenant-specific view") {
+ /*****************/
--- End diff --
Two tests for "read" functionality
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---