You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "Abhey (via GitHub)" <gi...@apache.org> on 2023/01/23 13:33:24 UTC

[GitHub] [phoenix-connectors] Abhey opened a new pull request, #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Abhey opened a new pull request, #92:
URL: https://github.com/apache/phoenix-connectors/pull/92

   This PR introduces the changes for adding the PySpark code references in the phoenix5-spark3 connector README.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] stoty commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1400381667

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m  3s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ master Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   | -1 :x: |  markdownlint  |   0m  2s |  The patch generated 33 new + 46 unchanged - 8 fixed = 79 total (was 54)  |
   | -1 :x: |  whitespace  |   0m  0s |  The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 31s |  The patch does not generate ASF License warnings.  |
   |  |   |   2m  2s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/2/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/phoenix-connectors/pull/92 |
   | Optional Tests | dupname asflicense markdownlint |
   | uname | Linux fdceb27a730b 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev/phoenix-connectors-personality.sh |
   | git revision | master / 4da44c7 |
   | markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/2/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
   | whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/2/artifact/yetus-general-check/output/whitespace-eol.txt |
   | Max. process+thread count | 47 (vs. ulimit of 30000) |
   | modules | C: phoenix5-spark3 U: phoenix5-spark3 |
   | Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/2/console |
   | versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] stoty commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1400367248

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   6m 18s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ master Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   | -1 :x: |  markdownlint  |   0m  1s |  The patch generated 30 new + 50 unchanged - 4 fixed = 80 total (was 54)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 31s |  The patch does not generate ASF License warnings.  |
   |  |   |   7m 18s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/1/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/phoenix-connectors/pull/92 |
   | Optional Tests | dupname asflicense markdownlint |
   | uname | Linux 8a914290e3a8 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev/phoenix-connectors-personality.sh |
   | git revision | master / 4da44c7 |
   | markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/1/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
   | Max. process+thread count | 47 (vs. ulimit of 30000) |
   | modules | C: phoenix5-spark3 U: phoenix5-spark3 |
   | Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/1/console |
   | versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] virajjasani commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1400759569

   @stoty you might also be interested in this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] stoty commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1400651464

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m  0s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ master Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  markdownlint  |   0m  2s |  The patch generated 0 new + 15 unchanged - 39 fixed = 15 total (was 54)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 30s |  The patch does not generate ASF License warnings.  |
   |  |   |   1m 59s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/5/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/phoenix-connectors/pull/92 |
   | Optional Tests | dupname asflicense markdownlint |
   | uname | Linux 822a16ddacb3 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev/phoenix-connectors-personality.sh |
   | git revision | master / 4da44c7 |
   | Max. process+thread count | 47 (vs. ulimit of 30000) |
   | modules | C: phoenix5-spark3 U: phoenix5-spark3 |
   | Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/5/console |
   | versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] Abhey commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "Abhey (via GitHub)" <gi...@apache.org>.
Abhey commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1409230823

   Thanks, @stoty for reviewing the PR. I will take a look at your review comments and address the same.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] stoty commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1400404549

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m  4s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ master Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   | -1 :x: |  markdownlint  |   0m  2s |  The patch generated 10 new + 28 unchanged - 26 fixed = 38 total (was 54)  |
   | -1 :x: |  whitespace  |   0m  0s |  The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 30s |  The patch does not generate ASF License warnings.  |
   |  |   |   2m  2s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/3/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/phoenix-connectors/pull/92 |
   | Optional Tests | dupname asflicense markdownlint |
   | uname | Linux fb3451f06b7c 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev/phoenix-connectors-personality.sh |
   | git revision | master / 4da44c7 |
   | markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/3/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
   | whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/3/artifact/yetus-general-check/output/whitespace-eol.txt |
   | Max. process+thread count | 47 (vs. ulimit of 30000) |
   | modules | C: phoenix5-spark3 U: phoenix5-spark3 |
   | Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/3/console |
   | versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] virajjasani commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1400759783

   Thanks for the PR @Abhey 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] stoty commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1400769839

   Would it be possible to add the relevant changes to the Spark2 README ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] Abhey commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "Abhey (via GitHub)" <gi...@apache.org>.
Abhey commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1409232833

   @stoty One question for you, I created a Docker Image with all the Pre-Requisites for testing the Phoenix-Spark connector. Do you think it's a good idea to add the reference to the same in the official documentation?
   
   Repository Link -
   * https://github.com/Abhey/phoenix-spark-connector-demo


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] stoty commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1400626524

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |:----:|----------:|--------:|:--------|
   | +0 :ok: |  reexec  |   1m  6s |  Docker mode activated.  |
   ||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  No case conflicting files found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  The patch does not contain any @author tags.  |
   ||| _ master Compile Tests _ |
   ||| _ Patch Compile Tests _ |
   | -1 :x: |  markdownlint  |   0m  1s |  The patch generated 3 new + 19 unchanged - 35 fixed = 22 total (was 54)  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  The patch has no whitespace issues.  |
   ||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 31s |  The patch does not generate ASF License warnings.  |
   |  |   |   2m  4s |   |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/4/artifact/yetus-general-check/output/Dockerfile |
   | GITHUB PR | https://github.com/apache/phoenix-connectors/pull/92 |
   | Optional Tests | dupname asflicense markdownlint |
   | uname | Linux 9e5b71556a5d 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev/phoenix-connectors-personality.sh |
   | git revision | master / 4da44c7 |
   | markdownlint | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/4/artifact/yetus-general-check/output/diff-patch-markdownlint.txt |
   | Max. process+thread count | 47 (vs. ulimit of 30000) |
   | modules | C: phoenix5-spark3 U: phoenix5-spark3 |
   | Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-92/4/console |
   | versions | git=2.7.4 maven=3.3.9 markdownlint=0.22.0 |
   | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] Abhey commented on pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "Abhey (via GitHub)" <gi...@apache.org>.
Abhey commented on PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#issuecomment-1400653837

   @virajjasani @tkhurana Please review the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [phoenix-connectors] stoty commented on a diff in pull request #92: PHOENIX-6859 Update phoenix5-spark3 README with PySpark code references

Posted by "stoty (via GitHub)" <gi...@apache.org>.
stoty commented on code in PR #92:
URL: https://github.com/apache/phoenix-connectors/pull/92#discussion_r1084392227


##########
phoenix5-spark3/README.md:
##########
@@ -15,18 +15,59 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as DataFrames,
-and enables persisting DataFrames back to Phoenix.
+# Phoenix5-Spark3 Connector
 
-## Configuring Spark to use the connector
+The phoenix5-spark3 plugin extends Phoenix's MapReduce support to allow Spark

Review Comment:
   I realize the we use "Plugin" on the website, but we should standardize on "Connector"



##########
phoenix5-spark3/README.md:
##########
@@ -15,18 +15,59 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as DataFrames,
-and enables persisting DataFrames back to Phoenix.
+# Phoenix5-Spark3 Connector
 
-## Configuring Spark to use the connector
+The phoenix5-spark3 plugin extends Phoenix's MapReduce support to allow Spark
+ to load Phoenix tables as DataFrames,
+ and enables persisting DataFrames back to Phoenix.
 
-Use the shaded connector JAR `phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar` .
-Apart from the shaded connector JAR, you also need to add the hbase mapredcp libraries and the hbase configuration directory to the classpath. The final classpath should be something like
+## Pre-Requisites
 
-`/etc/hbase/conf:$(hbase mapredcp):phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar`
+* Phoenix 5.1.2+
+* Spark 3.0.3+
 
-(add the exact paths as appropiate to your system)
-Both the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties need to be set the above classpath. You may add them spark-defaults.conf, or specify them on the spark-shell or spark-submit command line.
+## Why not JDBC
+
+Although Spark supports connecting directly to JDBC databases,
+ It’s only able to parallelize queries by partioning on a numeric column.
+ It also requires a known lower bound,
+ upper bound and partition count in order to create split queries.
+
+In contrast, the phoenix-spark integration is able to leverage the underlying
+ splits provided by Phoenix in order to retrieve and save data across multiple
+ workers. All that’s required is a database URL and a table name.

Review Comment:
   use "select statement" instead of "table name" ?



##########
phoenix5-spark3/README.md:
##########
@@ -15,18 +15,59 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as DataFrames,
-and enables persisting DataFrames back to Phoenix.
+# Phoenix5-Spark3 Connector
 
-## Configuring Spark to use the connector
+The phoenix5-spark3 plugin extends Phoenix's MapReduce support to allow Spark
+ to load Phoenix tables as DataFrames,
+ and enables persisting DataFrames back to Phoenix.
 
-Use the shaded connector JAR `phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar` .
-Apart from the shaded connector JAR, you also need to add the hbase mapredcp libraries and the hbase configuration directory to the classpath. The final classpath should be something like
+## Pre-Requisites
 
-`/etc/hbase/conf:$(hbase mapredcp):phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar`
+* Phoenix 5.1.2+
+* Spark 3.0.3+
 
-(add the exact paths as appropiate to your system)
-Both the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties need to be set the above classpath. You may add them spark-defaults.conf, or specify them on the spark-shell or spark-submit command line.
+## Why not JDBC
+
+Although Spark supports connecting directly to JDBC databases,
+ It’s only able to parallelize queries by partioning on a numeric column.
+ It also requires a known lower bound,
+ upper bound and partition count in order to create split queries.
+
+In contrast, the phoenix-spark integration is able to leverage the underlying
+ splits provided by Phoenix in order to retrieve and save data across multiple
+ workers. All that’s required is a database URL and a table name.
+ Optional SELECT columns can be given,
+ as well as pushdown predicates for efficient filtering.
+
+The choice of which method to use to access
+ Phoenix comes down to each specific use case.
+
+## Setup
+
+To setup connector add `phoenix5-spark3-shaded` JAR as

Review Comment:
   In most cases, you don't want to add the connector to the maven/compile classpath, it tends to cause conflicts when upgrading.
   
   We should move this to emd of the section, and add the caveat that this is only needed for the deprecated usages.



##########
phoenix5-spark3/README.md:
##########
@@ -39,7 +80,9 @@ UPSERT INTO TABLE1 (ID, COL1) VALUES (2, 'test_row_2');
 ```
 
 ### Load as a DataFrame using the DataSourceV2 API
+
 Scala example:
+

Review Comment:
   I know you didn't touch that part, but do we still need the SparkContext import ?



##########
phoenix5-spark3/README.md:
##########
@@ -39,7 +80,9 @@ UPSERT INTO TABLE1 (ID, COL1) VALUES (2, 'test_row_2');
 ```
 
 ### Load as a DataFrame using the DataSourceV2 API
+
 Scala example:
+

Review Comment:
   Maybe add comments to make  it obvious that you need to use a real ZK quorum , like 
   `//replace "phoenix-server:2181" with the real ZK quorum`



##########
phoenix5-spark3/README.md:
##########
@@ -15,18 +15,59 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as DataFrames,
-and enables persisting DataFrames back to Phoenix.
+# Phoenix5-Spark3 Connector
 
-## Configuring Spark to use the connector
+The phoenix5-spark3 plugin extends Phoenix's MapReduce support to allow Spark
+ to load Phoenix tables as DataFrames,
+ and enables persisting DataFrames back to Phoenix.
 
-Use the shaded connector JAR `phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar` .
-Apart from the shaded connector JAR, you also need to add the hbase mapredcp libraries and the hbase configuration directory to the classpath. The final classpath should be something like
+## Pre-Requisites
 
-`/etc/hbase/conf:$(hbase mapredcp):phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar`
+* Phoenix 5.1.2+
+* Spark 3.0.3+
 
-(add the exact paths as appropiate to your system)
-Both the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties need to be set the above classpath. You may add them spark-defaults.conf, or specify them on the spark-shell or spark-submit command line.
+## Why not JDBC
+
+Although Spark supports connecting directly to JDBC databases,
+ It’s only able to parallelize queries by partioning on a numeric column.
+ It also requires a known lower bound,
+ upper bound and partition count in order to create split queries.
+
+In contrast, the phoenix-spark integration is able to leverage the underlying
+ splits provided by Phoenix in order to retrieve and save data across multiple
+ workers. All that’s required is a database URL and a table name.
+ Optional SELECT columns can be given,
+ as well as pushdown predicates for efficient filtering.

Review Comment:
   This sounds like you need tospecify the pushdown predicates.
   Can you rephrase so that it's apparent that pushdown is automatic ?



##########
phoenix5-spark3/README.md:
##########
@@ -15,18 +15,59 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as DataFrames,
-and enables persisting DataFrames back to Phoenix.
+# Phoenix5-Spark3 Connector
 
-## Configuring Spark to use the connector
+The phoenix5-spark3 plugin extends Phoenix's MapReduce support to allow Spark
+ to load Phoenix tables as DataFrames,
+ and enables persisting DataFrames back to Phoenix.
 
-Use the shaded connector JAR `phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar` .
-Apart from the shaded connector JAR, you also need to add the hbase mapredcp libraries and the hbase configuration directory to the classpath. The final classpath should be something like
+## Pre-Requisites
 
-`/etc/hbase/conf:$(hbase mapredcp):phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar`
+* Phoenix 5.1.2+
+* Spark 3.0.3+
 
-(add the exact paths as appropiate to your system)
-Both the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties need to be set the above classpath. You may add them spark-defaults.conf, or specify them on the spark-shell or spark-submit command line.
+## Why not JDBC
+
+Although Spark supports connecting directly to JDBC databases,
+ It’s only able to parallelize queries by partioning on a numeric column.
+ It also requires a known lower bound,
+ upper bound and partition count in order to create split queries.
+
+In contrast, the phoenix-spark integration is able to leverage the underlying
+ splits provided by Phoenix in order to retrieve and save data across multiple
+ workers. All that’s required is a database URL and a table name.
+ Optional SELECT columns can be given,
+ as well as pushdown predicates for efficient filtering.
+
+The choice of which method to use to access
+ Phoenix comes down to each specific use case.

Review Comment:
   nit:
   This is super important, and we should have much more on this (though not necessarily in this ticket)



##########
phoenix5-spark3/README.md:
##########
@@ -15,18 +15,59 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as DataFrames,
-and enables persisting DataFrames back to Phoenix.
+# Phoenix5-Spark3 Connector
 
-## Configuring Spark to use the connector
+The phoenix5-spark3 plugin extends Phoenix's MapReduce support to allow Spark
+ to load Phoenix tables as DataFrames,
+ and enables persisting DataFrames back to Phoenix.
 
-Use the shaded connector JAR `phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar` .
-Apart from the shaded connector JAR, you also need to add the hbase mapredcp libraries and the hbase configuration directory to the classpath. The final classpath should be something like
+## Pre-Requisites
 
-`/etc/hbase/conf:$(hbase mapredcp):phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar`
+* Phoenix 5.1.2+
+* Spark 3.0.3+
 
-(add the exact paths as appropiate to your system)
-Both the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties need to be set the above classpath. You may add them spark-defaults.conf, or specify them on the spark-shell or spark-submit command line.
+## Why not JDBC
+
+Although Spark supports connecting directly to JDBC databases,
+ It’s only able to parallelize queries by partioning on a numeric column.
+ It also requires a known lower bound,
+ upper bound and partition count in order to create split queries.
+
+In contrast, the phoenix-spark integration is able to leverage the underlying
+ splits provided by Phoenix in order to retrieve and save data across multiple
+ workers. All that’s required is a database URL and a table name.
+ Optional SELECT columns can be given,
+ as well as pushdown predicates for efficient filtering.
+
+The choice of which method to use to access
+ Phoenix comes down to each specific use case.
+
+## Setup

Review Comment:
   Nit: this assumes that Phoenix and HBase/Spark are are both present and configured on the same nodes.
   Maybe worth mentioning it ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org