You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by ajaz rahman <aj...@yahoo.com> on 2022/02/09 11:00:38 UTC
Using Jdbc Driver to connect with phoenix
Hi,
I am using jdbc driver to connect with CDP Phoenix from nifi.
I am able to connect with CDP Phoenix using my spark code.
from pyspark.sql import SparkSessionspark2 = SparkSession.builder.appName("sparkanalysis").config("spark.driver.extraClassPath","/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/phoenix_queryserver/phoenix-client-hbase-2.2-5.1.0.7.1.6.0-297.jar").getOrCreate()
pgDF = spark2.read \.format("jdbc") \.option("url", "jdbc:phoenix:10.64.63.122:2181:/hbase") \.option("dbtable", "mydata_table") \.option("user", "") \.option("password", "") \.load()
pdDF.printSchema()
When trying to connect from nifi using same driver. I get error, snapshot is attached along this mail. Its a very simple nifi flow with just one task running an SQL.Connection properties attached as snapshot.
I have been trying crack this for a couple of days without success. Any help around this will receive a lot of love.
Regards,Ajaz-Ur-Rahman
Re: Using Jdbc Driver to connect with phoenix
Posted by ajaz rahman <aj...@yahoo.com>.
Hi,
I was able to resolve the issue, by reverse engineering the script which connect sqlline utility with phonenix. Supplemented with documentation from phoenix, i was able to figure out that we can connect nifi to CDP using a thin client driver ( the file name is phoenix-queryserver-client-xxxxx.jar . The configuration for the driver is attached along this email. It works now. Thanks everyone.
On Wednesday, February 9, 2022, 10:24:08 PM GMT+5, Pierre Villard <pi...@gmail.com> wrote:
Not saying this is a problem but, based on the screenshot, it seems you're using NiFi 1.4 which is a *very* old version. Using the latest is probably a good idea.
Thanks,Pierre
Le mer. 9 févr. 2022 à 18:02, ajaz rahman <aj...@yahoo.com> a écrit :
Hi,
yes we are using phoenix-client-hbase jar. I have copied this one from my CDP edge node to nifi. There were couple of other jars in the same folders where i found phoenix-client. I have not copied them to nifi. I will try to copy those and try to see if it works. Thanks for the clue.
ThanksAjaz-Ur-Rahman
On Wednesday, February 9, 2022, 06:00:02 PM GMT+5, Shawn Weeks <sw...@weeksconsulting.us> wrote:
It’s been a while, but I’ve connected to Phoenix using NiFi. It looks like the jdbc jar your using on the NiFi side is newer than what you have deployed to your Cloudera Environment. Are you using phoenix-client-hbase-2.2-5.1.0.7.1.6.0-297.jar in NiFi? There used to be additional jars that had to be in the same folder too, not sure if that’s the case now.
Thanks
Shawn
From: ajaz rahman <aj...@yahoo.com>
Sent: Wednesday, February 9, 2022 5:01 AM
To: users@nifi.apache.org
Subject: Using Jdbc Driver to connect with phoenix
Hi,
I am using jdbc driver to connect with CDP Phoenix from nifi.
I am able to connect with CDP Phoenix using my spark code.
from pyspark.sql import SparkSession
spark2 = SparkSession.builder.appName("sparkanalysis").config("spark.driver.extraClassPath","/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/phoenix_queryserver/phoenix-client-hbase-2.2-5.1.0.7.1.6.0-297.jar").getOrCreate()
pgDF = spark2.read \
.format("jdbc") \
.option("url", "jdbc:phoenix:10.64.63.122:2181:/hbase") \
.option("dbtable", "mydata_table") \
.option("user", "") \
.option("password", "") \
.load()
pdDF.printSchema()
When trying to connect from nifi using same driver. I get error, snapshot is attached along this mail. Its a very simple nifi flow with just one task running an SQL.
Connection properties attached as snapshot.
I have been trying crack this for a couple of days without success. Any help around this will receive a lot of love.
Regards,
Ajaz-Ur-Rahman
Re: Using Jdbc Driver to connect with phoenix
Posted by Pierre Villard <pi...@gmail.com>.
Not saying this is a problem but, based on the screenshot, it seems you're
using NiFi 1.4 which is a *very* old version. Using the latest is probably
a good idea.
Thanks,
Pierre
Le mer. 9 févr. 2022 à 18:02, ajaz rahman <aj...@yahoo.com> a écrit :
> Hi,
>
> yes we are using phoenix-client-hbase jar. I have copied this one from my
> CDP edge node to nifi. There were couple of other jars in the same folders
> where i found phoenix-client. I have not copied them to nifi. I will try to
> copy those and try to see if it works. Thanks for the clue.
>
> Thanks
> Ajaz-Ur-Rahman
>
> On Wednesday, February 9, 2022, 06:00:02 PM GMT+5, Shawn Weeks <
> sweeks@weeksconsulting.us> wrote:
>
>
> It’s been a while, but I’ve connected to Phoenix using NiFi. It looks like
> the jdbc jar your using on the NiFi side is newer than what you have
> deployed to your Cloudera Environment. Are you using
> phoenix-client-hbase-2.2-5.1.0.7.1.6.0-297.jar in NiFi? There used to be
> additional jars that had to be in the same folder too, not sure if that’s
> the case now.
>
>
>
> Thanks
>
> Shawn
>
>
>
> *From:* ajaz rahman <aj...@yahoo.com>
> *Sent:* Wednesday, February 9, 2022 5:01 AM
> *To:* users@nifi.apache.org
> *Subject:* Using Jdbc Driver to connect with phoenix
>
>
>
> Hi,
>
>
>
> I am using jdbc driver to connect with CDP Phoenix from nifi.
>
>
>
> I am able to connect with CDP Phoenix using my spark code.
>
>
>
> from pyspark.sql import SparkSession
>
> spark2 =
> SparkSession.builder.appName("sparkanalysis").config("spark.driver.extraClassPath","/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/phoenix_queryserver/phoenix-client-hbase-2.2-5.1.0.7.1.6.0-297.jar").getOrCreate()
>
>
>
> pgDF = spark2.read \
>
> .format("jdbc") \
>
> .option("url", "jdbc:phoenix:10.64.63.122:2181:/hbase") \
>
> .option("dbtable", "mydata_table") \
>
> .option("user", "") \
>
> .option("password", "") \
>
> .load()
>
>
>
> pdDF.printSchema()
>
>
>
>
>
> When trying to connect from nifi using same driver. I get error, snapshot
> is attached along this mail. Its a very simple nifi flow with just one
> task running an SQL.
>
> Connection properties attached as snapshot.
>
>
>
> I have been trying crack this for a couple of days without success. Any
> help around this will receive a lot of love.
>
>
>
>
>
> Regards,
>
> Ajaz-Ur-Rahman
>
Re: Using Jdbc Driver to connect with phoenix
Posted by ajaz rahman <aj...@yahoo.com>.
Hi,
yes we are using phoenix-client-hbase jar. I have copied this one from my CDP edge node to nifi. There were couple of other jars in the same folders where i found phoenix-client. I have not copied them to nifi. I will try to copy those and try to see if it works. Thanks for the clue.
ThanksAjaz-Ur-Rahman
On Wednesday, February 9, 2022, 06:00:02 PM GMT+5, Shawn Weeks <sw...@weeksconsulting.us> wrote:
<!--#yiv5876834086 _filtered {} _filtered {} _filtered {}#yiv5876834086 #yiv5876834086 p.yiv5876834086MsoNormal, #yiv5876834086 li.yiv5876834086MsoNormal, #yiv5876834086 div.yiv5876834086MsoNormal {margin:0in;font-size:11.0pt;font-family:"Calibri", sans-serif;}#yiv5876834086 span.yiv5876834086EmailStyle18 {font-family:"Calibri", sans-serif;color:windowtext;}#yiv5876834086 .yiv5876834086MsoChpDefault {font-size:10.0pt;} _filtered {}#yiv5876834086 div.yiv5876834086WordSection1 {}-->
It’s been a while, but I’ve connected to Phoenix using NiFi. It looks like the jdbc jar your using on the NiFi side is newer than what you have deployed to your Cloudera Environment. Are you using phoenix-client-hbase-2.2-5.1.0.7.1.6.0-297.jar in NiFi? There used to be additional jars that had to be in the same folder too, not sure if that’s the case now.
Thanks
Shawn
From: ajaz rahman <aj...@yahoo.com>
Sent: Wednesday, February 9, 2022 5:01 AM
To: users@nifi.apache.org
Subject: Using Jdbc Driver to connect with phoenix
Hi,
I am using jdbc driver to connect with CDP Phoenix from nifi.
I am able to connect with CDP Phoenix using my spark code.
from pyspark.sql import SparkSession
spark2 = SparkSession.builder.appName("sparkanalysis").config("spark.driver.extraClassPath","/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/phoenix_queryserver/phoenix-client-hbase-2.2-5.1.0.7.1.6.0-297.jar").getOrCreate()
pgDF = spark2.read \
.format("jdbc") \
.option("url", "jdbc:phoenix:10.64.63.122:2181:/hbase") \
.option("dbtable", "mydata_table") \
.option("user", "") \
.option("password", "") \
.load()
pdDF.printSchema()
When trying to connect from nifi using same driver. I get error, snapshot is attached along this mail. Its a very simple nifi flow with just one task running an SQL.
Connection properties attached as snapshot.
I have been trying crack this for a couple of days without success. Any help around this will receive a lot of love.
Regards,
Ajaz-Ur-Rahman
RE: Using Jdbc Driver to connect with phoenix
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
It’s been a while, but I’ve connected to Phoenix using NiFi. It looks like the jdbc jar your using on the NiFi side is newer than what you have deployed to your Cloudera Environment. Are you using phoenix-client-hbase-2.2-5.1.0.7.1.6.0-297.jar in NiFi? There used to be additional jars that had to be in the same folder too, not sure if that’s the case now.
Thanks
Shawn
From: ajaz rahman <aj...@yahoo.com>
Sent: Wednesday, February 9, 2022 5:01 AM
To: users@nifi.apache.org
Subject: Using Jdbc Driver to connect with phoenix
Hi,
I am using jdbc driver to connect with CDP Phoenix from nifi.
I am able to connect with CDP Phoenix using my spark code.
from pyspark.sql import SparkSession
spark2 = SparkSession.builder.appName("sparkanalysis").config("spark.driver.extraClassPath","/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/phoenix_queryserver/phoenix-client-hbase-2.2-5.1.0.7.1.6.0-297.jar").getOrCreate()
pgDF = spark2.read \
.format("jdbc") \
.option("url", "jdbc:phoenix:10.64.63.122:2181:/hbase") \
.option("dbtable", "mydata_table") \
.option("user", "") \
.option("password", "") \
.load()
pdDF.printSchema()
When trying to connect from nifi using same driver. I get error, snapshot is attached along this mail. Its a very simple nifi flow with just one task running an SQL.
Connection properties attached as snapshot.
I have been trying crack this for a couple of days without success. Any help around this will receive a lot of love.
Regards,
Ajaz-Ur-Rahman