Is operation subtracting two dataframe valid.

[Mich Talebzadeh <mi...@gmail.com>]

I am using Google Kubernetes Cluster with the docker image that I built
with PySpark 3.1.1 on prem and pushed the docker image to a  google
repository.


The py module generates some 100 rows of Random data and then writes it to
a BigQuery table.


Both write to and subsequent read  from BigQuery table show the correct
number of rows:


 Populated BigQuery table test.randomData

 rows written is  100

 Reading from BigQuery table test.randomData

 rows read in is  100

However, the following operation fails

       if df2.subtract(read_df).count() == 0:

            print("Data has been loaded OK to Oracle table")

        else:

            print("Data could not be loaded to Oracle table, quitting")

            sys.exit(1)


21/08/06 21:58:45 WARN org.apache.spark.scheduler.TaskSetManager: Lost task
0.0 in stage 8.0 (TID 11) (10.64.2.15 executor 1):
java.lang.UnsupportedOperationException: sun.misc.Unsafe or
java.nio.DirectByteBuffer.<init>(long, int) not available
        at
com.google.cloud.spark.bigquery.repackaged.io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:490)
        at
com.google.cloud.spark.bigquery.repackaged.io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:257)
        at
com.google.cloud.spark.bigquery.repackaged.io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:247)

Further down it shows:


py4j.protocol.Py4JJavaError: An error occurred while calling o116.count.

: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute,
tree:

OK this may be specific to BigQuery because as I rtecall this operation
could be done against an Oracle table.

Thanks

   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.