You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Gershi, Noam " <no...@citi.com> on 2020/02/11 06:03:19 UTC
Apache Beam with Hive
Hi,
I am searching for a detailed example how to use Apache Beam with Hive and/or Hcatalog?
Creating tables, inserting data and fetching it...
Noam Gershi Software Developer
ICG Technology - TLV Lab
T: +972 (3) 7405718
[http://www.citigroup.com/emeaemailresources/gra30973_EmailSignature.jpg]
Re: Apache Beam with Hive
Posted by Jozef Vilcek <jo...@gmail.com>.
Problem seems to be incompatibility of Hive's hcatalog version ... What
HCatalogIO expects and what you have on classpath. Beams HCatalogIO is is
compiled agains Hive 2.1, you are packing 1.2.0.
On Mon, Feb 17, 2020 at 7:47 AM Gershi, Noam <no...@citi.com> wrote:
> 2.19.0 did not work also...
>
>
> -----Original Message-----
> From: [google.com] Tomo Suzuki <su...@google.com>
> Sent: Friday, February 14, 2020 11:31 PM
> To: user@beam.apache.org
> Subject: Re: Apache Beam with Hive
>
> I see the Beam dependency 2.15 is a bit outdated. Would you try 2.19.0?
>
> On Thu, Feb 13, 2020 at 3:52 AM Gershi, Noam <no...@citi.com> wrote:
> >
> > Hi
> > Pom attached.
> >
> > -----Original Message-----
> > From:
> > [https://urldefense.proofpoint.com/v2/url?u=http-3A__google.com&d=DwIF
> > aQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLT
> > OAM&m=KliJIPbYBRpoKwpCFwR3xqXIqyf6i-XOZoBNwYS3uH8&s=-ippa2i9Q6aX5Vdq3E
> > NS37LBNCdexF17v8s3iSMipcE&e= ] Tomo Suzuki <su...@google.com>
> > Sent: Wednesday, February 12, 2020 4:56 PM
> > To: user@beam.apache.org
> > Subject: Re: Apache Beam with Hive
> >
> > HI Noam,
> >
> > It seems incompatible libraries are in your dependencies.
> > Would you share pom.xml (or build.gradle) that produced the
> NoSuchMethodError?
> >
> > On Wed, Feb 12, 2020 at 12:47 AM Gershi, Noam <no...@citi.com>
> wrote:
> > >
> > > I used HCatalogIO:
> > >
> > >
> > >
> > > Map<String, String> configProperties = new HashMap<>();
> > > configProperties.put("hive.metastore.uris", "....");
> > >
> > > pipeline.apply(HCatalogIO.read()
> > > .withConfigProperties(configProperties)
> > > .withDatabase(DB_NAME)
> > > .withTable(TEST_TABLE))
> > > .apply("to-string", MapElements.via(new
> SimpleFunction<HCatRecord, String>() {
> > > @Override
> > > public String apply(HCatRecord input) {
> > > return input.toString();
> > > }
> > > }))
> > >
> > > .apply(TextIO.write().to("my-logs/output.txt").withoutSharding());
> > >
> > >
> > >
> > > But I am getting this error:
> > >
> > >
> > >
> > > 20/02/11 07:49:24 INFO spark.SparkContext: Starting job: collect at
> > >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__BoundedDataset.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=xMTsprUk9mFvSvOcJWFCDcrzfvSEetFK1HgqSYnGbMQ&e=
> :93 Exception in thread "dag-scheduler-event-loop"
> java.lang.NoSuchMethodError:
> org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(Lorg/apache/hadoop/hive/conf/HiveConf;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient;
> > > at org.apache.beam.sdk.io
> .hcatalog.HCatalogUtils.createMetaStoreClient(
> https://urldefense.proofpoint.com/v2/url?u=http-3A__HCatalogUtils.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=uGrTnwLNA2_Ujs7ChnrVtTm-npQ2Myuu-NNK0yu66MA&e=
> :42)
> > > at org.apache.beam.sdk.io
> .hcatalog.HCatalogIO$BoundedHCatalogSource.getEstimatedSizeBytes(
> https://urldefense.proofpoint.com/v2/url?u=http-3A__HCatalogIO.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=9JEggr2DRm4S-qJMbmhlBZNciSLsMMbiSSvh8StIOHE&e=
> :323)
> > > at
> org.apache.beam.runners.spark.io.SourceRDD$Bounded.getPartitions(
> https://urldefense.proofpoint.com/v2/url?u=http-3A__SourceRDD.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=4tyzNpdlBvQTL8Zz27JmOkcXgAjDmLj2pWhpKuMFLZc&e=
> :104)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > > at
> org.apache.spark.ShuffleDependency.<init>(Dependency.scala:94)
> > > at
> org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:87)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:240)
> > > at
> org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:238)
> > > at scala.Option.getOrElse(Option.scala:121)
> > > at org.apache.spark.rdd.RDD.dependencies(RDD.scala:238)
> > > at
> org.apache.spark.scheduler.DAGScheduler.getShuffleDependencies(DAGScheduler.scala:512)
> > > at
> org.apache.spark.scheduler.DAGScheduler.getOrCreateParentStages(DAGScheduler.scala:461)
> > > at
> org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:448)
> > > at
> org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:962)
> > > at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2067)
> > > at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
> > > at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
> > > at
> > > org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
> > > 20/02/11 07:52:54 INFO util.JsonImpl: Shutting down HTTP client
> > >
> > >
> > >
> > >
> > >
> > > From: Gershi, Noam [ICG-IT]
> > > Sent: Tuesday, February 11, 2020 11:08 AM
> > > To: user@beam.apache.org
> > > Subject: RE: Apache Beam with Hive
> > >
> > >
> > >
> > > Thanx.
> > >
> > > Looks like Hcatalog could work.
> > >
> > > But - is the an example with ‘SELECT’ query?
> > >
> > >
> > >
> > > JdbcIO probably not good to me, since my spark cluster in already
> configured to work with Hive. So – When I am code Spark pipelines, I can
> write queries without the need to give the user/password. – I would like to
> have something similar in Apache Beam.
> > >
> > >
> > >
> > > From:
> > > [https://urldefense.proofpoint.com/v2/url?u=http-3A__gmail.com&d=DwI
> > > Fa
> > > Q&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vL
> > > TO
> > > AM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=IjORTrY1VF_Wi1G1p
> > > PT B4tTaYHMwaHFG1KATqPmvkfw&e= ] utkarsh.khare@gmail.com
> > > <ut...@gmail.com>
> > > Sent: Tuesday, February 11, 2020 10:26 AM
> > > To: user@beam.apache.org
> > > Subject: Re: Apache Beam with Hive
> > >
> > >
> > >
> > >
> > >
> > > While I've not created Hive tables using this .. reading and writing
> worked very well using HCatalogIO.
> > >
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__beam.apache.org
> > > _r
> > > eleases_javadoc_2.18.0_org_apache_beam_sdk_io_hcatalog_HCatalogIO.ht
> > > ml
> > > &d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXy
> > > qP
> > > YqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=c9eNd6p0i
> > > hI
> > > DovylFf8P8OjjZ2j6hcZgDnIO_0CR_8c&e=
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Feb 11, 2020 at 11:38 AM Jean-Baptiste Onofre <jb...@nanthrax.net>
> wrote:
> > >
> > > Hi,
> > >
> > >
> > >
> > > If you are ok to use Hive via JDBC, you can use JdbcIO and see the
> example in the javadoc:
> > >
> > >
> > >
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apac
> > > he
> > > _beam_blob_master_sdks_java_io_jdbc_src_main_java_org_apache_beam_sd
> > > k_
> > > io_jdbc_JdbcIO.java-23L82&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkY
> > > Yk
> > > cVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7
> > > A1 LD8RDrntO4o&s=D563ULZXnhz13d4Ro61dMXYNpPVHUP4VupjBeoMwZz4&e=
> > >
> > >
> > >
> > > Regards
> > >
> > > JB
> > >
> > >
> > >
> > > Le 11 févr. 2020 à 07:03, Gershi, Noam <no...@citi.com> a écrit
> :
> > >
> > >
> > >
> > > Hi,
> > >
> > >
> > >
> > > I am searching for a detailed example how to use Apache Beam with Hive
> and/or Hcatalog?
> > >
> > > Creating tables, inserting data and fetching it…
> > >
> > >
> > >
> > >
> > >
> > > Noam Gershi Software Developer
> > >
> > > ICG Technology – TLV Lab
> > >
> > > T: +972 (3) 7405718
> > >
> > >
> > >
> > > <image002.png>
> > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Tomo
>
>
>
> --
> Regards,
> Tomo
>
RE: Apache Beam with Hive
Posted by "Gershi, Noam " <no...@citi.com>.
2.19.0 did not work also...
-----Original Message-----
From: [google.com] Tomo Suzuki <su...@google.com>
Sent: Friday, February 14, 2020 11:31 PM
To: user@beam.apache.org
Subject: Re: Apache Beam with Hive
I see the Beam dependency 2.15 is a bit outdated. Would you try 2.19.0?
On Thu, Feb 13, 2020 at 3:52 AM Gershi, Noam <no...@citi.com> wrote:
>
> Hi
> Pom attached.
>
> -----Original Message-----
> From:
> [https://urldefense.proofpoint.com/v2/url?u=http-3A__google.com&d=DwIF
> aQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLT
> OAM&m=KliJIPbYBRpoKwpCFwR3xqXIqyf6i-XOZoBNwYS3uH8&s=-ippa2i9Q6aX5Vdq3E
> NS37LBNCdexF17v8s3iSMipcE&e= ] Tomo Suzuki <su...@google.com>
> Sent: Wednesday, February 12, 2020 4:56 PM
> To: user@beam.apache.org
> Subject: Re: Apache Beam with Hive
>
> HI Noam,
>
> It seems incompatible libraries are in your dependencies.
> Would you share pom.xml (or build.gradle) that produced the NoSuchMethodError?
>
> On Wed, Feb 12, 2020 at 12:47 AM Gershi, Noam <no...@citi.com> wrote:
> >
> > I used HCatalogIO:
> >
> >
> >
> > Map<String, String> configProperties = new HashMap<>();
> > configProperties.put("hive.metastore.uris", "....");
> >
> > pipeline.apply(HCatalogIO.read()
> > .withConfigProperties(configProperties)
> > .withDatabase(DB_NAME)
> > .withTable(TEST_TABLE))
> > .apply("to-string", MapElements.via(new SimpleFunction<HCatRecord, String>() {
> > @Override
> > public String apply(HCatRecord input) {
> > return input.toString();
> > }
> > }))
> >
> > .apply(TextIO.write().to("my-logs/output.txt").withoutSharding());
> >
> >
> >
> > But I am getting this error:
> >
> >
> >
> > 20/02/11 07:49:24 INFO spark.SparkContext: Starting job: collect at
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__BoundedDataset.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=xMTsprUk9mFvSvOcJWFCDcrzfvSEetFK1HgqSYnGbMQ&e= :93 Exception in thread "dag-scheduler-event-loop" java.lang.NoSuchMethodError: org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(Lorg/apache/hadoop/hive/conf/HiveConf;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient;
> > at org.apache.beam.sdk.io.hcatalog.HCatalogUtils.createMetaStoreClient(https://urldefense.proofpoint.com/v2/url?u=http-3A__HCatalogUtils.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=uGrTnwLNA2_Ujs7ChnrVtTm-npQ2Myuu-NNK0yu66MA&e= :42)
> > at org.apache.beam.sdk.io.hcatalog.HCatalogIO$BoundedHCatalogSource.getEstimatedSizeBytes(https://urldefense.proofpoint.com/v2/url?u=http-3A__HCatalogIO.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=9JEggr2DRm4S-qJMbmhlBZNciSLsMMbiSSvh8StIOHE&e= :323)
> > at org.apache.beam.runners.spark.io.SourceRDD$Bounded.getPartitions(https://urldefense.proofpoint.com/v2/url?u=http-3A__SourceRDD.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=4tyzNpdlBvQTL8Zz27JmOkcXgAjDmLj2pWhpKuMFLZc&e= :104)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:94)
> > at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:87)
> > at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:240)
> > at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:238)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.dependencies(RDD.scala:238)
> > at org.apache.spark.scheduler.DAGScheduler.getShuffleDependencies(DAGScheduler.scala:512)
> > at org.apache.spark.scheduler.DAGScheduler.getOrCreateParentStages(DAGScheduler.scala:461)
> > at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:448)
> > at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:962)
> > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2067)
> > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
> > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
> > at
> > org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
> > 20/02/11 07:52:54 INFO util.JsonImpl: Shutting down HTTP client
> >
> >
> >
> >
> >
> > From: Gershi, Noam [ICG-IT]
> > Sent: Tuesday, February 11, 2020 11:08 AM
> > To: user@beam.apache.org
> > Subject: RE: Apache Beam with Hive
> >
> >
> >
> > Thanx.
> >
> > Looks like Hcatalog could work.
> >
> > But - is the an example with ‘SELECT’ query?
> >
> >
> >
> > JdbcIO probably not good to me, since my spark cluster in already configured to work with Hive. So – When I am code Spark pipelines, I can write queries without the need to give the user/password. – I would like to have something similar in Apache Beam.
> >
> >
> >
> > From:
> > [https://urldefense.proofpoint.com/v2/url?u=http-3A__gmail.com&d=DwI
> > Fa
> > Q&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vL
> > TO
> > AM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=IjORTrY1VF_Wi1G1p
> > PT B4tTaYHMwaHFG1KATqPmvkfw&e= ] utkarsh.khare@gmail.com
> > <ut...@gmail.com>
> > Sent: Tuesday, February 11, 2020 10:26 AM
> > To: user@beam.apache.org
> > Subject: Re: Apache Beam with Hive
> >
> >
> >
> >
> >
> > While I've not created Hive tables using this .. reading and writing worked very well using HCatalogIO.
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__beam.apache.org
> > _r
> > eleases_javadoc_2.18.0_org_apache_beam_sdk_io_hcatalog_HCatalogIO.ht
> > ml
> > &d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXy
> > qP
> > YqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=c9eNd6p0i
> > hI
> > DovylFf8P8OjjZ2j6hcZgDnIO_0CR_8c&e=
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Feb 11, 2020 at 11:38 AM Jean-Baptiste Onofre <jb...@nanthrax.net> wrote:
> >
> > Hi,
> >
> >
> >
> > If you are ok to use Hive via JDBC, you can use JdbcIO and see the example in the javadoc:
> >
> >
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apac
> > he
> > _beam_blob_master_sdks_java_io_jdbc_src_main_java_org_apache_beam_sd
> > k_
> > io_jdbc_JdbcIO.java-23L82&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkY
> > Yk
> > cVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7
> > A1 LD8RDrntO4o&s=D563ULZXnhz13d4Ro61dMXYNpPVHUP4VupjBeoMwZz4&e=
> >
> >
> >
> > Regards
> >
> > JB
> >
> >
> >
> > Le 11 févr. 2020 à 07:03, Gershi, Noam <no...@citi.com> a écrit :
> >
> >
> >
> > Hi,
> >
> >
> >
> > I am searching for a detailed example how to use Apache Beam with Hive and/or Hcatalog?
> >
> > Creating tables, inserting data and fetching it…
> >
> >
> >
> >
> >
> > Noam Gershi Software Developer
> >
> > ICG Technology – TLV Lab
> >
> > T: +972 (3) 7405718
> >
> >
> >
> > <image002.png>
> >
> >
>
>
>
> --
> Regards,
> Tomo
--
Regards,
Tomo
Re: Apache Beam with Hive
Posted by Tomo Suzuki <su...@google.com>.
I see the Beam dependency 2.15 is a bit outdated. Would you try 2.19.0?
On Thu, Feb 13, 2020 at 3:52 AM Gershi, Noam <no...@citi.com> wrote:
>
> Hi
> Pom attached.
>
> -----Original Message-----
> From: [google.com] Tomo Suzuki <su...@google.com>
> Sent: Wednesday, February 12, 2020 4:56 PM
> To: user@beam.apache.org
> Subject: Re: Apache Beam with Hive
>
> HI Noam,
>
> It seems incompatible libraries are in your dependencies.
> Would you share pom.xml (or build.gradle) that produced the NoSuchMethodError?
>
> On Wed, Feb 12, 2020 at 12:47 AM Gershi, Noam <no...@citi.com> wrote:
> >
> > I used HCatalogIO:
> >
> >
> >
> > Map<String, String> configProperties = new HashMap<>();
> > configProperties.put("hive.metastore.uris", "....");
> >
> > pipeline.apply(HCatalogIO.read()
> > .withConfigProperties(configProperties)
> > .withDatabase(DB_NAME)
> > .withTable(TEST_TABLE))
> > .apply("to-string", MapElements.via(new SimpleFunction<HCatRecord, String>() {
> > @Override
> > public String apply(HCatRecord input) {
> > return input.toString();
> > }
> > }))
> >
> > .apply(TextIO.write().to("my-logs/output.txt").withoutSharding());
> >
> >
> >
> > But I am getting this error:
> >
> >
> >
> > 20/02/11 07:49:24 INFO spark.SparkContext: Starting job: collect at
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__BoundedDataset.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=xMTsprUk9mFvSvOcJWFCDcrzfvSEetFK1HgqSYnGbMQ&e= :93 Exception in thread "dag-scheduler-event-loop" java.lang.NoSuchMethodError: org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(Lorg/apache/hadoop/hive/conf/HiveConf;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient;
> > at org.apache.beam.sdk.io.hcatalog.HCatalogUtils.createMetaStoreClient(https://urldefense.proofpoint.com/v2/url?u=http-3A__HCatalogUtils.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=uGrTnwLNA2_Ujs7ChnrVtTm-npQ2Myuu-NNK0yu66MA&e= :42)
> > at org.apache.beam.sdk.io.hcatalog.HCatalogIO$BoundedHCatalogSource.getEstimatedSizeBytes(https://urldefense.proofpoint.com/v2/url?u=http-3A__HCatalogIO.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=9JEggr2DRm4S-qJMbmhlBZNciSLsMMbiSSvh8StIOHE&e= :323)
> > at org.apache.beam.runners.spark.io.SourceRDD$Bounded.getPartitions(https://urldefense.proofpoint.com/v2/url?u=http-3A__SourceRDD.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=4tyzNpdlBvQTL8Zz27JmOkcXgAjDmLj2pWhpKuMFLZc&e= :104)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> > at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:94)
> > at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:87)
> > at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:240)
> > at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:238)
> > at scala.Option.getOrElse(Option.scala:121)
> > at org.apache.spark.rdd.RDD.dependencies(RDD.scala:238)
> > at org.apache.spark.scheduler.DAGScheduler.getShuffleDependencies(DAGScheduler.scala:512)
> > at org.apache.spark.scheduler.DAGScheduler.getOrCreateParentStages(DAGScheduler.scala:461)
> > at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:448)
> > at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:962)
> > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2067)
> > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
> > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
> > at
> > org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
> > 20/02/11 07:52:54 INFO util.JsonImpl: Shutting down HTTP client
> >
> >
> >
> >
> >
> > From: Gershi, Noam [ICG-IT]
> > Sent: Tuesday, February 11, 2020 11:08 AM
> > To: user@beam.apache.org
> > Subject: RE: Apache Beam with Hive
> >
> >
> >
> > Thanx.
> >
> > Looks like Hcatalog could work.
> >
> > But - is the an example with ‘SELECT’ query?
> >
> >
> >
> > JdbcIO probably not good to me, since my spark cluster in already configured to work with Hive. So – When I am code Spark pipelines, I can write queries without the need to give the user/password. – I would like to have something similar in Apache Beam.
> >
> >
> >
> > From:
> > [https://urldefense.proofpoint.com/v2/url?u=http-3A__gmail.com&d=DwIFa
> > Q&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTO
> > AM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=IjORTrY1VF_Wi1G1pPT
> > B4tTaYHMwaHFG1KATqPmvkfw&e= ] utkarsh.khare@gmail.com
> > <ut...@gmail.com>
> > Sent: Tuesday, February 11, 2020 10:26 AM
> > To: user@beam.apache.org
> > Subject: Re: Apache Beam with Hive
> >
> >
> >
> >
> >
> > While I've not created Hive tables using this .. reading and writing worked very well using HCatalogIO.
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__beam.apache.org_r
> > eleases_javadoc_2.18.0_org_apache_beam_sdk_io_hcatalog_HCatalogIO.html
> > &d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqP
> > YqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=c9eNd6p0ihI
> > DovylFf8P8OjjZ2j6hcZgDnIO_0CR_8c&e=
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Feb 11, 2020 at 11:38 AM Jean-Baptiste Onofre <jb...@nanthrax.net> wrote:
> >
> > Hi,
> >
> >
> >
> > If you are ok to use Hive via JDBC, you can use JdbcIO and see the example in the javadoc:
> >
> >
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache
> > _beam_blob_master_sdks_java_io_jdbc_src_main_java_org_apache_beam_sdk_
> > io_jdbc_JdbcIO.java-23L82&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYk
> > cVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1
> > LD8RDrntO4o&s=D563ULZXnhz13d4Ro61dMXYNpPVHUP4VupjBeoMwZz4&e=
> >
> >
> >
> > Regards
> >
> > JB
> >
> >
> >
> > Le 11 févr. 2020 à 07:03, Gershi, Noam <no...@citi.com> a écrit :
> >
> >
> >
> > Hi,
> >
> >
> >
> > I am searching for a detailed example how to use Apache Beam with Hive and/or Hcatalog?
> >
> > Creating tables, inserting data and fetching it…
> >
> >
> >
> >
> >
> > Noam Gershi Software Developer
> >
> > ICG Technology – TLV Lab
> >
> > T: +972 (3) 7405718
> >
> >
> >
> > <image002.png>
> >
> >
>
>
>
> --
> Regards,
> Tomo
--
Regards,
Tomo
RE: Apache Beam with Hive
Posted by "Gershi, Noam " <no...@citi.com>.
Hi
Pom attached.
-----Original Message-----
From: [google.com] Tomo Suzuki <su...@google.com>
Sent: Wednesday, February 12, 2020 4:56 PM
To: user@beam.apache.org
Subject: Re: Apache Beam with Hive
HI Noam,
It seems incompatible libraries are in your dependencies.
Would you share pom.xml (or build.gradle) that produced the NoSuchMethodError?
On Wed, Feb 12, 2020 at 12:47 AM Gershi, Noam <no...@citi.com> wrote:
>
> I used HCatalogIO:
>
>
>
> Map<String, String> configProperties = new HashMap<>();
> configProperties.put("hive.metastore.uris", "....");
>
> pipeline.apply(HCatalogIO.read()
> .withConfigProperties(configProperties)
> .withDatabase(DB_NAME)
> .withTable(TEST_TABLE))
> .apply("to-string", MapElements.via(new SimpleFunction<HCatRecord, String>() {
> @Override
> public String apply(HCatRecord input) {
> return input.toString();
> }
> }))
>
> .apply(TextIO.write().to("my-logs/output.txt").withoutSharding());
>
>
>
> But I am getting this error:
>
>
>
> 20/02/11 07:49:24 INFO spark.SparkContext: Starting job: collect at
> https://urldefense.proofpoint.com/v2/url?u=http-3A__BoundedDataset.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=xMTsprUk9mFvSvOcJWFCDcrzfvSEetFK1HgqSYnGbMQ&e= :93 Exception in thread "dag-scheduler-event-loop" java.lang.NoSuchMethodError: org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(Lorg/apache/hadoop/hive/conf/HiveConf;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient;
> at org.apache.beam.sdk.io.hcatalog.HCatalogUtils.createMetaStoreClient(https://urldefense.proofpoint.com/v2/url?u=http-3A__HCatalogUtils.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=uGrTnwLNA2_Ujs7ChnrVtTm-npQ2Myuu-NNK0yu66MA&e= :42)
> at org.apache.beam.sdk.io.hcatalog.HCatalogIO$BoundedHCatalogSource.getEstimatedSizeBytes(https://urldefense.proofpoint.com/v2/url?u=http-3A__HCatalogIO.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=9JEggr2DRm4S-qJMbmhlBZNciSLsMMbiSSvh8StIOHE&e= :323)
> at org.apache.beam.runners.spark.io.SourceRDD$Bounded.getPartitions(https://urldefense.proofpoint.com/v2/url?u=http-3A__SourceRDD.java&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=4tyzNpdlBvQTL8Zz27JmOkcXgAjDmLj2pWhpKuMFLZc&e= :104)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:94)
> at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:87)
> at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:240)
> at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:238)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.dependencies(RDD.scala:238)
> at org.apache.spark.scheduler.DAGScheduler.getShuffleDependencies(DAGScheduler.scala:512)
> at org.apache.spark.scheduler.DAGScheduler.getOrCreateParentStages(DAGScheduler.scala:461)
> at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:448)
> at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:962)
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2067)
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
> at
> org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
> 20/02/11 07:52:54 INFO util.JsonImpl: Shutting down HTTP client
>
>
>
>
>
> From: Gershi, Noam [ICG-IT]
> Sent: Tuesday, February 11, 2020 11:08 AM
> To: user@beam.apache.org
> Subject: RE: Apache Beam with Hive
>
>
>
> Thanx.
>
> Looks like Hcatalog could work.
>
> But - is the an example with ‘SELECT’ query?
>
>
>
> JdbcIO probably not good to me, since my spark cluster in already configured to work with Hive. So – When I am code Spark pipelines, I can write queries without the need to give the user/password. – I would like to have something similar in Apache Beam.
>
>
>
> From:
> [https://urldefense.proofpoint.com/v2/url?u=http-3A__gmail.com&d=DwIFa
> Q&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTO
> AM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=IjORTrY1VF_Wi1G1pPT
> B4tTaYHMwaHFG1KATqPmvkfw&e= ] utkarsh.khare@gmail.com
> <ut...@gmail.com>
> Sent: Tuesday, February 11, 2020 10:26 AM
> To: user@beam.apache.org
> Subject: Re: Apache Beam with Hive
>
>
>
>
>
> While I've not created Hive tables using this .. reading and writing worked very well using HCatalogIO.
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__beam.apache.org_r
> eleases_javadoc_2.18.0_org_apache_beam_sdk_io_hcatalog_HCatalogIO.html
> &d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqP
> YqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1LD8RDrntO4o&s=c9eNd6p0ihI
> DovylFf8P8OjjZ2j6hcZgDnIO_0CR_8c&e=
>
>
>
>
>
>
>
> On Tue, Feb 11, 2020 at 11:38 AM Jean-Baptiste Onofre <jb...@nanthrax.net> wrote:
>
> Hi,
>
>
>
> If you are ok to use Hive via JDBC, you can use JdbcIO and see the example in the javadoc:
>
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache
> _beam_blob_master_sdks_java_io_jdbc_src_main_java_org_apache_beam_sdk_
> io_jdbc_JdbcIO.java-23L82&d=DwIFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYk
> cVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=AgJosEDem12NXaKJZhxwD2jn-v5Tq7A1
> LD8RDrntO4o&s=D563ULZXnhz13d4Ro61dMXYNpPVHUP4VupjBeoMwZz4&e=
>
>
>
> Regards
>
> JB
>
>
>
> Le 11 févr. 2020 à 07:03, Gershi, Noam <no...@citi.com> a écrit :
>
>
>
> Hi,
>
>
>
> I am searching for a detailed example how to use Apache Beam with Hive and/or Hcatalog?
>
> Creating tables, inserting data and fetching it…
>
>
>
>
>
> Noam Gershi Software Developer
>
> ICG Technology – TLV Lab
>
> T: +972 (3) 7405718
>
>
>
> <image002.png>
>
>
--
Regards,
Tomo
Re: Apache Beam with Hive
Posted by Tomo Suzuki <su...@google.com>.
HI Noam,
It seems incompatible libraries are in your dependencies.
Would you share pom.xml (or build.gradle) that produced the NoSuchMethodError?
On Wed, Feb 12, 2020 at 12:47 AM Gershi, Noam <no...@citi.com> wrote:
>
> I used HCatalogIO:
>
>
>
> Map<String, String> configProperties = new HashMap<>();
> configProperties.put("hive.metastore.uris", "....");
>
> pipeline.apply(HCatalogIO.read()
> .withConfigProperties(configProperties)
> .withDatabase(DB_NAME)
> .withTable(TEST_TABLE))
> .apply("to-string", MapElements.via(new SimpleFunction<HCatRecord, String>() {
> @Override
> public String apply(HCatRecord input) {
> return input.toString();
> }
> }))
> .apply(TextIO.write().to("my-logs/output.txt").withoutSharding());
>
>
>
> But I am getting this error:
>
>
>
> 20/02/11 07:49:24 INFO spark.SparkContext: Starting job: collect at BoundedDataset.java:93
> Exception in thread "dag-scheduler-event-loop" java.lang.NoSuchMethodError: org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(Lorg/apache/hadoop/hive/conf/HiveConf;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient;
> at org.apache.beam.sdk.io.hcatalog.HCatalogUtils.createMetaStoreClient(HCatalogUtils.java:42)
> at org.apache.beam.sdk.io.hcatalog.HCatalogIO$BoundedHCatalogSource.getEstimatedSizeBytes(HCatalogIO.java:323)
> at org.apache.beam.runners.spark.io.SourceRDD$Bounded.getPartitions(SourceRDD.java:104)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
> at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:94)
> at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:87)
> at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:240)
> at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:238)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.dependencies(RDD.scala:238)
> at org.apache.spark.scheduler.DAGScheduler.getShuffleDependencies(DAGScheduler.scala:512)
> at org.apache.spark.scheduler.DAGScheduler.getOrCreateParentStages(DAGScheduler.scala:461)
> at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:448)
> at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:962)
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2067)
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
> 20/02/11 07:52:54 INFO util.JsonImpl: Shutting down HTTP client
>
>
>
>
>
> From: Gershi, Noam [ICG-IT]
> Sent: Tuesday, February 11, 2020 11:08 AM
> To: user@beam.apache.org
> Subject: RE: Apache Beam with Hive
>
>
>
> Thanx.
>
> Looks like Hcatalog could work.
>
> But - is the an example with ‘SELECT’ query?
>
>
>
> JdbcIO probably not good to me, since my spark cluster in already configured to work with Hive. So – When I am code Spark pipelines, I can write queries without the need to give the user/password. – I would like to have something similar in Apache Beam.
>
>
>
> From: [gmail.com] utkarsh.khare@gmail.com <ut...@gmail.com>
> Sent: Tuesday, February 11, 2020 10:26 AM
> To: user@beam.apache.org
> Subject: Re: Apache Beam with Hive
>
>
>
>
>
> While I've not created Hive tables using this .. reading and writing worked very well using HCatalogIO.
>
> https://beam.apache.org/releases/javadoc/2.18.0/org/apache/beam/sdk/io/hcatalog/HCatalogIO.html
>
>
>
>
>
>
>
> On Tue, Feb 11, 2020 at 11:38 AM Jean-Baptiste Onofre <jb...@nanthrax.net> wrote:
>
> Hi,
>
>
>
> If you are ok to use Hive via JDBC, you can use JdbcIO and see the example in the javadoc:
>
>
>
> https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L82
>
>
>
> Regards
>
> JB
>
>
>
> Le 11 févr. 2020 à 07:03, Gershi, Noam <no...@citi.com> a écrit :
>
>
>
> Hi,
>
>
>
> I am searching for a detailed example how to use Apache Beam with Hive and/or Hcatalog?
>
> Creating tables, inserting data and fetching it…
>
>
>
>
>
> Noam Gershi Software Developer
>
> ICG Technology – TLV Lab
>
> T: +972 (3) 7405718
>
>
>
> <image002.png>
>
>
--
Regards,
Tomo
RE: Apache Beam with Hive
Posted by "Gershi, Noam " <no...@citi.com>.
I used HCatalogIO:
Map<String, String> configProperties = new HashMap<>();
configProperties.put("hive.metastore.uris", "....");
pipeline.apply(HCatalogIO.read()
.withConfigProperties(configProperties)
.withDatabase(DB_NAME)
.withTable(TEST_TABLE))
.apply("to-string", MapElements.via(new SimpleFunction<HCatRecord, String>() {
@Override
public String apply(HCatRecord input) {
return input.toString();
}
}))
.apply(TextIO.write().to("my-logs/output.txt").withoutSharding());
But I am getting this error:
20/02/11 07:49:24 INFO spark.SparkContext: Starting job: collect at BoundedDataset.java:93
Exception in thread "dag-scheduler-event-loop" java.lang.NoSuchMethodError: org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(Lorg/apache/hadoop/hive/conf/HiveConf;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient;
at org.apache.beam.sdk.io.hcatalog.HCatalogUtils.createMetaStoreClient(HCatalogUtils.java:42)
at org.apache.beam.sdk.io.hcatalog.HCatalogIO$BoundedHCatalogSource.getEstimatedSizeBytes(HCatalogIO.java:323)
at org.apache.beam.runners.spark.io.SourceRDD$Bounded.getPartitions(SourceRDD.java:104)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:94)
at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:87)
at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:240)
at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:238)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.dependencies(RDD.scala:238)
at org.apache.spark.scheduler.DAGScheduler.getShuffleDependencies(DAGScheduler.scala:512)
at org.apache.spark.scheduler.DAGScheduler.getOrCreateParentStages(DAGScheduler.scala:461)
at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:448)
at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:962)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2067)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
20/02/11 07:52:54 INFO util.JsonImpl: Shutting down HTTP client
From: Gershi, Noam [ICG-IT]
Sent: Tuesday, February 11, 2020 11:08 AM
To: user@beam.apache.org
Subject: RE: Apache Beam with Hive
Thanx.
Looks like Hcatalog could work.
But - is the an example with ‘SELECT’ query?
JdbcIO probably not good to me, since my spark cluster in already configured to work with Hive. So – When I am code Spark pipelines, I can write queries without the need to give the user/password. – I would like to have something similar in Apache Beam.
From: [gmail.com] utkarsh.khare@gmail.com<ma...@gmail.com> <ut...@gmail.com>>
Sent: Tuesday, February 11, 2020 10:26 AM
To: user@beam.apache.org<ma...@beam.apache.org>
Subject: Re: Apache Beam with Hive
While I've not created Hive tables using this .. reading and writing worked very well using HCatalogIO.
https://beam.apache.org/releases/javadoc/2.18.0/org/apache/beam/sdk/io/hcatalog/HCatalogIO.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__beam.apache.org_releases_javadoc_2.18.0_org_apache_beam_sdk_io_hcatalog_HCatalogIO.html&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=H-EFU9Q7yATgV6oX55NU_3j5EMt2X2k-9B3sJSMou9M&s=NsHbYvkYO2ZYHO43DxCjEdgW_eCBBZJZCMsdq_lZK4o&e=>
On Tue, Feb 11, 2020 at 11:38 AM Jean-Baptiste Onofre <jb...@nanthrax.net>> wrote:
Hi,
If you are ok to use Hive via JDBC, you can use JdbcIO and see the example in the javadoc:
https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L82<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_beam_blob_master_sdks_java_io_jdbc_src_main_java_org_apache_beam_sdk_io_jdbc_JdbcIO.java-23L82&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=H-EFU9Q7yATgV6oX55NU_3j5EMt2X2k-9B3sJSMou9M&s=OMH2F23vKbWVyjdXOmsA2hg78Fmp-oxskiaqrnprcII&e=>
Regards
JB
Le 11 févr. 2020 à 07:03, Gershi, Noam <no...@citi.com>> a écrit :
Hi,
I am searching for a detailed example how to use Apache Beam with Hive and/or Hcatalog?
Creating tables, inserting data and fetching it…
Noam Gershi Software Developer
ICG Technology – TLV Lab
T: +972 (3) 7405718
<image002.png>
RE: Apache Beam with Hive
Posted by "Gershi, Noam " <no...@citi.com>.
Thanx.
Looks like Hcatalog could work.
But - is the an example with ‘SELECT’ query?
JdbcIO probably not good to me, since my spark cluster in already configured to work with Hive. So – When I am code Spark pipelines, I can write queries without the need to give the user/password. – I would like to have something similar in Apache Beam.
From: [gmail.com] utkarsh.khare@gmail.com <ut...@gmail.com>
Sent: Tuesday, February 11, 2020 10:26 AM
To: user@beam.apache.org
Subject: Re: Apache Beam with Hive
While I've not created Hive tables using this .. reading and writing worked very well using HCatalogIO.
https://beam.apache.org/releases/javadoc/2.18.0/org/apache/beam/sdk/io/hcatalog/HCatalogIO.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__beam.apache.org_releases_javadoc_2.18.0_org_apache_beam_sdk_io_hcatalog_HCatalogIO.html&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=H-EFU9Q7yATgV6oX55NU_3j5EMt2X2k-9B3sJSMou9M&s=NsHbYvkYO2ZYHO43DxCjEdgW_eCBBZJZCMsdq_lZK4o&e=>
On Tue, Feb 11, 2020 at 11:38 AM Jean-Baptiste Onofre <jb...@nanthrax.net>> wrote:
Hi,
If you are ok to use Hive via JDBC, you can use JdbcIO and see the example in the javadoc:
https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L82<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_beam_blob_master_sdks_java_io_jdbc_src_main_java_org_apache_beam_sdk_io_jdbc_JdbcIO.java-23L82&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=H-EFU9Q7yATgV6oX55NU_3j5EMt2X2k-9B3sJSMou9M&s=OMH2F23vKbWVyjdXOmsA2hg78Fmp-oxskiaqrnprcII&e=>
Regards
JB
Le 11 févr. 2020 à 07:03, Gershi, Noam <no...@citi.com>> a écrit :
Hi,
I am searching for a detailed example how to use Apache Beam with Hive and/or Hcatalog?
Creating tables, inserting data and fetching it…
Noam Gershi Software Developer
ICG Technology – TLV Lab
T: +972 (3) 7405718
<image002.png>
Re: Apache Beam with Hive
Posted by ut...@gmail.com.
While I've not created Hive tables using this .. reading and writing worked
very well using HCatalogIO.
https://beam.apache.org/releases/javadoc/2.18.0/org/apache/beam/sdk/io/hcatalog/HCatalogIO.html
On Tue, Feb 11, 2020 at 11:38 AM Jean-Baptiste Onofre <jb...@nanthrax.net>
wrote:
> Hi,
>
> If you are ok to use Hive via JDBC, you can use JdbcIO and see the example
> in the javadoc:
>
>
> https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L82
>
> Regards
> JB
>
> Le 11 févr. 2020 à 07:03, Gershi, Noam <no...@citi.com> a écrit :
>
> Hi,
>
> I am searching for a detailed example how to use Apache Beam with Hive
> and/or Hcatalog?
> Creating tables, inserting data and fetching it…
>
>
> Noam Gershi Software Developer
> ICG Technology – TLV Lab
> *T*: +972 (3) 7405718
>
> <image002.png>
>
>
>
Re: Apache Beam with Hive
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Hi,
If you are ok to use Hive via JDBC, you can use JdbcIO and see the example in the javadoc:
https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L82 <https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L82>
Regards
JB
> Le 11 févr. 2020 à 07:03, Gershi, Noam <no...@citi.com> a écrit :
>
> Hi,
>
> I am searching for a detailed example how to use Apache Beam with Hive and/or Hcatalog?
> Creating tables, inserting data and fetching it…
>
>
> Noam Gershi Software Developer
> ICG Technology – TLV Lab
> T: +972 (3) 7405718
>
> <image002.png>