You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@crunch.apache.org by "Barry,Nathan" <NB...@CERNER.COM> on 2014/03/12 16:27:28 UTC
convert a scrunch.PCollection to a crunch.PCollection ?
Hi
I'm migrating some code from crunch to scrunch and I'm running into an issue leveraging a crunch based API that takes a crunch PCollection as a parameter but I have a scrunch PCollection which results in compilation errors when trying to pass the scrunch Pcollection into the API
What am I missing? I _think_ I have the proper conversion imports in place.
[ERROR] found : org.apache.crunch.scrunch.PCollection[org.apache.hadoop.hbase.client.Put]
[ERROR] required: org.apache.crunch.PCollection[org.apache.hadoop.hbase.client.Put]
Thanks,
Nathan
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
Re: convert a scrunch.PCollection to a crunch.PCollection ?
Posted by "Barry,Nathan" <NB...@CERNER.COM>.
_doh_
Missed that one
From: Josh Wills <jw...@cloudera.com>>
Reply-To: "user@crunch.apache.org<ma...@crunch.apache.org>" <us...@crunch.apache.org>>
Date: Wednesday, March 12, 2014 at 10:44 AM
To: "user@crunch.apache.org<ma...@crunch.apache.org>" <us...@crunch.apache.org>>
Subject: Re: convert a scrunch.PCollection to a crunch.PCollection ?
Scrunch PCollections have a member called "native" that returns the underlying Java PCollection. I think we could allow the conversion to happen automatically by updating o.a.c.s.Conversions to include implicit methods for going _back_ from a Scrunch PCollection to a Java PCollection (right now, only the Java -> Scala methods are provided.)
On Wed, Mar 12, 2014 at 8:27 AM, Barry,Nathan <NB...@cerner.com>> wrote:
Hi
I'm migrating some code from crunch to scrunch and I'm running into an issue leveraging a crunch based API that takes a crunch PCollection as a parameter but I have a scrunch PCollection which results in compilation errors when trying to pass the scrunch Pcollection into the API
What am I missing? I _think_ I have the proper conversion imports in place.
[ERROR] found : org.apache.crunch.scrunch.PCollection[org.apache.hadoop.hbase.client.Put]
[ERROR] required: org.apache.crunch.PCollection[org.apache.hadoop.hbase.client.Put]
Thanks,
Nathan
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024<tel:%28%2B1%29%20%28816%29221-1024>.
--
Director of Data Science
Cloudera<https://urldefense.proofpoint.com/v1/url?u=http://www.cloudera.com&k=PmKqfXspAHNo6iYJ48Q45A%3D%3D%0A&r=bSlK0OoXv%2FhAR3K6OwkDzQ7GANpYeh6g8gdpXiWlUWw%3D%0A&m=rS1hwZwuXljlQh3r9aRV1rqOhPpYA7Og49uiDg5M9nw%3D%0A&s=25da0547e6d98dfef47f42328782c82f4453fcd421f86080e034da44032f672e>
Twitter: @josh_wills<https://urldefense.proofpoint.com/v1/url?u=http://twitter.com/josh_wills&k=PmKqfXspAHNo6iYJ48Q45A%3D%3D%0A&r=bSlK0OoXv%2FhAR3K6OwkDzQ7GANpYeh6g8gdpXiWlUWw%3D%0A&m=rS1hwZwuXljlQh3r9aRV1rqOhPpYA7Og49uiDg5M9nw%3D%0A&s=38d71c8b9d15c622c35a99bb2211d2329cbe88933309c5b4d78a0e6481c0b0a3>
Re: convert a scrunch.PCollection to a crunch.PCollection ?
Posted by Josh Wills <jw...@cloudera.com>.
Scrunch PCollections have a member called "native" that returns the
underlying Java PCollection. I think we could allow the conversion to
happen automatically by updating o.a.c.s.Conversions to include implicit
methods for going _back_ from a Scrunch PCollection to a Java PCollection
(right now, only the Java -> Scala methods are provided.)
On Wed, Mar 12, 2014 at 8:27 AM, Barry,Nathan <NB...@cerner.com> wrote:
> Hi
>
> I'm migrating some code from crunch to scrunch and I'm running into an
> issue leveraging a crunch based API that takes a crunch PCollection as a
> parameter but I have a scrunch PCollection which results in compilation
> errors when trying to pass the scrunch Pcollection into the API
>
> What am I missing? I _think_ I have the proper conversion imports in
> place.
>
> [ERROR] found : org.apache.crunch.scrunch.
> PCollection[org.apache.hadoop.hbase.client.Put]
> [ERROR] required: org.apache.crunch.PCollection[org.apache.hadoop.hbase.
> client.Put]
>
> Thanks,
>
> Nathan
>
> CONFIDENTIALITY NOTICE This message and any included attachments are
> from Cerner Corporation and are intended only for the addressee. The
> information contained in this message is confidential and may constitute
> inside or non-public information under international, federal, or state
> securities laws. Unauthorized forwarding, printing, copying, distribution,
> or use of such information is strictly prohibited and may be unlawful. If
> you are not the addressee, please promptly delete this message and notify
> the sender of the delivery error by e-mail or you may call Cerner's
> corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>