You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Umesh Kacha <um...@gmail.com> on 2019/03/07 10:11:33 UTC
HoodiException
Hi I am trying to use Hoodie 0.4.4 jar in spark-shell and when I try to
save spark DataFrame as hoodi table I get the exception saying
NoClassDefFoundError: com/uber/hoodie/exception/HoodiException
Similar error occurs for HoodieWriteConfig values and
DataSourceWriteOptions classes.
Do I need to include others jars I can see hoodie-spark-0.4.4.jar is uber
fat jar which contains all transitive depenendices? No?
Please advise. Thank you.
Re: HoodiException
Posted by Vinoth Chandar <vi...@apache.org>.
Ah you are using spark shell. Sorry missed that
On Thu, Mar 7, 2019 at 8:52 AM Balaji Varadarajan
<v....@ymail.com.invalid> wrote:
> Hi Umesh,
> hoodie-spark-bundle is a shaded fat jar whereas hoodie-spark is not a fat
> jar.
> Balaji.V
>
> On Thursday, March 7, 2019, 8:50:22 AM PST, Umesh Kacha <
> umesh.kacha@gmail.com> wrote:
>
> Ok Balaji thanks will try to use hoodie-spark-bundle show below. What is
> the different between two though hoodie-spark and hoodie-spark-bundle?
>
> <groupId>com.uber.hoodie</groupId>
> <artifactId>hoodie-spark-bundle</artifactId>
> <version>0.4.5</version>
>
>
> On Thu, Mar 7, 2019 at 9:14 PM Balaji Varadarajan
> <v....@ymail.com.invalid> wrote:
>
> > Hi Umesh,
> > From your email, it looks like you are using hoodie-spark jar. Please
> use
> > hoodie-spark-bundle jar instead of hoodie-spark jar. Here is an example
> > from quickstart:
> > Quickstart | Hudi
> >
> >
> > |
> > |
> > | |
> > Quickstart | Hudi
> >
> >
> > |
> >
> > |
> >
> > |
> >
> >
> >
> >
> > Balaji.V
> > On Thursday, March 7, 2019, 7:00:17 AM PST, Vinoth Chandar <
> > vinoth@apache.org> wrote:
> >
> > Hi Umesh,
> >
> > Normally spark users build a fat/uber jar with all depedencies. This is
> the
> > same process to be followed for hoodie-spark as well, which will pull all
> > the dependencies like hoodie-common as well.
> >
> > Are you doing something similar?
> >
> > Thanks
> > Vinoth
> >
> > On Thu, Mar 7, 2019 at 2:11 AM Umesh Kacha <um...@gmail.com>
> wrote:
> >
> > > Hi I am trying to use Hoodie 0.4.4 jar in spark-shell and when I try to
> > > save spark DataFrame as hoodi table I get the exception saying
> > >
> > > NoClassDefFoundError: com/uber/hoodie/exception/HoodiException
> > >
> > > Similar error occurs for HoodieWriteConfig values and
> > > DataSourceWriteOptions classes.
> > >
> > > Do I need to include others jars I can see hoodie-spark-0.4.4.jar is
> uber
> > > fat jar which contains all transitive depenendices? No?
> > >
> > > Please advise. Thank you.
> > >
> >
>
Re: HoodiException
Posted by Balaji Varadarajan <v....@ymail.com.INVALID>.
Hi Umesh,
hoodie-spark-bundle is a shaded fat jar whereas hoodie-spark is not a fat jar.
Balaji.V
On Thursday, March 7, 2019, 8:50:22 AM PST, Umesh Kacha <um...@gmail.com> wrote:
Ok Balaji thanks will try to use hoodie-spark-bundle show below. What is
the different between two though hoodie-spark and hoodie-spark-bundle?
<groupId>com.uber.hoodie</groupId>
<artifactId>hoodie-spark-bundle</artifactId>
<version>0.4.5</version>
On Thu, Mar 7, 2019 at 9:14 PM Balaji Varadarajan
<v....@ymail.com.invalid> wrote:
> Hi Umesh,
> From your email, it looks like you are using hoodie-spark jar. Please use
> hoodie-spark-bundle jar instead of hoodie-spark jar. Here is an example
> from quickstart:
> Quickstart | Hudi
>
>
> |
> |
> | |
> Quickstart | Hudi
>
>
> |
>
> |
>
> |
>
>
>
>
> Balaji.V
> On Thursday, March 7, 2019, 7:00:17 AM PST, Vinoth Chandar <
> vinoth@apache.org> wrote:
>
> Hi Umesh,
>
> Normally spark users build a fat/uber jar with all depedencies. This is the
> same process to be followed for hoodie-spark as well, which will pull all
> the dependencies like hoodie-common as well.
>
> Are you doing something similar?
>
> Thanks
> Vinoth
>
> On Thu, Mar 7, 2019 at 2:11 AM Umesh Kacha <um...@gmail.com> wrote:
>
> > Hi I am trying to use Hoodie 0.4.4 jar in spark-shell and when I try to
> > save spark DataFrame as hoodi table I get the exception saying
> >
> > NoClassDefFoundError: com/uber/hoodie/exception/HoodiException
> >
> > Similar error occurs for HoodieWriteConfig values and
> > DataSourceWriteOptions classes.
> >
> > Do I need to include others jars I can see hoodie-spark-0.4.4.jar is uber
> > fat jar which contains all transitive depenendices? No?
> >
> > Please advise. Thank you.
> >
>
Re: HoodiException
Posted by Umesh Kacha <um...@gmail.com>.
Ok Balaji thanks will try to use hoodie-spark-bundle show below. What is
the different between two though hoodie-spark and hoodie-spark-bundle?
<groupId>com.uber.hoodie</groupId>
<artifactId>hoodie-spark-bundle</artifactId>
<version>0.4.5</version>
On Thu, Mar 7, 2019 at 9:14 PM Balaji Varadarajan
<v....@ymail.com.invalid> wrote:
> Hi Umesh,
> From your email, it looks like you are using hoodie-spark jar. Please use
> hoodie-spark-bundle jar instead of hoodie-spark jar. Here is an example
> from quickstart:
> Quickstart | Hudi
>
>
> |
> |
> | |
> Quickstart | Hudi
>
>
> |
>
> |
>
> |
>
>
>
>
> Balaji.V
> On Thursday, March 7, 2019, 7:00:17 AM PST, Vinoth Chandar <
> vinoth@apache.org> wrote:
>
> Hi Umesh,
>
> Normally spark users build a fat/uber jar with all depedencies. This is the
> same process to be followed for hoodie-spark as well, which will pull all
> the dependencies like hoodie-common as well.
>
> Are you doing something similar?
>
> Thanks
> Vinoth
>
> On Thu, Mar 7, 2019 at 2:11 AM Umesh Kacha <um...@gmail.com> wrote:
>
> > Hi I am trying to use Hoodie 0.4.4 jar in spark-shell and when I try to
> > save spark DataFrame as hoodi table I get the exception saying
> >
> > NoClassDefFoundError: com/uber/hoodie/exception/HoodiException
> >
> > Similar error occurs for HoodieWriteConfig values and
> > DataSourceWriteOptions classes.
> >
> > Do I need to include others jars I can see hoodie-spark-0.4.4.jar is uber
> > fat jar which contains all transitive depenendices? No?
> >
> > Please advise. Thank you.
> >
>
Re: HoodiException
Posted by Balaji Varadarajan <v....@ymail.com.INVALID>.
Hi Umesh,
From your email, it looks like you are using hoodie-spark jar. Please use hoodie-spark-bundle jar instead of hoodie-spark jar. Here is an example from quickstart:
Quickstart | Hudi
|
|
| |
Quickstart | Hudi
|
|
|
Balaji.V
On Thursday, March 7, 2019, 7:00:17 AM PST, Vinoth Chandar <vi...@apache.org> wrote:
Hi Umesh,
Normally spark users build a fat/uber jar with all depedencies. This is the
same process to be followed for hoodie-spark as well, which will pull all
the dependencies like hoodie-common as well.
Are you doing something similar?
Thanks
Vinoth
On Thu, Mar 7, 2019 at 2:11 AM Umesh Kacha <um...@gmail.com> wrote:
> Hi I am trying to use Hoodie 0.4.4 jar in spark-shell and when I try to
> save spark DataFrame as hoodi table I get the exception saying
>
> NoClassDefFoundError: com/uber/hoodie/exception/HoodiException
>
> Similar error occurs for HoodieWriteConfig values and
> DataSourceWriteOptions classes.
>
> Do I need to include others jars I can see hoodie-spark-0.4.4.jar is uber
> fat jar which contains all transitive depenendices? No?
>
> Please advise. Thank you.
>
Re: HoodiException
Posted by Vinoth Chandar <vi...@apache.org>.
Hi Umesh,
Normally spark users build a fat/uber jar with all depedencies. This is the
same process to be followed for hoodie-spark as well, which will pull all
the dependencies like hoodie-common as well.
Are you doing something similar?
Thanks
Vinoth
On Thu, Mar 7, 2019 at 2:11 AM Umesh Kacha <um...@gmail.com> wrote:
> Hi I am trying to use Hoodie 0.4.4 jar in spark-shell and when I try to
> save spark DataFrame as hoodi table I get the exception saying
>
> NoClassDefFoundError: com/uber/hoodie/exception/HoodiException
>
> Similar error occurs for HoodieWriteConfig values and
> DataSourceWriteOptions classes.
>
> Do I need to include others jars I can see hoodie-spark-0.4.4.jar is uber
> fat jar which contains all transitive depenendices? No?
>
> Please advise. Thank you.
>