You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by Andras Beni <an...@cloudera.com> on 2020/07/20 15:49:12 UTC
Re: sparkmagic + Hive Classnotfound exception
Hello Murat,
Have you managed to fix the problem? If not, maybe a stack trace would
help us identify the problem.
For the next time: user@livy.incubator.apache.org mailing list is more
suitable for this kind of question.
Best regards,
Andras
On Fri, Jul 10, 2020 at 1:32 PM murat migdisoglu
<mu...@gmail.com> wrote:
>
> Hello,
>
> I'm testing sparkmagic library to interact with Livy using Jupyter. During
> the session creation phase, I see a ClassNotFound exception related to
> Hive. I didn't enable the hive context as we don't use it
> (livy.repl.enable-hive-context
> = false)
> *java.lang.NoClassDefFoundError: org/apache/spark/sql/hive/HiveContext*
>
> Is there anybody else experiencing this problem ?
>
> Best
> Murat
>
> --
> "Talkers aren’t good doers. Rest assured that we’re going there to use our
> hands, not our tongues."
> W. Shakespeare
Re: sparkmagic + Hive Classnotfound exception
Posted by murat migdisoglu <mu...@gmail.com>.
>
>
> Hello Andras, Thank you for your interest in my problem.
> I still couldn't fix the problem.
> I copy below the stack traces: (jupyter and yarn)
> *on jupyter:*
>
>> ---------------------------------------------------------------------------LivyUnexpectedStatusException Traceback (most recent call last)/opt/conda/lib/python3.8/site-packages/hdijupyterutils/ipywidgetfactory.py in submit_clicked(self, button) 63 64 def submit_clicked(self, button):---> 65 self.parent_widget.run()
>> /opt/conda/lib/python3.8/site-packages/sparkmagic/controllerwidget/createsessionwidget.py in run(self) 56 57 try:---> 58 self.spark_controller.add_session(alias, endpoint, skip, properties) 59 except ValueError as e: 60 self.ipython_display.send_error("""Could not add session with
>> /opt/conda/lib/python3.8/site-packages/sparkmagic/livyclientlib/sparkcontroller.py in add_session(self, name, endpoint, skip_if_exists, properties) 88 session = self._livy_session(http_client, properties, self.ipython_display) 89 self.session_manager.add_session(name, session)---> 90 session.start() 91 92 def get_session_id_for_client(self, name):
>> /opt/conda/lib/python3.8/site-packages/sparkmagic/livyclientlib/livysession.py in start(self) 162 else: 163 command = Command("sqlContext")--> 164 (success, out, mimetype) = command.execute(self) 165 if success: 166 self.ipython_display.writeln(u"SparkContext available as 'sc'.")
>> /opt/conda/lib/python3.8/site-packages/sparkmagic/livyclientlib/command.py in execute(self, session) 38 statement_id = -1 39 try:---> 40 session.wait_for_idle() 41 data = {u"code": self.code} 42 response = session.http_client.post_statement(session.id, data)
>> /opt/conda/lib/python3.8/site-packages/sparkmagic/livyclientlib/livysession.py in wait_for_idle(self, seconds_to_wait) 256 .format(self.id, self.status) 257 self.logger.error(error)--> 258 raise LivyUnexpectedStatusException(u'{} See logs:\n{}'.format(error, self.get_logs())) 259 260 if seconds_to_wait <= 0.0:
>>
>>
>> *on yarn*
>> Exception in thread "Thread-30" java.lang.NoClassDefFoundError:
>> org/apache/spark/sql/hive/HiveContext
>> at java.lang.Class.getDeclaredMethods0(Native Method)
>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
>> at java.lang.Class.privateGetPublicMethods(Class.java:2902)
>> at java.lang.Class.getMethods(Class.java:1615)
>> at
>> py4j.reflection.ReflectionEngine.getMethodsByNameAndLength(ReflectionEngine.java:345)
>> at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:305)
>> at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
>> at py4j.Gateway.invoke(Gateway.java:274)
>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>> at py4j.GatewayConnection.run(GatewayConnection.java:238)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.spark.sql.hive.HiveContext
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>> ... 12 more
>>
>> On Mon, Jul 20, 2020 at 6:49 PM Andras Beni <an...@cloudera.com>
>> wrote:
>>
>>> Hello Murat,
>>>
>>> Have you managed to fix the problem? If not, maybe a stack trace would
>>> help us identify the problem.
>>>
>>> For the next time: user@livy.incubator.apache.org mailing list is more
>>> suitable for this kind of question.
>>>
>>> Best regards,
>>> Andras
>>>
>>> On Fri, Jul 10, 2020 at 1:32 PM murat migdisoglu
>>> <mu...@gmail.com> wrote:
>>> >
>>> > Hello,
>>> >
>>> > I'm testing sparkmagic library to interact with Livy using Jupyter.
>>> During
>>> > the session creation phase, I see a ClassNotFound exception related to
>>> > Hive. I didn't enable the hive context as we don't use it
>>> > (livy.repl.enable-hive-context
>>> > = false)
>>> > *java.lang.NoClassDefFoundError: org/apache/spark/sql/hive/HiveContext*
>>> >
>>> > Is there anybody else experiencing this problem ?
>>> >
>>> > Best
>>> > Murat
>>> >
>>> > --
>>> > "Talkers aren’t good doers. Rest assured that we’re going there to use
>>> our
>>> > hands, not our tongues."
>>> > W. Shakespeare
>>>
>>
>>
>> --
>> "Talkers aren’t good doers. Rest assured that we’re going there to use
>> our hands, not our tongues."
>> W. Shakespeare
>>
>>
>> --
>> "Talkers aren’t good doers. Rest assured that we’re going there to use
>> our hands, not our tongues."
>> W. Shakespeare
>>
>
>
> --
> "Talkers aren’t good doers. Rest assured that we’re going there to use
> our hands, not our tongues."
> W. Shakespeare
>
--
"Talkers aren’t good doers. Rest assured that we’re going there to use our
hands, not our tongues."
W. Shakespeare