You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Neil Dev <ne...@gmail.com> on 2015/03/23 16:37:19 UTC
Starting sparkthrift server
Hi,
I am having issues starting spark-thriftserver. I'm running spark 1.3.o
with Hadoop 2.4.0. I would like to be able to change its port too so, I can
hive hive-thriftserver as well as spark-thriftserver running at the same
time.
Starting sparkthrift server:-
sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077
--executor-memory 2G
Error:-
I created the folder manually but still getting the following error----
Exception in thread "main" java.lang.IllegalArgumentException: Log
directory /tmp/spark-events does not exist.
I am getting the following error
15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error:
org.apache.thrift.transport.TTransportException: Could not create
ServerSocket on address0.0.0.0/0.0.0.0:10000.
at
org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:93)
at
org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:79)
at
org.apache.hive.service.auth.HiveAuthFactory.getServerSocket(HiveAuthFactory.java:236)
at
org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:69)
at java.lang.Thread.run(Thread.java:745)
Thanks
Neil
Re: Starting sparkthrift server
Posted by Denny Lee <de...@gmail.com>.
In that case, can you use the configurations to specify the folders? I'm
wondering if this actually Hive in play here and somehow the
/tmp/spark-events is being specified for the logs for Hive?
On Mon, Mar 23, 2015 at 2:00 PM Anubhav Agarwal <an...@gmail.com> wrote:
> When I start spark-shell (for example) it does not write to the
> /tmp/spark-events folder. It remains empty. I have even tried it after
> giving that folder rwx permission for user, group and others.
>
> Neil's colleague,
> Anu
>
> On Mon, Mar 23, 2015 at 4:50 PM, Denny Lee <de...@gmail.com> wrote:
>
>> When you say the job has access, do you mean that when you run
>> spark-submit
>> or spark-shell (for example), it is able to write to the /tmp/spark-events
>> folder?
>>
>>
>> On Mon, Mar 23, 2015 at 1:02 PM Neil Dev <ne...@gmail.com> wrote:
>>
>> > we are running this right now as root user and the folder
>> /tmp/spark-events
>> > was manually created and the Job has access to this folder
>> >
>> > On Mon, Mar 23, 2015 at 3:38 PM, Denny Lee <de...@gmail.com>
>> wrote:
>> >
>> >> It appears that you are running the thrift-server using the
>> spark-events
>> >> account but the /tmp/spark-events folder doesn't exist or the user
>> running
>> >> thrift-server does not have access to it. Have you been able to run
>> Hive
>> >> using the spark-events user so that way the /tmp/spark-events folder
>> has
>> >> been created. If you need to reassign the scratch dir / log dir to
>> another
>> >> folder (instead of /tmp/spark-events), you could use --hiveconf to
>> assign
>> >> those to another folder.
>> >>
>> >>
>> >> On Mon, Mar 23, 2015 at 8:39 AM Neil Dev <ne...@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I am having issues starting spark-thriftserver. I'm running spark
>> 1.3.o
>> >>> with Hadoop 2.4.0. I would like to be able to change its port too so,
>> I
>> >>> can
>> >>> hive hive-thriftserver as well as spark-thriftserver running at the
>> same
>> >>> time.
>> >>>
>> >>> Starting sparkthrift server:-
>> >>> sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077
>> >>> --executor-memory 2G
>> >>>
>> >>> Error:-
>> >>> I created the folder manually but still getting the following
>> error----
>> >>> Exception in thread "main" java.lang.IllegalArgumentException: Log
>> >>> directory /tmp/spark-events does not exist.
>> >>>
>> >>>
>> >>> I am getting the following error
>> >>> 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error:
>> >>> org.apache.thrift.transport.TTransportException: Could not create
>> >>> ServerSocket on address0.0.0.0/0.0.0.0:10000.
>> >>> at
>> >>>
>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:93)
>> >>> at
>> >>>
>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:79)
>> >>> at
>> >>> org.apache.hive.service.auth.HiveAuthFactory.getServerSocket(
>> >>> HiveAuthFactory.java:236)
>> >>> at
>> >>> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.
>> >>> run(ThriftBinaryCLIService.java:69)
>> >>> at java.lang.Thread.run(Thread.java:745)
>> >>>
>> >>> Thanks
>> >>> Neil
>> >>>
>> >>
>> >
>>
>
>
Re: Starting sparkthrift server
Posted by Anubhav Agarwal <an...@gmail.com>.
When I start spark-shell (for example) it does not write to the
/tmp/spark-events folder. It remains empty. I have even tried it after
giving that folder rwx permission for user, group and others.
Neil's colleague,
Anu
On Mon, Mar 23, 2015 at 4:50 PM, Denny Lee <de...@gmail.com> wrote:
> When you say the job has access, do you mean that when you run spark-submit
> or spark-shell (for example), it is able to write to the /tmp/spark-events
> folder?
>
>
> On Mon, Mar 23, 2015 at 1:02 PM Neil Dev <ne...@gmail.com> wrote:
>
> > we are running this right now as root user and the folder
> /tmp/spark-events
> > was manually created and the Job has access to this folder
> >
> > On Mon, Mar 23, 2015 at 3:38 PM, Denny Lee <de...@gmail.com>
> wrote:
> >
> >> It appears that you are running the thrift-server using the spark-events
> >> account but the /tmp/spark-events folder doesn't exist or the user
> running
> >> thrift-server does not have access to it. Have you been able to run
> Hive
> >> using the spark-events user so that way the /tmp/spark-events folder has
> >> been created. If you need to reassign the scratch dir / log dir to
> another
> >> folder (instead of /tmp/spark-events), you could use --hiveconf to
> assign
> >> those to another folder.
> >>
> >>
> >> On Mon, Mar 23, 2015 at 8:39 AM Neil Dev <ne...@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> I am having issues starting spark-thriftserver. I'm running spark 1.3.o
> >>> with Hadoop 2.4.0. I would like to be able to change its port too so, I
> >>> can
> >>> hive hive-thriftserver as well as spark-thriftserver running at the
> same
> >>> time.
> >>>
> >>> Starting sparkthrift server:-
> >>> sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077
> >>> --executor-memory 2G
> >>>
> >>> Error:-
> >>> I created the folder manually but still getting the following error----
> >>> Exception in thread "main" java.lang.IllegalArgumentException: Log
> >>> directory /tmp/spark-events does not exist.
> >>>
> >>>
> >>> I am getting the following error
> >>> 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error:
> >>> org.apache.thrift.transport.TTransportException: Could not create
> >>> ServerSocket on address0.0.0.0/0.0.0.0:10000.
> >>> at
> >>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:93)
> >>> at
> >>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:79)
> >>> at
> >>> org.apache.hive.service.auth.HiveAuthFactory.getServerSocket(
> >>> HiveAuthFactory.java:236)
> >>> at
> >>> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.
> >>> run(ThriftBinaryCLIService.java:69)
> >>> at java.lang.Thread.run(Thread.java:745)
> >>>
> >>> Thanks
> >>> Neil
> >>>
> >>
> >
>
Re: Starting sparkthrift server
Posted by Denny Lee <de...@gmail.com>.
When you say the job has access, do you mean that when you run spark-submit
or spark-shell (for example), it is able to write to the /tmp/spark-events
folder?
On Mon, Mar 23, 2015 at 1:02 PM Neil Dev <ne...@gmail.com> wrote:
> we are running this right now as root user and the folder /tmp/spark-events
> was manually created and the Job has access to this folder
>
> On Mon, Mar 23, 2015 at 3:38 PM, Denny Lee <de...@gmail.com> wrote:
>
>> It appears that you are running the thrift-server using the spark-events
>> account but the /tmp/spark-events folder doesn't exist or the user running
>> thrift-server does not have access to it. Have you been able to run Hive
>> using the spark-events user so that way the /tmp/spark-events folder has
>> been created. If you need to reassign the scratch dir / log dir to another
>> folder (instead of /tmp/spark-events), you could use --hiveconf to assign
>> those to another folder.
>>
>>
>> On Mon, Mar 23, 2015 at 8:39 AM Neil Dev <ne...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am having issues starting spark-thriftserver. I'm running spark 1.3.o
>>> with Hadoop 2.4.0. I would like to be able to change its port too so, I
>>> can
>>> hive hive-thriftserver as well as spark-thriftserver running at the same
>>> time.
>>>
>>> Starting sparkthrift server:-
>>> sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077
>>> --executor-memory 2G
>>>
>>> Error:-
>>> I created the folder manually but still getting the following error----
>>> Exception in thread "main" java.lang.IllegalArgumentException: Log
>>> directory /tmp/spark-events does not exist.
>>>
>>>
>>> I am getting the following error
>>> 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error:
>>> org.apache.thrift.transport.TTransportException: Could not create
>>> ServerSocket on address0.0.0.0/0.0.0.0:10000.
>>> at
>>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:93)
>>> at
>>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:79)
>>> at
>>> org.apache.hive.service.auth.HiveAuthFactory.getServerSocket(
>>> HiveAuthFactory.java:236)
>>> at
>>> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.
>>> run(ThriftBinaryCLIService.java:69)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Thanks
>>> Neil
>>>
>>
>
Re: Starting sparkthrift server
Posted by Neil Dev <ne...@gmail.com>.
we are running this right now as root user and the folder /tmp/spark-events
was manually created and the Job has access to this folder
On Mon, Mar 23, 2015 at 3:38 PM, Denny Lee <de...@gmail.com> wrote:
> It appears that you are running the thrift-server using the spark-events
> account but the /tmp/spark-events folder doesn't exist or the user running
> thrift-server does not have access to it. Have you been able to run Hive
> using the spark-events user so that way the /tmp/spark-events folder has
> been created. If you need to reassign the scratch dir / log dir to another
> folder (instead of /tmp/spark-events), you could use --hiveconf to assign
> those to another folder.
>
>
> On Mon, Mar 23, 2015 at 8:39 AM Neil Dev <ne...@gmail.com> wrote:
>
>> Hi,
>>
>> I am having issues starting spark-thriftserver. I'm running spark 1.3.o
>> with Hadoop 2.4.0. I would like to be able to change its port too so, I
>> can
>> hive hive-thriftserver as well as spark-thriftserver running at the same
>> time.
>>
>> Starting sparkthrift server:-
>> sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077
>> --executor-memory 2G
>>
>> Error:-
>> I created the folder manually but still getting the following error----
>> Exception in thread "main" java.lang.IllegalArgumentException: Log
>> directory /tmp/spark-events does not exist.
>>
>>
>> I am getting the following error
>> 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error:
>> org.apache.thrift.transport.TTransportException: Could not create
>> ServerSocket on address0.0.0.0/0.0.0.0:10000.
>> at
>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:93)
>> at
>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:79)
>> at
>> org.apache.hive.service.auth.HiveAuthFactory.getServerSocket(
>> HiveAuthFactory.java:236)
>> at
>> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.
>> run(ThriftBinaryCLIService.java:69)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Thanks
>> Neil
>>
>
Re: Starting sparkthrift server
Posted by Denny Lee <de...@gmail.com>.
It appears that you are running the thrift-server using the spark-events
account but the /tmp/spark-events folder doesn't exist or the user running
thrift-server does not have access to it. Have you been able to run Hive
using the spark-events user so that way the /tmp/spark-events folder has
been created. If you need to reassign the scratch dir / log dir to another
folder (instead of /tmp/spark-events), you could use --hiveconf to assign
those to another folder.
On Mon, Mar 23, 2015 at 8:39 AM Neil Dev <ne...@gmail.com> wrote:
> Hi,
>
> I am having issues starting spark-thriftserver. I'm running spark 1.3.o
> with Hadoop 2.4.0. I would like to be able to change its port too so, I can
> hive hive-thriftserver as well as spark-thriftserver running at the same
> time.
>
> Starting sparkthrift server:-
> sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077
> --executor-memory 2G
>
> Error:-
> I created the folder manually but still getting the following error----
> Exception in thread "main" java.lang.IllegalArgumentException: Log
> directory /tmp/spark-events does not exist.
>
>
> I am getting the following error
> 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error:
> org.apache.thrift.transport.TTransportException: Could not create
> ServerSocket on address0.0.0.0/0.0.0.0:10000.
> at
> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:93)
> at
> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:79)
> at
> org.apache.hive.service.auth.HiveAuthFactory.getServerSocket(
> HiveAuthFactory.java:236)
> at
> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.
> run(ThriftBinaryCLIService.java:69)
> at java.lang.Thread.run(Thread.java:745)
>
> Thanks
> Neil
>