You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Denny Lee <de...@gmail.com> on 2014/09/04 07:47:00 UTC

Starting Thriftserver via hostname on Spark 1.1 RC4?

When I start the thrift server (on Spark 1.1 RC4) via:
./sbin/start-thriftserver.sh --master spark://hostname:7077 --driver-class-path $CLASSPATH

It appears that the thrift server is starting off of localhost as opposed to hostname.  I have set the spark-env.sh to use the hostname, modified the /etc/hosts for the hostname, and it appears to work properly.

But when I start the thrift server, connectivity can only be via localhost:10000 as opposed to hostname:10000.

Any ideas on what configurations I may be setting incorrectly here?

Thanks!
Denny

Re: Starting Thriftserver via hostname on Spark 1.1 RC4?

Posted by Denny Lee <de...@gmail.com>.
Ahh got it - I knew I was missing something  - appreciate the clarification! :)


On September 4, 2014 at 10:27:44, Cheng Lian (lian.cs.zju@gmail.com) wrote:

You may configure listening host and port in the same way as HiveServer2 of Hive, namely:

via environment variables

HIVE_SERVER2_THRIFT_BIND_HOST

HIVE_SERVER2_THRIFT_PORT
via system properties

hive.server2.thrift.bind.host

hive.server2.thrift.port
For the latter, you may pass the properties via command line:


$ ./sbin/start-thriftserver2 --hiveconf hive.server2.thrift.bind.host <your-hostname> --hiveconf hive.server2.thrift.port <your-port>

This behavior is inherited from Hive since Spark SQL Thrift server is a variant of HiveServer2.

​


On Wed, Sep 3, 2014 at 10:47 PM, Denny Lee <de...@gmail.com> wrote:
When I start the thrift server (on Spark 1.1 RC4) via:
./sbin/start-thriftserver.sh --master spark://hostname:7077 --driver-class-path $CLASSPATH

It appears that the thrift server is starting off of localhost as opposed to hostname.  I have set the spark-env.sh to use the hostname, modified the /etc/hosts for the hostname, and it appears to work properly.

But when I start the thrift server, connectivity can only be via localhost:10000 as opposed to hostname:10000.

Any ideas on what configurations I may be setting incorrectly here?

Thanks!
Denny

Re: Starting Thriftserver via hostname on Spark 1.1 RC4?

Posted by Cheng Lian <li...@gmail.com>.
You may configure listening host and port in the same way as HiveServer2 of
Hive, namely:

   -

   via environment variables
    - HIVE_SERVER2_THRIFT_BIND_HOST
      - HIVE_SERVER2_THRIFT_PORT
    -

   via system properties
    - hive.server2.thrift.bind.host
      - hive.server2.thrift.port

For the latter, you may pass the properties via command line:

$ ./sbin/start-thriftserver2 --hiveconf hive.server2.thrift.bind.host
<your-hostname> --hiveconf hive.server2.thrift.port <your-port>

This behavior is inherited from Hive since Spark SQL Thrift server is a
variant of HiveServer2.
​


On Wed, Sep 3, 2014 at 10:47 PM, Denny Lee <de...@gmail.com> wrote:

> When I start the thrift server (on Spark 1.1 RC4) via:
> ./sbin/start-thriftserver.sh --master spark://hostname:7077
> --driver-class-path $CLASSPATH
>
> It appears that the thrift server is starting off of localhost as opposed
> to hostname.  I have set the spark-env.sh to use the hostname, modified the
> /etc/hosts for the hostname, and it appears to work properly.
>
> But when I start the thrift server, connectivity can only be via
> localhost:10000 as opposed to hostname:10000.
>
> Any ideas on what configurations I may be setting incorrectly here?
>
> Thanks!
> Denny
>
>

Re: Starting Thriftserver via hostname on Spark 1.1 RC4?

Posted by Gurvinder Singh <gu...@uninett.no>.
I want to add that there a regression when using pyspark to read data
from HDFS. its performance during map tasks has gone down approx 1 ->
0.5x. I have tested the 1.0.2 and the performance was fine, but the 1.1
release candidate has this issue. I tested by setting the following
properties to make sure it was not due to these.

set("spark.io.compression.codec","lzf").set("spark.shuffle.spill","false")

in conf object. Let me know if you need further information.

Regards,
Gurvinder
On 09/04/2014 07:47 AM, Denny Lee wrote:
> When I start the thrift server (on Spark 1.1 RC4) via:
> ./sbin/start-thriftserver.sh --master spark://hostname:7077
> --driver-class-path $CLASSPATH
> 
> It appears that the thrift server is starting off of localhost as
> opposed to hostname.  I have set the spark-env.sh to use the hostname,
> modified the /etc/hosts for the hostname, and it appears to work properly.
> 
> But when I start the thrift server, connectivity can only be via
> localhost:10000 as opposed to hostname:10000.
> 
> Any ideas on what configurations I may be setting incorrectly here?
> 
> Thanks!
> Denny
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org