You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@gobblin.apache.org by Vicky Kak <vi...@gmail.com> on 2017/10/18 11:28:12 UTC
GAAS installation issues.
Hey Guys,
I have been trying to configure GAAS based and have got the following
observations
1) I am using gobblin-service.sh start to start the Orchestrator
Application however I don't see it getting started. I am getting the
following information in the master.out log as
WARN [NativeCodeLoader] Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
ERROR [GobblinServiceManager] Callback error:
gobblin.util.callbacks.CallbacksDispatcher$CallbackCallable@1f51431 on
gobblin.service.modules.orchestration.Orchestrator@66f659e6
:java.lang.UnsupportedOperationException
I could see that the gobblin-service.sh is expecting HADOOP_HOME to be set
in the following line
CLASSPATH=${FWDIR_CONF}:${GOBBLIN_JARS}:${SERVICE_CONF_DIR}:${HADOOP_HOME}/lib
I am not sure what needs to be set here, I did configure the HADOOP_HOME to
the hadoop-2.7.2 however I do still see the same issue.
2) Using the $GOBBIN_HOME/bin/./gobblin-service.sh start is starting the
GobblinServiceManager process. I figured out the port for the process using
vicky@vicky-Latitude-E5570:~/git/bpuholdings/dip/dip-dev/runtime/gobblin-dist/bin$
netstat -a -p | grep 1460
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 [::]:48039 [::]:*
LISTEN 1460/java
unix 2 [ ] STREAM CONNECTED 2557156
1460/java
unix 2 [ ] STREAM CONNECTED 2549041
1460/java
The server is listining on the 48039 but does not list the JobFlows (
http://localhost:48039/flowconfigs) , Since I am staring it for the first
time there would be no JobFlows I would expect the empty list. I saw the
implementation of FlowConfigsResource which does not have the
implementation to render the FlowConfigs by default.
I would propose to have a GET call for listing all the Flow Specs. There
should be a way to configure the default port for the Orchestrator ( i.e
GobblinServiceManager).
3) I did tweak the log4j-cluster.properties at the
GOBBLIN_HOME/conf/service, however don't see the changes are being taken
from there. I am unable to find out how the log4j related properties will
be taken from the log4j-cluster.properties file. I would expect the log4j
properties as log4j.properties. I did try to change the
log4j-cluster.properties to log4j.properties however did not get the
expected results. Can some one explain how it is being used?
4) I did not configure the properties in the
GOBBLIN_HOME/conf/service/application.conf and tried to bring up the
Gobblin cluster by calling "./gobblin-cluster-master.sh start" and
"./gobblin-cluster-worker.sh start", this did not help as there are errors
coming.
When I am calling the goblin-cluster-master.sh I see the following error
appearing in the master.out
Exception in thread "main" java.io.FileNotFoundException:
log4j-cluster.properties (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at
gobblin.util.logs.Log4jConfigurationHelper.updateLog4jConfiguration(Log4jConfigurationHelper.java:51)
at
gobblin.cluster.GobblinClusterManager.main(GobblinClusterManager.java:723)
In this case the log4j-cluster.properties are being read.
Can someone jot the steps to get the GAAS up?
Here are the steps that I think should be done
1) Bring the Orchestrator up by invoking
$GOBBLIN_HOME/bin/gobblin-service.sh start
2) Bring the Gobblin Cluster up by invoking
$GOBBLIN_HOME/bin/gobblin-cluster-master.sh start
3) Brink the Gobblin Cluster worker's by invoking
$GOBBLIN_HOME/bin/gobblin-cluster-worker.sh start
And for all the above steps we need to configure the Helix/Kafka etc.
Regards,
Vicky
Re: GAAS installation issues.
Posted by Vicky Kak <vi...@gmail.com>.
I did spend some time looking at the code found in the code the reference
of the log4j-cluster.properties for the cluster related classes i.e
gobblin.cluster.GobblinClusterManager and gobblin.cluster.GobblinTaskRunner
Log4jConfigurationHelper.updateLog4jConfiguration(GobblinTaskRunner.class,
GobblinClusterConfigurationKeys.GOBBLIN_CLUSTER_LOG4J_CONFIGURATION_FILE,
GobblinClusterConfigurationKeys.GOBBLIN_CLUSTER_LOG4J_CONFIGURATION_FILE);
On Wed, Oct 18, 2017 at 5:10 PM, Vicky Kak <vi...@gmail.com> wrote:
> >>ERROR [GobblinServiceManager] Callback error:
> gobblin.util.callbacks.CallbacksDispatcher$CallbackCallable@1f51431 on
> gobblin.service.modules.orchestration.Orchestrator@66f659e6:java.lang.>>
> UnsupportedOperationException
>
> Setting the proper value for the following properties got rid of the above
> error
> ${gobblin.service.work.dir}
>
> Regards,
> Vicky
>
>
>
> On Wed, Oct 18, 2017 at 4:58 PM, Vicky Kak <vi...@gmail.com> wrote:
>
>> Hey Guys,
>>
>> I have been trying to configure GAAS based and have got the following
>> observations
>>
>> 1) I am using gobblin-service.sh start to start the Orchestrator
>> Application however I don't see it getting started. I am getting the
>> following information in the master.out log as
>>
>> WARN [NativeCodeLoader] Unable to load native-hadoop library for your
>> platform... using builtin-java classes where applicable
>> ERROR [GobblinServiceManager] Callback error:
>> gobblin.util.callbacks.CallbacksDispatcher$CallbackCallable@1f51431 on
>> gobblin.service.modules.orchestration.Orchestrator@66f659e6:
>> java.lang.UnsupportedOperationException
>>
>> I could see that the gobblin-service.sh is expecting HADOOP_HOME to be
>> set in the following line
>> CLASSPATH=${FWDIR_CONF}:${GOBBLIN_JARS}:${SERVICE_CONF_DIR}:
>> ${HADOOP_HOME}/lib
>>
>> I am not sure what needs to be set here, I did configure the HADOOP_HOME
>> to the hadoop-2.7.2 however I do still see the same issue.
>>
>> 2) Using the $GOBBIN_HOME/bin/./gobblin-service.sh start is starting the
>> GobblinServiceManager process. I figured out the port for the process using
>>
>> vicky@vicky-Latitude-E5570:~/git/bpuholdings/dip/dip-dev/runtime/gobblin-dist/bin$
>> netstat -a -p | grep 1460
>> (Not all processes could be identified, non-owned process info
>> will not be shown, you would have to be root to see it all.)
>> tcp6 0 0 [::]:48039 [::]:*
>> LISTEN 1460/java
>> unix 2 [ ] STREAM CONNECTED 2557156
>> 1460/java
>> unix 2 [ ] STREAM CONNECTED 2549041
>> 1460/java
>>
>>
>> The server is listining on the 48039 but does not list the JobFlows (
>> http://localhost:48039/flowconfigs) , Since I am staring it for the
>> first time there would be no JobFlows I would expect the empty list. I saw
>> the implementation of FlowConfigsResource which does not have the
>> implementation to render the FlowConfigs by default.
>>
>> I would propose to have a GET call for listing all the Flow Specs. There
>> should be a way to configure the default port for the Orchestrator ( i.e
>> GobblinServiceManager).
>>
>> 3) I did tweak the log4j-cluster.properties at the
>> GOBBLIN_HOME/conf/service, however don't see the changes are being taken
>> from there. I am unable to find out how the log4j related properties will
>> be taken from the log4j-cluster.properties file. I would expect the log4j
>> properties as log4j.properties. I did try to change the
>> log4j-cluster.properties to log4j.properties however did not get the
>> expected results. Can some one explain how it is being used?
>>
>> 4) I did not configure the properties in the
>> GOBBLIN_HOME/conf/service/application.conf and tried to bring up the
>> Gobblin cluster by calling "./gobblin-cluster-master.sh start" and
>> "./gobblin-cluster-worker.sh start", this did not help as there are errors
>> coming.
>>
>> When I am calling the goblin-cluster-master.sh I see the following error
>> appearing in the master.out
>>
>> Exception in thread "main" java.io.FileNotFoundException:
>> log4j-cluster.properties (No such file or directory)
>> at java.io.FileInputStream.open0(Native Method)
>> at java.io.FileInputStream.open(FileInputStream.java:195)
>> at java.io.FileInputStream.<init>(FileInputStream.java:138)
>> at java.io.FileInputStream.<init>(FileInputStream.java:93)
>> at gobblin.util.logs.Log4jConfigurationHelper.updateLog4jConfig
>> uration(Log4jConfigurationHelper.java:51)
>> at gobblin.cluster.GobblinClusterManager.main(GobblinClusterMan
>> ager.java:723)
>>
>> In this case the log4j-cluster.properties are being read.
>>
>> Can someone jot the steps to get the GAAS up?
>>
>>
>> Here are the steps that I think should be done
>>
>> 1) Bring the Orchestrator up by invoking
>> $GOBBLIN_HOME/bin/gobblin-service.sh start
>>
>> 2) Bring the Gobblin Cluster up by invoking
>> $GOBBLIN_HOME/bin/gobblin-cluster-master.sh start
>>
>> 3) Brink the Gobblin Cluster worker's by invoking
>> $GOBBLIN_HOME/bin/gobblin-cluster-worker.sh start
>>
>> And for all the above steps we need to configure the Helix/Kafka etc.
>>
>>
>> Regards,
>> Vicky
>>
>>
>>
>
Re: GAAS installation issues.
Posted by Vicky Kak <vi...@gmail.com>.
>>ERROR [GobblinServiceManager] Callback error: gobblin.util.callbacks.
CallbacksDispatcher$CallbackCallable@1f51431 on gobblin.service.modules.
orchestration.Orchestrator@66f659e6:java.lang.
>>UnsupportedOperationException
Setting the proper value for the following properties got rid of the above
error
${gobblin.service.work.dir}
Regards,
Vicky
On Wed, Oct 18, 2017 at 4:58 PM, Vicky Kak <vi...@gmail.com> wrote:
> Hey Guys,
>
> I have been trying to configure GAAS based and have got the following
> observations
>
> 1) I am using gobblin-service.sh start to start the Orchestrator
> Application however I don't see it getting started. I am getting the
> following information in the master.out log as
>
> WARN [NativeCodeLoader] Unable to load native-hadoop library for your
> platform... using builtin-java classes where applicable
> ERROR [GobblinServiceManager] Callback error: gobblin.util.callbacks.
> CallbacksDispatcher$CallbackCallable@1f51431 on gobblin.service.modules.
> orchestration.Orchestrator@66f659e6:java.lang.
> UnsupportedOperationException
>
> I could see that the gobblin-service.sh is expecting HADOOP_HOME to be set
> in the following line
> CLASSPATH=${FWDIR_CONF}:${GOBBLIN_JARS}:${SERVICE_CONF_
> DIR}:${HADOOP_HOME}/lib
>
> I am not sure what needs to be set here, I did configure the HADOOP_HOME
> to the hadoop-2.7.2 however I do still see the same issue.
>
> 2) Using the $GOBBIN_HOME/bin/./gobblin-service.sh start is starting the
> GobblinServiceManager process. I figured out the port for the process using
>
> vicky@vicky-Latitude-E5570:~/git/bpuholdings/dip/dip-dev/runtime/gobblin-dist/bin$
> netstat -a -p | grep 1460
> (Not all processes could be identified, non-owned process info
> will not be shown, you would have to be root to see it all.)
> tcp6 0 0 [::]:48039 [::]:*
> LISTEN 1460/java
> unix 2 [ ] STREAM CONNECTED 2557156
> 1460/java
> unix 2 [ ] STREAM CONNECTED 2549041
> 1460/java
>
>
> The server is listining on the 48039 but does not list the JobFlows (
> http://localhost:48039/flowconfigs) , Since I am staring it for the first
> time there would be no JobFlows I would expect the empty list. I saw the
> implementation of FlowConfigsResource which does not have the
> implementation to render the FlowConfigs by default.
>
> I would propose to have a GET call for listing all the Flow Specs. There
> should be a way to configure the default port for the Orchestrator ( i.e
> GobblinServiceManager).
>
> 3) I did tweak the log4j-cluster.properties at the
> GOBBLIN_HOME/conf/service, however don't see the changes are being taken
> from there. I am unable to find out how the log4j related properties will
> be taken from the log4j-cluster.properties file. I would expect the log4j
> properties as log4j.properties. I did try to change the
> log4j-cluster.properties to log4j.properties however did not get the
> expected results. Can some one explain how it is being used?
>
> 4) I did not configure the properties in the GOBBLIN_HOME/conf/service/application.conf
> and tried to bring up the Gobblin cluster by calling
> "./gobblin-cluster-master.sh start" and "./gobblin-cluster-worker.sh
> start", this did not help as there are errors coming.
>
> When I am calling the goblin-cluster-master.sh I see the following error
> appearing in the master.out
>
> Exception in thread "main" java.io.FileNotFoundException:
> log4j-cluster.properties (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at java.io.FileInputStream.<init>(FileInputStream.java:93)
> at gobblin.util.logs.Log4jConfigurationHelper.
> updateLog4jConfiguration(Log4jConfigurationHelper.java:51)
> at gobblin.cluster.GobblinClusterManager.main(
> GobblinClusterManager.java:723)
>
> In this case the log4j-cluster.properties are being read.
>
> Can someone jot the steps to get the GAAS up?
>
>
> Here are the steps that I think should be done
>
> 1) Bring the Orchestrator up by invoking
> $GOBBLIN_HOME/bin/gobblin-service.sh start
>
> 2) Bring the Gobblin Cluster up by invoking
> $GOBBLIN_HOME/bin/gobblin-cluster-master.sh start
>
> 3) Brink the Gobblin Cluster worker's by invoking
> $GOBBLIN_HOME/bin/gobblin-cluster-worker.sh start
>
> And for all the above steps we need to configure the Helix/Kafka etc.
>
>
> Regards,
> Vicky
>
>
>