You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@falcon.apache.org by Mihai Popa <MP...@nextpathway.com> on 2015/09/28 20:15:45 UTC

Falcon cluster mirroring

Hi,


I've been trying to setup mirroring between two clusters.

One runs BigInsights 3.0.2, the other Hortonworks 2.3.

I'm running Falcon on the Hortonworks cluster (which is actually just a sandbox).

I have no trouble defining the target cluster (on the localhost), but cannot define the source cluster (on the remote BigInsights install).

The problem is with the "execute" interface; I've tried a series of ports starting with 9000 which is the default namenode, 9001, 50030.


<interface type='execute' endpoint='bivm.ibm.com:50030' version='2.2.0'/>

(I do have a DNS entry for bivm.ibm.com)


The error that I get is:


org.apache.falcon.entity.parser.ValidationException: Invalid Execute server or port: bivm.ibm.com:50030



It seems like this version of BIgInsights does not come with Yarn, and from what I see, the execute interface tries to connect to yarn's port.

Any insight (pun intended) is greatly appreciated.



thank you in advance




mihai popa

Re: Falcon cluster mirroring

Posted by Sandeep Samudrala <sa...@gmail.com>.

The 'execute' endpoint has to point to jobtracker RPC port.
Its generally 8021 on hdp cluster, can you try giving 8021?
If it doesn't work, find out the job tracker rpc port and put the same in
the cluster.xml.

Thanks,
-sandeep

On Mon, Sep 28, 2015 at 11:45 PM, Mihai Popa <MP...@nextpathway.com> wrote:

> Hi,
>
>
> I've been trying to setup mirroring between two clusters.
>
> One runs BigInsights 3.0.2, the other Hortonworks 2.3.
>
> I'm running Falcon on the Hortonworks cluster (which is actually just a
> sandbox).
>
> I have no trouble defining the target cluster (on the localhost), but
> cannot define the source cluster (on the remote BigInsights install).
>
> The problem is with the "execute" interface; I've tried a series of ports
> starting with 9000 which is the default namenode, 9001, 50030.
>
>
> <interface type='execute' endpoint='bivm.ibm.com:50030' version='2.2.0'/>
>
> (I do have a DNS entry for bivm.ibm.com)
>
>
> The error that I get is:
>
>
> org.apache.falcon.entity.parser.ValidationException: Invalid Execute
> server or port: bivm.ibm.com:50030
>
>
>
> It seems like this version of BIgInsights does not come with Yarn, and
> from what I see, the execute interface tries to connect to yarn's port.
>
> Any insight (pun intended) is greatly appreciated.
>
>
>
> thank you in advance
>
>
>
>
> mihai popa
>

Re: Falcon cluster mirroring

Posted by Peeyush Bishnoi <bp...@yahoo.co.in>.

Hi Mihai,
Looking to the error which you have got, it looks like that Falcon could not able to validate the execute services of BigInsights.
To mirror the data from BigInsights (source) to HDP sandbox(target), you can try to use Falcon HDFS replication recipe. To use Falcon HDFS replication recipe, run it on target i.e HDP sandbox.1. Define cluster entity with services from HDP sandbox (target) and submit on target.2. In hdfs-replication.properties of recipe , specify source HDFS address with input data path from BigInsights and target HDFS address with output data path. Also in properties, specify that you want to run recipe on target for attribute "falcon.recipe.cluster.name".
For more information on usage of HDFS replication recipe, please refer following doc:http://falcon.apache.org/Recipes.html

I hope this will help you to mirror your data.

Please let us know, if you have any questions.
Thanks,---Peeyush

On Tuesday, 29 September 2015 7:37 AM, Mihai Popa <MP...@nextpathway.com> wrote:

Hi,

I've been trying to setup mirroring between two clusters.

One runs BigInsights 3.0.2, the other Hortonworks 2.3.

I'm running Falcon on the Hortonworks cluster (which is actually just a sandbox).

I have no trouble defining the target cluster (on the localhost), but cannot define the source cluster (on the remote BigInsights install).

The problem is with the "execute" interface; I've tried a series of ports starting with 9000 which is the default namenode, 9001, 50030.

(I do have a DNS entry for bivm.ibm.com)

The error that I get is:

org.apache.falcon.entity.parser.ValidationException: Invalid Execute server or port: bivm.ibm.com:50030

It seems like this version of BIgInsights does not come with Yarn, and from what I see, the execute interface tries to connect to yarn's port.

Any insight (pun intended) is greatly appreciated.

thank you in advance

mihai popa