how to connect to database having the DB properties file outside the resources folder or applcation

[Venky <ve...@gmail.com>]

Hi,

Req1: I have around 20 operators. In every operator I want to connect to
database to mark my processing as failed when there is an Io Exception or
any cluster level exception. I wanted to use plain java class here, please
let me know how we can implement it. If not suggest the better approach

Req2: I don't want the DB properties file within the application package
and wanted to maintain the DB properties outside in a properties file and
able to use them while connecting to database to fulfill my Req1. Please
let me know how we can maintain the DB properties file outside.

Req3: How can I capture cluster level exceptions like RPC Failure and etc?

Regards,
Venkat.

Re: how to connect to database having the DB properties file outside the resources folder or applcation

[Aniruddha Thombare <an...@datatorrent.com>]

About req 3:
AFAIK CDH does provide REST access to cluster health parameters among many
other things.
Not sure about ambari though

Thanks,

A

_____________________________________
Sent with difficulty, I mean handheld ;)

On 17 Nov 2016 8:25 pm, "Munagala Ramanath" <ra...@datatorrent.com> wrote:

> For Req1, a better approach might be to have a single "error reporting"
> operator that connects to the DB;
> all the other operators can send custom error tuples to it with
> appropriate details of the type of error.
> That way, if there are issues that are DB-related, you have 1 place to
> debug them instead of 20.
>
> With that approach, your DB operator will need 20 input ports and a new
> one added each time you add
> an additional operator to the DAG; an alternative is to use the
> OperatorRequest mechanism to
> communicate with the DB operator instead of the normal ports-and-tuples
> method. This is illustrated
> in the example at https://github.com/DataTorrent/examples/tree/
> master/tutorials/throttle
> where it is used to modulate the output data rate of the input operator
> when the downstream operators
> are unable to keep up.
>
> For Req2, you can put the DB properties in an HDFS location and read it in
> the setup() callback of the
> operator. The path to that location can be in your regular properties file
> within the app package.
> A similar, though more elaborate method is to run a service on a known
> port which can be queried
> for those properties; the operators will then connect to that service in
> setup() to retrieve the info.
>
> For Req3, cluster-level issues are generally not made available to
> user-level callbacks; you'll need
> an external process(es) to monitor log files, invoke REST APIs etc.
>
> Ram
>
>
>
> On Thu, Nov 17, 2016 at 5:50 AM, Venky <ve...@gmail.com> wrote:
>
>> Hi,
>>
>> Req1: I have around 20 operators. In every operator I want to connect to
>> database to mark my processing as failed when there is an Io Exception or
>> any cluster level exception. I wanted to use plain java class here, please
>> let me know how we can implement it. If not suggest the better approach
>>
>> Req2: I don't want the DB properties file within the application package
>> and wanted to maintain the DB properties outside in a properties file and
>> able to use them while connecting to database to fulfill my Req1. Please
>> let me know how we can maintain the DB properties file outside.
>>
>> Req3: How can I capture cluster level exceptions like RPC Failure and etc?
>>
>> Regards,
>> Venkat.
>>
>
>

Re: how to connect to database having the DB properties file outside the resources folder or applcation

[Munagala Ramanath <ra...@datatorrent.com>]

For Req1, a better approach might be to have a single "error reporting"
operator that connects to the DB;
all the other operators can send custom error tuples to it with appropriate
details of the type of error.
That way, if there are issues that are DB-related, you have 1 place to
debug them instead of 20.

With that approach, your DB operator will need 20 input ports and a new one
added each time you add
an additional operator to the DAG; an alternative is to use the
OperatorRequest mechanism to
communicate with the DB operator instead of the normal ports-and-tuples
method. This is illustrated
in the example at
https://github.com/DataTorrent/examples/tree/master/tutorials/throttle
where it is used to modulate the output data rate of the input operator
when the downstream operators
are unable to keep up.

For Req2, you can put the DB properties in an HDFS location and read it in
the setup() callback of the
operator. The path to that location can be in your regular properties file
within the app package.
A similar, though more elaborate method is to run a service on a known port
which can be queried
for those properties; the operators will then connect to that service in
setup() to retrieve the info.

For Req3, cluster-level issues are generally not made available to
user-level callbacks; you'll need
an external process(es) to monitor log files, invoke REST APIs etc.

Ram



On Thu, Nov 17, 2016 at 5:50 AM, Venky <ve...@gmail.com> wrote:

> Hi,
>
> Req1: I have around 20 operators. In every operator I want to connect to
> database to mark my processing as failed when there is an Io Exception or
> any cluster level exception. I wanted to use plain java class here, please
> let me know how we can implement it. If not suggest the better approach
>
> Req2: I don't want the DB properties file within the application package
> and wanted to maintain the DB properties outside in a properties file and
> able to use them while connecting to database to fulfill my Req1. Please
> let me know how we can maintain the DB properties file outside.
>
> Req3: How can I capture cluster level exceptions like RPC Failure and etc?
>
> Regards,
> Venkat.
>