You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "Kottapalli, Venkatesh" <VK...@DIRECTV.com> on 2015/12/15 03:25:23 UTC
configure filesplitter to read batch data from different folders
Hi,
I am trying to configure filesplitter to read files from different directories based on parameters that I wish to provide as input to the application?
* One way of doing it is to read data from a properties file on server into application.java and set those values into filesplitter config.
* I am wondering if there is a way to pass arguments to the submitted application so that I can set my filesplitter configuration dynamically from the below command rather than from properties?
POST /ws/v2/appPackages/{owner}/{packageName}/{packageVersion}/applications/{appName}/launch[?config={configName}&originalAppId={originalAppId}&queue={queueName}]
Regards,
Venkatesh.
RE: configure filesplitter to read batch data from different folders
Posted by "Kottapalli, Venkatesh" <VK...@DIRECTV.com>.
Thanks David.
-----Original Message-----
From: David Yan [mailto:david@datatorrent.com]
Sent: Monday, December 14, 2015 6:58 PM
To: dev@apex.incubator.apache.org
Subject: Re: configure filesplitter to read batch data from different folders
Hi Venkatesh,
The POST command you listed actually accepts the following as payload:
{
"{propertyName}" : "{propertyValue}",
...
}
The propertyName can be
"dt.application.app_name.operator.fileSplitter.prop.scanner.files" and the propertyValue can be "/user/data/date=12-14-2015/ver=6/".
David
On Mon, Dec 14, 2015 at 6:25 PM, Kottapalli, Venkatesh < VKottapalli@directv.com> wrote:
> Hi,
>
> I am trying to configure filesplitter to read files
> from different directories based on parameters that I wish to provide
> as input to the application?
>
>
> * One way of doing it is to read data from a properties file on
> server into application.java and set those values into filesplitter config.
>
>
> * I am wondering if there is a way to pass arguments to the
> submitted application so that I can set my filesplitter configuration
> dynamically from the below command rather than from properties?
>
>
>
> POST
> /ws/v2/appPackages/{owner}/{packageName}/{packageVersion}/applications
> /{appName}/launch[?config={configName}&originalAppId={originalAppId}&q
> ueue={queueName}]
> Regards,
> Venkatesh.
>
Re: configure filesplitter to read batch data from different folders
Posted by David Yan <da...@datatorrent.com>.
Hi Venkatesh,
The POST command you listed actually accepts the following as payload:
{
"{propertyName}" : "{propertyValue}",
...
}
The propertyName can be
"dt.application.app_name.operator.fileSplitter.prop.scanner.files" and the
propertyValue can be "/user/data/date=12-14-2015/ver=6/".
David
On Mon, Dec 14, 2015 at 6:25 PM, Kottapalli, Venkatesh <
VKottapalli@directv.com> wrote:
> Hi,
>
> I am trying to configure filesplitter to read files from
> different directories based on parameters that I wish to provide as input
> to the application?
>
>
> * One way of doing it is to read data from a properties file on
> server into application.java and set those values into filesplitter config.
>
>
> * I am wondering if there is a way to pass arguments to the
> submitted application so that I can set my filesplitter configuration
> dynamically from the below command rather than from properties?
>
>
>
> POST
> /ws/v2/appPackages/{owner}/{packageName}/{packageVersion}/applications/{appName}/launch[?config={configName}&originalAppId={originalAppId}&queue={queueName}]
> Regards,
> Venkatesh.
>
RE: configure filesplitter to read batch data from different folders
Posted by "Kottapalli, Venkatesh" <VK...@DIRECTV.com>.
Hi Chandini,
It is not from a running application. I am aware of that way but it doesn’t suit my use case.
Here is the example.
<property>
<name>dt.application.app_name.operator.fileSplitter.prop.scanner.files </name>
<value> /user/data/date=12-14-2015/ver=6/</value>
</property>
Now when I am submitting my application, I need to change the date and version in property so that it picks files from a different directory on a different day. I would like to do it while submitting my application package using the dt gateway so that I can set it as below.
Assuming arg1 is generated by scripts and passed as argument with the POST command.
then,
conf.set("dt.application.app_name.operator.fileSplitter.prop.scanner.files ", arg1);
Regards,
Venkatesh.
-----Original Message-----
From: Chandni Singh [mailto:chandni@datatorrent.com]
Sent: Monday, December 14, 2015 6:32 PM
To: dev@apex.incubator.apache.org
Subject: Re: configure filesplitter to read batch data from different folders
There is a way to dynamically change properties of an operator in a running application. Is that what you are looking for?
On Mon, Dec 14, 2015 at 6:25 PM, Kottapalli, Venkatesh < VKottapalli@directv.com> wrote:
> Hi,
>
> I am trying to configure filesplitter to read files
> from different directories based on parameters that I wish to provide
> as input to the application?
>
>
> * One way of doing it is to read data from a properties file on
> server into application.java and set those values into filesplitter config.
>
>
> * I am wondering if there is a way to pass arguments to the
> submitted application so that I can set my filesplitter configuration
> dynamically from the below command rather than from properties?
>
>
>
> POST
> /ws/v2/appPackages/{owner}/{packageName}/{packageVersion}/applications
> /{appName}/launch[?config={configName}&originalAppId={originalAppId}&q
> ueue={queueName}]
> Regards,
> Venkatesh.
>
Re: configure filesplitter to read batch data from different folders
Posted by Chandni Singh <ch...@datatorrent.com>.
There is a way to dynamically change properties of an operator in a running
application. Is that what you are looking for?
On Mon, Dec 14, 2015 at 6:25 PM, Kottapalli, Venkatesh <
VKottapalli@directv.com> wrote:
> Hi,
>
> I am trying to configure filesplitter to read files from
> different directories based on parameters that I wish to provide as input
> to the application?
>
>
> * One way of doing it is to read data from a properties file on
> server into application.java and set those values into filesplitter config.
>
>
> * I am wondering if there is a way to pass arguments to the
> submitted application so that I can set my filesplitter configuration
> dynamically from the below command rather than from properties?
>
>
>
> POST
> /ws/v2/appPackages/{owner}/{packageName}/{packageVersion}/applications/{appName}/launch[?config={configName}&originalAppId={originalAppId}&queue={queueName}]
> Regards,
> Venkatesh.
>