You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Akshay Gore <go...@gmail.com> on 2016/01/12 10:28:21 UTC
Concrete classes in Malhar
Hi,
I am developing the sample Apex application to explore Malhar operators. In
my case, I need to write the processed data to HDFS. I did not find
concrete class for file writing, instead I got AbstractFileOutput operator
class which will have to implement it. Can I have references/ examples to
how to implement the abstract classes in Malhar ? If not, it would be nice
to have such classes in Malhar.
--
Thanks,
Akshay
Re: Concrete classes in Malhar
Posted by Yogi Devendra <de...@datatorrent.com>.
Thanks for your suggestion.
Can you please create a JIRA for this on
https://issues.apache.org/jira/browse/APEXMALHAR
This will help the community to take that up sometime in future.
~ Yogi
On Tue, Jan 12, 2016 at 5:00 PM, Akshay Gore <go...@gmail.com> wrote:
> Thanks Yogi. I am able to perform file write operation on HDFS.
>
> One suggestion here, Malhar library should have some concrete classes like
> hdfs file input/output operations so that it would be easy for a new Apex
> user to quickly start with.
>
> -Akshay
>
> On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <de...@datatorrent.com>
> wrote:
>
>> Here is the Javadoc:
>>
>>
>> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>>
>> Here is the user guide:
>>
>> http://docs.datatorrent.com/operators/file_output/
>>
>> You will need to provide implementation for couple of methods:
>>
>> 1. protected abstract String getFileName(INPUT tuple)
>> You could simply return "output.txt"
>>
>> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>>
>> You need to provide some way to serialize input tuple.
>> It could be as simple as return tuple.toString().getBytes()
>>
>> It would be great if you use some IDE like eclipse or intellij; it should
>> give you suggestions, auto-complete template for methods to be overridden.
>>
>> ~ Yogi
>>
>> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am developing the sample Apex application to explore Malhar operators.
>>> In my case, I need to write the processed data to HDFS. I did not find
>>> concrete class for file writing, instead I got AbstractFileOutput operator
>>> class which will have to implement it. Can I have references/ examples to
>>> how to implement the abstract classes in Malhar ? If not, it would be nice
>>> to have such classes in Malhar.
>>>
>>> --
>>> Thanks,
>>> Akshay
>>>
>>
>>
>
>
> --
> Thanks,
> Akshay
>
Re: Concrete classes in Malhar
Posted by Yogi Devendra <yo...@apache.org>.
Hi,
I would like to take up this JIRA.
I have Username: yogidevendra on https://issues.apache.org/jira.
Can one of the JIRA admins please add me to the developers, users group on
JIRA. Currently, I am only member of jira-users group.
For this JIRA; I am not able to 'Assign it to me'.
~ Yogi
On 13 January 2016 at 11:12, Akshay Gore <go...@gmail.com> wrote:
> Please the related JIRA at APEXMALHAR-1971
> <https://issues.apache.org/jira/browse/APEXMALHAR-1971>
>
>
> Thanks,
> Akshay
>
> On Wed, Jan 13, 2016 at 10:40 AM, Akshay Gore <go...@gmail.com>
> wrote:
>
>> Thanks all for your feedback.
>>
>> I will create a JIRA and update here.
>>
>> --Akshay
>> On 13-Jan-2016 10:27, "Yogi Devendra" <de...@datatorrent.com> wrote:
>>
>>> Akshay,
>>>
>>> Can you please create a JIRA for this?
>>> Also add Permalink
>>> <https://mail-archives.apache.org/mod_mbox/incubator-apex-users/201601.mbox/%3CCAJxXgwMG721Gu7MN5QShPYVYadtTgQesm%2BeHfR%2BD%2B0Yt%2ByXrEw%40mail.gmail.com%3E> for
>>> the email thread in the JIRA description.
>>> and paste the JIRA link on this thread.
>>>
>>> ~ Yogi
>>>
>>> On Wed, Jan 13, 2016 at 3:50 AM, Pramod Immaneni <pramod@datatorrent.com
>>> > wrote:
>>>
>>>> Agreed. We have created trivial implementations for other operators
>>>> like kafka string, line by line file input so on. This may be a candidate
>>>> for that.
>>>>
>>>> On Tue, Jan 12, 2016 at 3:30 AM, Akshay Gore <go...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Yogi. I am able to perform file write operation on HDFS.
>>>>>
>>>>> One suggestion here, Malhar library should have some concrete classes
>>>>> like hdfs file input/output operations so that it would be easy for a new
>>>>> Apex user to quickly start with.
>>>>>
>>>>> -Akshay
>>>>>
>>>>> On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <
>>>>> devendra@datatorrent.com> wrote:
>>>>>
>>>>>> Here is the Javadoc:
>>>>>>
>>>>>>
>>>>>> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>>>>>>
>>>>>> Here is the user guide:
>>>>>>
>>>>>> http://docs.datatorrent.com/operators/file_output/
>>>>>>
>>>>>> You will need to provide implementation for couple of methods:
>>>>>>
>>>>>> 1. protected abstract String getFileName(INPUT tuple)
>>>>>> You could simply return "output.txt"
>>>>>>
>>>>>> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>>>>>>
>>>>>> You need to provide some way to serialize input tuple.
>>>>>> It could be as simple as return tuple.toString().getBytes()
>>>>>>
>>>>>> It would be great if you use some IDE like eclipse or intellij; it
>>>>>> should give you suggestions, auto-complete template for methods to be
>>>>>> overridden.
>>>>>>
>>>>>> ~ Yogi
>>>>>>
>>>>>> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am developing the sample Apex application to explore Malhar
>>>>>>> operators. In my case, I need to write the processed data to HDFS. I did
>>>>>>> not find concrete class for file writing, instead I got AbstractFileOutput
>>>>>>> operator class which will have to implement it. Can I have references/
>>>>>>> examples to how to implement the abstract classes in Malhar ? If not, it
>>>>>>> would be nice to have such classes in Malhar.
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>> Akshay
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Akshay
>>>>>
>>>>
>>>>
>>>
>
>
> --
> Thanks,
> Akshay
>
Re: Concrete classes in Malhar
Posted by Akshay Gore <go...@gmail.com>.
Please the related JIRA at APEXMALHAR-1971
<https://issues.apache.org/jira/browse/APEXMALHAR-1971>
Thanks,
Akshay
On Wed, Jan 13, 2016 at 10:40 AM, Akshay Gore <go...@gmail.com> wrote:
> Thanks all for your feedback.
>
> I will create a JIRA and update here.
>
> --Akshay
> On 13-Jan-2016 10:27, "Yogi Devendra" <de...@datatorrent.com> wrote:
>
>> Akshay,
>>
>> Can you please create a JIRA for this?
>> Also add Permalink
>> <https://mail-archives.apache.org/mod_mbox/incubator-apex-users/201601.mbox/%3CCAJxXgwMG721Gu7MN5QShPYVYadtTgQesm%2BeHfR%2BD%2B0Yt%2ByXrEw%40mail.gmail.com%3E> for
>> the email thread in the JIRA description.
>> and paste the JIRA link on this thread.
>>
>> ~ Yogi
>>
>> On Wed, Jan 13, 2016 at 3:50 AM, Pramod Immaneni <pr...@datatorrent.com>
>> wrote:
>>
>>> Agreed. We have created trivial implementations for other operators like
>>> kafka string, line by line file input so on. This may be a candidate for
>>> that.
>>>
>>> On Tue, Jan 12, 2016 at 3:30 AM, Akshay Gore <go...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Yogi. I am able to perform file write operation on HDFS.
>>>>
>>>> One suggestion here, Malhar library should have some concrete classes
>>>> like hdfs file input/output operations so that it would be easy for a new
>>>> Apex user to quickly start with.
>>>>
>>>> -Akshay
>>>>
>>>> On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <
>>>> devendra@datatorrent.com> wrote:
>>>>
>>>>> Here is the Javadoc:
>>>>>
>>>>>
>>>>> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>>>>>
>>>>> Here is the user guide:
>>>>>
>>>>> http://docs.datatorrent.com/operators/file_output/
>>>>>
>>>>> You will need to provide implementation for couple of methods:
>>>>>
>>>>> 1. protected abstract String getFileName(INPUT tuple)
>>>>> You could simply return "output.txt"
>>>>>
>>>>> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>>>>>
>>>>> You need to provide some way to serialize input tuple.
>>>>> It could be as simple as return tuple.toString().getBytes()
>>>>>
>>>>> It would be great if you use some IDE like eclipse or intellij; it
>>>>> should give you suggestions, auto-complete template for methods to be
>>>>> overridden.
>>>>>
>>>>> ~ Yogi
>>>>>
>>>>> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am developing the sample Apex application to explore Malhar
>>>>>> operators. In my case, I need to write the processed data to HDFS. I did
>>>>>> not find concrete class for file writing, instead I got AbstractFileOutput
>>>>>> operator class which will have to implement it. Can I have references/
>>>>>> examples to how to implement the abstract classes in Malhar ? If not, it
>>>>>> would be nice to have such classes in Malhar.
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>> Akshay
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Akshay
>>>>
>>>
>>>
>>
--
Thanks,
Akshay
Re: Concrete classes in Malhar
Posted by Akshay Gore <go...@gmail.com>.
Thanks all for your feedback.
I will create a JIRA and update here.
--Akshay
On 13-Jan-2016 10:27, "Yogi Devendra" <de...@datatorrent.com> wrote:
> Akshay,
>
> Can you please create a JIRA for this?
> Also add Permalink
> <https://mail-archives.apache.org/mod_mbox/incubator-apex-users/201601.mbox/%3CCAJxXgwMG721Gu7MN5QShPYVYadtTgQesm%2BeHfR%2BD%2B0Yt%2ByXrEw%40mail.gmail.com%3E> for
> the email thread in the JIRA description.
> and paste the JIRA link on this thread.
>
> ~ Yogi
>
> On Wed, Jan 13, 2016 at 3:50 AM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
>> Agreed. We have created trivial implementations for other operators like
>> kafka string, line by line file input so on. This may be a candidate for
>> that.
>>
>> On Tue, Jan 12, 2016 at 3:30 AM, Akshay Gore <go...@gmail.com>
>> wrote:
>>
>>> Thanks Yogi. I am able to perform file write operation on HDFS.
>>>
>>> One suggestion here, Malhar library should have some concrete classes
>>> like hdfs file input/output operations so that it would be easy for a new
>>> Apex user to quickly start with.
>>>
>>> -Akshay
>>>
>>> On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <devendra@datatorrent.com
>>> > wrote:
>>>
>>>> Here is the Javadoc:
>>>>
>>>>
>>>> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>>>>
>>>> Here is the user guide:
>>>>
>>>> http://docs.datatorrent.com/operators/file_output/
>>>>
>>>> You will need to provide implementation for couple of methods:
>>>>
>>>> 1. protected abstract String getFileName(INPUT tuple)
>>>> You could simply return "output.txt"
>>>>
>>>> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>>>>
>>>> You need to provide some way to serialize input tuple.
>>>> It could be as simple as return tuple.toString().getBytes()
>>>>
>>>> It would be great if you use some IDE like eclipse or intellij; it
>>>> should give you suggestions, auto-complete template for methods to be
>>>> overridden.
>>>>
>>>> ~ Yogi
>>>>
>>>> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am developing the sample Apex application to explore Malhar
>>>>> operators. In my case, I need to write the processed data to HDFS. I did
>>>>> not find concrete class for file writing, instead I got AbstractFileOutput
>>>>> operator class which will have to implement it. Can I have references/
>>>>> examples to how to implement the abstract classes in Malhar ? If not, it
>>>>> would be nice to have such classes in Malhar.
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Akshay
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Akshay
>>>
>>
>>
>
Re: Concrete classes in Malhar
Posted by Yogi Devendra <de...@datatorrent.com>.
Akshay,
Can you please create a JIRA for this?
Also add Permalink
<https://mail-archives.apache.org/mod_mbox/incubator-apex-users/201601.mbox/%3CCAJxXgwMG721Gu7MN5QShPYVYadtTgQesm%2BeHfR%2BD%2B0Yt%2ByXrEw%40mail.gmail.com%3E>
for
the email thread in the JIRA description.
and paste the JIRA link on this thread.
~ Yogi
On Wed, Jan 13, 2016 at 3:50 AM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> Agreed. We have created trivial implementations for other operators like
> kafka string, line by line file input so on. This may be a candidate for
> that.
>
> On Tue, Jan 12, 2016 at 3:30 AM, Akshay Gore <go...@gmail.com> wrote:
>
>> Thanks Yogi. I am able to perform file write operation on HDFS.
>>
>> One suggestion here, Malhar library should have some concrete classes
>> like hdfs file input/output operations so that it would be easy for a new
>> Apex user to quickly start with.
>>
>> -Akshay
>>
>> On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <de...@datatorrent.com>
>> wrote:
>>
>>> Here is the Javadoc:
>>>
>>>
>>> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>>>
>>> Here is the user guide:
>>>
>>> http://docs.datatorrent.com/operators/file_output/
>>>
>>> You will need to provide implementation for couple of methods:
>>>
>>> 1. protected abstract String getFileName(INPUT tuple)
>>> You could simply return "output.txt"
>>>
>>> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>>>
>>> You need to provide some way to serialize input tuple.
>>> It could be as simple as return tuple.toString().getBytes()
>>>
>>> It would be great if you use some IDE like eclipse or intellij; it
>>> should give you suggestions, auto-complete template for methods to be
>>> overridden.
>>>
>>> ~ Yogi
>>>
>>> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am developing the sample Apex application to explore Malhar
>>>> operators. In my case, I need to write the processed data to HDFS. I did
>>>> not find concrete class for file writing, instead I got AbstractFileOutput
>>>> operator class which will have to implement it. Can I have references/
>>>> examples to how to implement the abstract classes in Malhar ? If not, it
>>>> would be nice to have such classes in Malhar.
>>>>
>>>> --
>>>> Thanks,
>>>> Akshay
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks,
>> Akshay
>>
>
>
Re: Concrete classes in Malhar
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Agreed. We have created trivial implementations for other operators like
kafka string, line by line file input so on. This may be a candidate for
that.
On Tue, Jan 12, 2016 at 3:30 AM, Akshay Gore <go...@gmail.com> wrote:
> Thanks Yogi. I am able to perform file write operation on HDFS.
>
> One suggestion here, Malhar library should have some concrete classes like
> hdfs file input/output operations so that it would be easy for a new Apex
> user to quickly start with.
>
> -Akshay
>
> On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <de...@datatorrent.com>
> wrote:
>
>> Here is the Javadoc:
>>
>>
>> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>>
>> Here is the user guide:
>>
>> http://docs.datatorrent.com/operators/file_output/
>>
>> You will need to provide implementation for couple of methods:
>>
>> 1. protected abstract String getFileName(INPUT tuple)
>> You could simply return "output.txt"
>>
>> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>>
>> You need to provide some way to serialize input tuple.
>> It could be as simple as return tuple.toString().getBytes()
>>
>> It would be great if you use some IDE like eclipse or intellij; it should
>> give you suggestions, auto-complete template for methods to be overridden.
>>
>> ~ Yogi
>>
>> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am developing the sample Apex application to explore Malhar operators.
>>> In my case, I need to write the processed data to HDFS. I did not find
>>> concrete class for file writing, instead I got AbstractFileOutput operator
>>> class which will have to implement it. Can I have references/ examples to
>>> how to implement the abstract classes in Malhar ? If not, it would be nice
>>> to have such classes in Malhar.
>>>
>>> --
>>> Thanks,
>>> Akshay
>>>
>>
>>
>
>
> --
> Thanks,
> Akshay
>
Re: Concrete classes in Malhar
Posted by Munagala Ramanath <ra...@datatorrent.com>.
My note was merely to point to additional existing samples, not in
opposition to creating
new concrete classes.
Ram
On Tue, Jan 12, 2016 at 7:51 AM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:
> I agree with akshay. We should have some basic ready to use operators in
> Malhar.
> Correct me if I'm wrong but demo is not usually added as dependency;
> library & contrib is.
>
> ~ Chinmay
> On 12 Jan 2016 20:37, "Munagala Ramanath" <ra...@datatorrent.com> wrote:
>
>> There are some concrete classes in the demos subdirectory of Malhar, see
>> for example:
>>
>>
>>
>> *demos/frauddetect/src/main/java/com/datatorrent/demos/frauddetect/operator/HdfsStringOutputOperator.java*
>>
>> *demos/wordcount/src/main/java/com/datatorrent/demos/wordcount/WordCountWriter.java*
>>
>> Ram
>>
>> On Tue, Jan 12, 2016 at 3:30 AM, Akshay Gore <go...@gmail.com>
>> wrote:
>>
>>> Thanks Yogi. I am able to perform file write operation on HDFS.
>>>
>>> One suggestion here, Malhar library should have some concrete classes
>>> like hdfs file input/output operations so that it would be easy for a new
>>> Apex user to quickly start with.
>>>
>>> -Akshay
>>>
>>> On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <devendra@datatorrent.com
>>> > wrote:
>>>
>>>> Here is the Javadoc:
>>>>
>>>>
>>>> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>>>>
>>>> Here is the user guide:
>>>>
>>>> http://docs.datatorrent.com/operators/file_output/
>>>>
>>>> You will need to provide implementation for couple of methods:
>>>>
>>>> 1. protected abstract String getFileName(INPUT tuple)
>>>> You could simply return "output.txt"
>>>>
>>>> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>>>>
>>>> You need to provide some way to serialize input tuple.
>>>> It could be as simple as return tuple.toString().getBytes()
>>>>
>>>> It would be great if you use some IDE like eclipse or intellij; it
>>>> should give you suggestions, auto-complete template for methods to be
>>>> overridden.
>>>>
>>>> ~ Yogi
>>>>
>>>> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am developing the sample Apex application to explore Malhar
>>>>> operators. In my case, I need to write the processed data to HDFS. I did
>>>>> not find concrete class for file writing, instead I got AbstractFileOutput
>>>>> operator class which will have to implement it. Can I have references/
>>>>> examples to how to implement the abstract classes in Malhar ? If not, it
>>>>> would be nice to have such classes in Malhar.
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Akshay
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Akshay
>>>
>>
>>
Re: Concrete classes in Malhar
Posted by Amol Kekre <am...@datatorrent.com>.
I too agree with Akshay.
Thks
Amol
On Tue, Jan 12, 2016 at 7:51 AM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:
> I agree with akshay. We should have some basic ready to use operators in
> Malhar.
> Correct me if I'm wrong but demo is not usually added as dependency;
> library & contrib is.
>
> ~ Chinmay
> On 12 Jan 2016 20:37, "Munagala Ramanath" <ra...@datatorrent.com> wrote:
>
>> There are some concrete classes in the demos subdirectory of Malhar, see
>> for example:
>>
>>
>>
>> *demos/frauddetect/src/main/java/com/datatorrent/demos/frauddetect/operator/HdfsStringOutputOperator.java*
>>
>> *demos/wordcount/src/main/java/com/datatorrent/demos/wordcount/WordCountWriter.java*
>>
>> Ram
>>
>> On Tue, Jan 12, 2016 at 3:30 AM, Akshay Gore <go...@gmail.com>
>> wrote:
>>
>>> Thanks Yogi. I am able to perform file write operation on HDFS.
>>>
>>> One suggestion here, Malhar library should have some concrete classes
>>> like hdfs file input/output operations so that it would be easy for a new
>>> Apex user to quickly start with.
>>>
>>> -Akshay
>>>
>>> On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <devendra@datatorrent.com
>>> > wrote:
>>>
>>>> Here is the Javadoc:
>>>>
>>>>
>>>> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>>>>
>>>> Here is the user guide:
>>>>
>>>> http://docs.datatorrent.com/operators/file_output/
>>>>
>>>> You will need to provide implementation for couple of methods:
>>>>
>>>> 1. protected abstract String getFileName(INPUT tuple)
>>>> You could simply return "output.txt"
>>>>
>>>> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>>>>
>>>> You need to provide some way to serialize input tuple.
>>>> It could be as simple as return tuple.toString().getBytes()
>>>>
>>>> It would be great if you use some IDE like eclipse or intellij; it
>>>> should give you suggestions, auto-complete template for methods to be
>>>> overridden.
>>>>
>>>> ~ Yogi
>>>>
>>>> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am developing the sample Apex application to explore Malhar
>>>>> operators. In my case, I need to write the processed data to HDFS. I did
>>>>> not find concrete class for file writing, instead I got AbstractFileOutput
>>>>> operator class which will have to implement it. Can I have references/
>>>>> examples to how to implement the abstract classes in Malhar ? If not, it
>>>>> would be nice to have such classes in Malhar.
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Akshay
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Akshay
>>>
>>
>>
Re: Concrete classes in Malhar
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
I agree with akshay. We should have some basic ready to use operators in
Malhar.
Correct me if I'm wrong but demo is not usually added as dependency;
library & contrib is.
~ Chinmay
On 12 Jan 2016 20:37, "Munagala Ramanath" <ra...@datatorrent.com> wrote:
> There are some concrete classes in the demos subdirectory of Malhar, see
> for example:
>
>
>
> *demos/frauddetect/src/main/java/com/datatorrent/demos/frauddetect/operator/HdfsStringOutputOperator.java*
>
> *demos/wordcount/src/main/java/com/datatorrent/demos/wordcount/WordCountWriter.java*
>
> Ram
>
> On Tue, Jan 12, 2016 at 3:30 AM, Akshay Gore <go...@gmail.com> wrote:
>
>> Thanks Yogi. I am able to perform file write operation on HDFS.
>>
>> One suggestion here, Malhar library should have some concrete classes
>> like hdfs file input/output operations so that it would be easy for a new
>> Apex user to quickly start with.
>>
>> -Akshay
>>
>> On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <de...@datatorrent.com>
>> wrote:
>>
>>> Here is the Javadoc:
>>>
>>>
>>> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>>>
>>> Here is the user guide:
>>>
>>> http://docs.datatorrent.com/operators/file_output/
>>>
>>> You will need to provide implementation for couple of methods:
>>>
>>> 1. protected abstract String getFileName(INPUT tuple)
>>> You could simply return "output.txt"
>>>
>>> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>>>
>>> You need to provide some way to serialize input tuple.
>>> It could be as simple as return tuple.toString().getBytes()
>>>
>>> It would be great if you use some IDE like eclipse or intellij; it
>>> should give you suggestions, auto-complete template for methods to be
>>> overridden.
>>>
>>> ~ Yogi
>>>
>>> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am developing the sample Apex application to explore Malhar
>>>> operators. In my case, I need to write the processed data to HDFS. I did
>>>> not find concrete class for file writing, instead I got AbstractFileOutput
>>>> operator class which will have to implement it. Can I have references/
>>>> examples to how to implement the abstract classes in Malhar ? If not, it
>>>> would be nice to have such classes in Malhar.
>>>>
>>>> --
>>>> Thanks,
>>>> Akshay
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks,
>> Akshay
>>
>
>
Re: Concrete classes in Malhar
Posted by Munagala Ramanath <ra...@datatorrent.com>.
There are some concrete classes in the demos subdirectory of Malhar, see
for example:
*demos/frauddetect/src/main/java/com/datatorrent/demos/frauddetect/operator/HdfsStringOutputOperator.java*
*demos/wordcount/src/main/java/com/datatorrent/demos/wordcount/WordCountWriter.java*
Ram
On Tue, Jan 12, 2016 at 3:30 AM, Akshay Gore <go...@gmail.com> wrote:
> Thanks Yogi. I am able to perform file write operation on HDFS.
>
> One suggestion here, Malhar library should have some concrete classes like
> hdfs file input/output operations so that it would be easy for a new Apex
> user to quickly start with.
>
> -Akshay
>
> On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <de...@datatorrent.com>
> wrote:
>
>> Here is the Javadoc:
>>
>>
>> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>>
>> Here is the user guide:
>>
>> http://docs.datatorrent.com/operators/file_output/
>>
>> You will need to provide implementation for couple of methods:
>>
>> 1. protected abstract String getFileName(INPUT tuple)
>> You could simply return "output.txt"
>>
>> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>>
>> You need to provide some way to serialize input tuple.
>> It could be as simple as return tuple.toString().getBytes()
>>
>> It would be great if you use some IDE like eclipse or intellij; it should
>> give you suggestions, auto-complete template for methods to be overridden.
>>
>> ~ Yogi
>>
>> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am developing the sample Apex application to explore Malhar operators.
>>> In my case, I need to write the processed data to HDFS. I did not find
>>> concrete class for file writing, instead I got AbstractFileOutput operator
>>> class which will have to implement it. Can I have references/ examples to
>>> how to implement the abstract classes in Malhar ? If not, it would be nice
>>> to have such classes in Malhar.
>>>
>>> --
>>> Thanks,
>>> Akshay
>>>
>>
>>
>
>
> --
> Thanks,
> Akshay
>
Re: Concrete classes in Malhar
Posted by Akshay Gore <go...@gmail.com>.
Thanks Yogi. I am able to perform file write operation on HDFS.
One suggestion here, Malhar library should have some concrete classes like
hdfs file input/output operations so that it would be easy for a new Apex
user to quickly start with.
-Akshay
On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra <de...@datatorrent.com>
wrote:
> Here is the Javadoc:
>
>
> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
>
> Here is the user guide:
>
> http://docs.datatorrent.com/operators/file_output/
>
> You will need to provide implementation for couple of methods:
>
> 1. protected abstract String getFileName(INPUT tuple)
> You could simply return "output.txt"
>
> 2. protected abstract byte[] getBytesForTuple(INPUT tuple)
>
> You need to provide some way to serialize input tuple.
> It could be as simple as return tuple.toString().getBytes()
>
> It would be great if you use some IDE like eclipse or intellij; it should
> give you suggestions, auto-complete template for methods to be overridden.
>
> ~ Yogi
>
> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com> wrote:
>
>> Hi,
>>
>> I am developing the sample Apex application to explore Malhar operators.
>> In my case, I need to write the processed data to HDFS. I did not find
>> concrete class for file writing, instead I got AbstractFileOutput operator
>> class which will have to implement it. Can I have references/ examples to
>> how to implement the abstract classes in Malhar ? If not, it would be nice
>> to have such classes in Malhar.
>>
>> --
>> Thanks,
>> Akshay
>>
>
>
--
Thanks,
Akshay
Re: Concrete classes in Malhar
Posted by Yogi Devendra <de...@datatorrent.com>.
Here is the Javadoc:
https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html
Here is the user guide:
http://docs.datatorrent.com/operators/file_output/
You will need to provide implementation for couple of methods:
1. protected abstract String getFileName(INPUT tuple)
You could simply return "output.txt"
2. protected abstract byte[] getBytesForTuple(INPUT tuple)
You need to provide some way to serialize input tuple.
It could be as simple as return tuple.toString().getBytes()
It would be great if you use some IDE like eclipse or intellij; it should
give you suggestions, auto-complete template for methods to be overridden.
~ Yogi
On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore <go...@gmail.com> wrote:
> Hi,
>
> I am developing the sample Apex application to explore Malhar operators.
> In my case, I need to write the processed data to HDFS. I did not find
> concrete class for file writing, instead I got AbstractFileOutput operator
> class which will have to implement it. Can I have references/ examples to
> how to implement the abstract classes in Malhar ? If not, it would be nice
> to have such classes in Malhar.
>
> --
> Thanks,
> Akshay
>