You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Priyanka Gugale <pr...@datatorrent.com> on 2016/10/04 10:17:59 UTC
Re: Writing to external systems in reconciled fashion
I am reconsidering using WindowDataManager or Hbase instead of ManagedState
to implement SpillableQueue.
Let me know if anyone have any thoughts on same.
-Priyanka
On Thu, Sep 29, 2016 at 5:54 PM, Priyanka Gugale <pr...@datatorrent.com>
wrote:
> Hi,
>
> Please refer to this link for earlier discussion reference: https://lists.
> apache.org/list.html?dev@apex.apache.org
>
> -Priyanka
>
> On Thu, Sep 29, 2016 at 5:46 PM, Priyanka Gugale <priyanka@datatorrent.com
> > wrote:
>
>> Dear community,
>>
>> We had a discussion
>> <https://mail.google.com/mail/u/0/#search/label%3ADiscussion-Forum+reconcile/153bfa055614fd8d?compose=15775dd6fda3cc52>
>> before about having reconciler for writing to JDBC output operator. I am
>> proposing to write a reconciler plugin which should be generic enough to
>> work with most of output Systems. Please refer this
>> <https://docs.google.com/document/d/1zKjpBG1nTZ1f3_SjFLVSf5_9Ly9Xl0kzhD5QN6-ZqS0/edit?usp=sharing>
>> document for design details. We also have a jira
>> <https://issues.apache.org/jira/browse/APEXMALHAR-2037> for same.
>>
>> Please provide your feedback.
>>
>> -Priyanka
>>
>>
>
Re: Writing to external systems in reconciled fashion
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Hi Priyanka,
tuples stored in HFile will not be replayed in same order at the
output as HFile will save tuples in different order. If order is
important then you could use
org.apache.apex.malhar.lib.wal.FileSystemWAL which is like a on-disk
queue. Or your could directly
use more high level SpillableArrayListImpl.
- Tushar.
On Tue, Oct 4, 2016 at 3:48 PM, Priyanka Gugale
<pr...@datatorrent.com> wrote:
> I am specifically looking into HFiles when I mentioned Hbase.
>
> -Priyanka
>
> On Tue, Oct 4, 2016 at 3:47 PM, Priyanka Gugale <pr...@datatorrent.com>
> wrote:
>
>> I am reconsidering using WindowDataManager or Hbase instead of
>> ManagedState to implement SpillableQueue.
>> Let me know if anyone have any thoughts on same.
>>
>> -Priyanka
>>
>> On Thu, Sep 29, 2016 at 5:54 PM, Priyanka Gugale <priyanka@datatorrent.com
>> > wrote:
>>
>>> Hi,
>>>
>>> Please refer to this link for earlier discussion reference:
>>> https://lists.apache.org/list.html?dev@apex.apache.org
>>>
>>> -Priyanka
>>>
>>> On Thu, Sep 29, 2016 at 5:46 PM, Priyanka Gugale <
>>> priyanka@datatorrent.com> wrote:
>>>
>>>> Dear community,
>>>>
>>>> We had a discussion
>>>> <https://mail.google.com/mail/u/0/#search/label%3ADiscussion-Forum+reconcile/153bfa055614fd8d?compose=15775dd6fda3cc52>
>>>> before about having reconciler for writing to JDBC output operator. I am
>>>> proposing to write a reconciler plugin which should be generic enough to
>>>> work with most of output Systems. Please refer this
>>>> <https://docs.google.com/document/d/1zKjpBG1nTZ1f3_SjFLVSf5_9Ly9Xl0kzhD5QN6-ZqS0/edit?usp=sharing>
>>>> document for design details. We also have a jira
>>>> <https://issues.apache.org/jira/browse/APEXMALHAR-2037> for same.
>>>>
>>>> Please provide your feedback.
>>>>
>>>> -Priyanka
>>>>
>>>>
>>>
>>
Re: Writing to external systems in reconciled fashion
Posted by Priyanka Gugale <pr...@datatorrent.com>.
I am specifically looking into HFiles when I mentioned Hbase.
-Priyanka
On Tue, Oct 4, 2016 at 3:47 PM, Priyanka Gugale <pr...@datatorrent.com>
wrote:
> I am reconsidering using WindowDataManager or Hbase instead of
> ManagedState to implement SpillableQueue.
> Let me know if anyone have any thoughts on same.
>
> -Priyanka
>
> On Thu, Sep 29, 2016 at 5:54 PM, Priyanka Gugale <priyanka@datatorrent.com
> > wrote:
>
>> Hi,
>>
>> Please refer to this link for earlier discussion reference:
>> https://lists.apache.org/list.html?dev@apex.apache.org
>>
>> -Priyanka
>>
>> On Thu, Sep 29, 2016 at 5:46 PM, Priyanka Gugale <
>> priyanka@datatorrent.com> wrote:
>>
>>> Dear community,
>>>
>>> We had a discussion
>>> <https://mail.google.com/mail/u/0/#search/label%3ADiscussion-Forum+reconcile/153bfa055614fd8d?compose=15775dd6fda3cc52>
>>> before about having reconciler for writing to JDBC output operator. I am
>>> proposing to write a reconciler plugin which should be generic enough to
>>> work with most of output Systems. Please refer this
>>> <https://docs.google.com/document/d/1zKjpBG1nTZ1f3_SjFLVSf5_9Ly9Xl0kzhD5QN6-ZqS0/edit?usp=sharing>
>>> document for design details. We also have a jira
>>> <https://issues.apache.org/jira/browse/APEXMALHAR-2037> for same.
>>>
>>> Please provide your feedback.
>>>
>>> -Priyanka
>>>
>>>
>>
>