You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streams.apache.org by rbnks <gi...@git.apache.org> on 2014/05/15 20:40:33 UTC
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
GitHub user rbnks opened a pull request:
https://github.com/apache/incubator-streams/pull/15
Fixed bug for STREAMS-80
Fixed bug that would cause processors and writers to end execution early and orphan data. Bug was introduced in a recent commit.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rbnks/incubator-streams STREAMS-80
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-streams/pull/15.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15
----
commit 00a2a27766788c92698ae91a9f128f253aaa9891
Author: rebanks <re...@w2odigital.com>
Date: 2014-05-15T18:39:10Z
Fixed bug for STREAMS-80
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by smashew <gi...@git.apache.org>.
Github user smashew commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-43397816
FYI... PR 16
https://github.com/apache/incubator-streams/pull/16
Additionally satisfies this request and many others with a large suite of tests to ensure orphaned data does not occur. I don't know how, if at all, these two PRs would merge together. :(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by rbnks <gi...@git.apache.org>.
Github user rbnks commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-44430371
Closing this out and moving discussion to the list serve is fine.
As far as the issue of processors/writers running other threads the task is unaware of should not be an issue. The task calls cleanUp() on all components before exiting, and it should be the components responsibility to shut down any of its threads and/or async tasks. If the processor/writer is not emptying internal queues and shutting down threads that is an error in the processor/writer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by mfranklin <gi...@git.apache.org>.
Github user mfranklin commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-43529420
This does not appear to be the best way to solve for the problem you are encountering. The semantics of the setRunning(false) method indicate that I want the component to terminate now, not flush the remaining items then terminate. We either need to not set running to false until the queue is exhausted, or have another method that is something like flushBacklogAndTerminate()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by smashew <gi...@git.apache.org>.
Github user smashew commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-43530900
There are other ways to do it as well. Including having an executor thread that tracks the 'running' state by asking if there are active items in the queue. This is implemented on my fork if you care to have a gander.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by rbnks <gi...@git.apache.org>.
Github user rbnks commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-48402594
I assumed someone else would close this. but i'll officially close this request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by smashew <gi...@git.apache.org>.
Github user smashew commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-43531042
Essentially, once the provider is terminated the hose is 'shut off' and all the remaining queues deplete until they are satisfied, however long that takes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by mfranklin <gi...@git.apache.org>.
Github user mfranklin commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-43804369
Right. That happens today with at least the ElasticsearchPersistWriter. There is an open thread that needs to get summarized with next steps to give more control to the readers & writers on their state.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by smashew <gi...@git.apache.org>.
Github user smashew commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-43803287
They are also not aware if the processor or writer is actually running. If the user in 'write(StreamsDatum datum)' starts a thread to process that, there isn't any information for LocalStreamsBuilder to know that is happening. So, the queue is depleted, but another thread is hanging around with a datum.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by rbnks <gi...@git.apache.org>.
Github user rbnks closed the pull request at:
https://github.com/apache/incubator-streams/pull/15
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by mfranklin <gi...@git.apache.org>.
Github user mfranklin commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-43795138
LocalBuilder appears to be checking the queue sizes on components before calling shutdown, so the only way the queues are not flushed is if shutdown is triggered from outside of the start method. This would be due to a shutdown hook or other external call.
The question is what is the desired behavior. If I hit CTRL-C on my console, do I want it to flush all the remaining data or do I want it to stop?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by smashew <gi...@git.apache.org>.
Github user smashew commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-43804858
In the runtime environments, save LocalStreamsBuilder everything should be run as a thread in a thread monitor, which can then be checked to see if there is an active queue. Then we should document that the 'writeDatum' method is already threaded and the user shouldn't thread the call further. Or at least know that it is tracking the method 'writeDatum' to establish that it's effort is completed. (see my fork for a reference implementation of exactly this)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by mfranklin <gi...@git.apache.org>.
Github user mfranklin commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-43940847
@rbnks, I recommend we take this discussion to the thread and close this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Fixed bug for STREAMS-80
Posted by rbnks <gi...@git.apache.org>.
Github user rbnks commented on the pull request:
https://github.com/apache/incubator-streams/pull/15#issuecomment-43766976
@mfranklin So I can see how maybe we want to change the name of keepRunning, or we might want to have a shut off now method in the future. But currently we only have one way to shut down the queue. And checking if 'datum != null' will flush all of the data out of the queue before it shuts. And since local mode shuts down each task in upstream to downstream order, we can know we will flush all of the data.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---