You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by 舒琦 <sh...@eefung.com> on 2017/08/24 06:17:38 UTC
Question about stop in SystemProducer
Hi,
I write a SystemProducer for HDFS and everything is fine. When samza job is shutdown, the stop method in SystemProducer is called, but the close files operation(may need some time, there may be cache data to be flushed) in stop method is not executed completely.
How can I resolve this problem?
Thanks!
————————
Qi Shu
Fwd: Question about stop in SystemProducer
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Hi Shuqi,
Glad that you figured out the problem. In general, we recommend Filesystem
instances should always be scoped to their corresponding objects and
discourage sharing.
Please do let us know should you need more help!
Thanks
---------- Forwarded message ---------
From: 舒琦 <sh...@eefung.com>
Date: Thu, Aug 24, 2017 at 11:43 PM
Subject: Re: Question about stop in SystemProducer
To: <de...@samza.apache.org>
Hi Jagadish,
I resolved the problem. Samza must close the FileSystem somewhere
when “shutdown” is triggered. Now I don’t get FileSystem instance from
cache and everything works fine.
Thanks!
————————
Qi Shu
> 在 2017年8月24日，15:06，舒琦 <sh...@eefung.com> 写道：
>
> Yes, in the same thread.
>
> File list is not empty, because after samza job shutdown, all files in
writing are not in complete state.
>
> ————————
> Qi Shu
>
>> 在 2017年8月24日，15:00，Jagadish Venkatraman <ja...@gmail.com> 写道：
>>
>>>> Now the problem is not program hang, but the container quit
>> before the “closeFiles” method executed completely.
>>
>> It's unlikely that the container quit without returning from *close*. Are
>> you sure there were filesList is not empty? I'd suggest adding more
logging.
>>
>> I'm assuming you are calling *closeFiles* in the same thread?
>>
>> On Wed, Aug 23, 2017 at 11:53 PM, 舒琦 <sh...@eefung.com> wrote:
>>
>>> Hi Jagadish,
>>>
>>> When samza job is shutdown, “Begin to stop” is printed out in the
>>> log file, but not the log in closeFiles.
>>> Now the problem is not program hang, but the container quit before
>>> the “closeFiles” method executed completely.
>>>
>>> Thanks.
>>>
>>> ————————
>>> Qi Shu
>>>
>>>> 在 2017年8月24日，14:44，Jagadish Venkatraman <ja...@gmail.com> 写道：
>>>>
>>>> Is "Begin to close file" printed? Where exactly is your application
>>> stalled
>>>> ? I'd suggest you take a stack dump.
>>>>
>>>> On Wed, Aug 23, 2017 at 11:32 PM, 舒琦 <sh...@eefung.com> wrote:
>>>>
>>>>> Hi Jagadish,
>>>>>
>>>>> Below is part of the related code, log “"Begin to stop” is
>>> printed
>>>>> out.
>>>>>
>>>>> Thank you!
>>>>>
>>>>> public abstract class MyProducer implements SystemProducer {
>>>>>
>>>>> @Override
>>>>> public void stop() {
>>>>> LOGGER.info("Begin to stop");
>>>>>
>>>>> closeFiles();
>>>>>
>>>>> LOGGER.info("End to stop");
>>>>> }
>>>>>
>>>>> @Override
>>>>> protected void closeFiles() throws IOException {
>>>>> closeFiles(statusFiles);
>>>>> closeFiles(interactionFiles);
>>>>> }
>>>>>
>>>>> private void closeFiles(Map<String, FileInWriting> files) throws
>>>>> IOException {
>>>>> for (FileInWriting statusFile : files.values()) {
>>>>> LOGGER.info("Begin to close file: {}",
>>>>> statusFile.getFilePath());
>>>>> statusFile.getWriter().hflush();
>>>>> statusFile.getWriter().close();
>>>>> renameFile(statusFile.getFilePath());
>>>>> LOGGER.info("Successfully close file: {}",
>>>>> statusFile.getFilePath());
>>>>> }
>>>>>
>>>>> files.clear();
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> ————————
>>>>> Qi Shu
>>>>>
>>>>>> 在 2017年8月24日，14:21，Jagadish Venkatraman <ja...@gmail.com> 写道：
>>>>>>
>>>>>> Hi Qi,
>>>>>>
>>>>>>>> the stop method in SystemProducer is called, but the close files
>>>>>> operation(may need some time, there may be cache data to be flushed)
in
>>>>>> stop method is not executed completely
>>>>>>
>>>>>> Are you seeing the *close()* method hang? SystemProducer.*close* is a
>>>>>> synchronous operation, and will block for the *close* method to
finish.
>>>>>>
>>>>>> Best,
>>>>>> Jagadish
>>>>>>
>>>>>> On Wed, Aug 23, 2017 at 11:17 PM, 舒琦 <sh...@eefung.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I write a SystemProducer for HDFS and everything is fine. When
>>>>>>> samza job is shutdown, the stop method in SystemProducer is called,
>>> but
>>>>> the
>>>>>>> close files operation(may need some time, there may be cache data to
>>> be
>>>>>>> flushed) in stop method is not executed completely.
>>>>>>>
>>>>>>> How can I resolve this problem?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> ————————
>>>>>>> Qi Shu
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jagadish V,
>>>>>> Graduate Student,
>>>>>> Department of Computer Science,
>>>>>> Stanford University
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Jagadish V,
>>>> Graduate Student,
>>>> Department of Computer Science,
>>>> Stanford University
>>>
>>>
>>
>>
>> --
>> Jagadish V,
>> Graduate Student,
>> Department of Computer Science,
>> Stanford University
>
--
Sent from my iphone.
Re: Question about stop in SystemProducer
Posted by 舒琦 <sh...@eefung.com>.
Hi Jagadish,
I resolved the problem. Samza must close the FileSystem somewhere when “shutdown” is triggered. Now I don’t get FileSystem instance from cache and everything works fine.
Thanks!
————————
Qi Shu
> 在 2017年8月24日，15:06，舒琦 <sh...@eefung.com> 写道：
>
> Yes, in the same thread.
>
> File list is not empty, because after samza job shutdown, all files in writing are not in complete state.
>
> ————————
> Qi Shu
>
>> 在 2017年8月24日，15:00，Jagadish Venkatraman <ja...@gmail.com> 写道：
>>
>>>> Now the problem is not program hang, but the container quit
>> before the “closeFiles” method executed completely.
>>
>> It's unlikely that the container quit without returning from *close*. Are
>> you sure there were filesList is not empty? I'd suggest adding more logging.
>>
>> I'm assuming you are calling *closeFiles* in the same thread?
>>
>> On Wed, Aug 23, 2017 at 11:53 PM, 舒琦 <sh...@eefung.com> wrote:
>>
>>> Hi Jagadish,
>>>
>>> When samza job is shutdown, “Begin to stop” is printed out in the
>>> log file, but not the log in closeFiles.
>>> Now the problem is not program hang, but the container quit before
>>> the “closeFiles” method executed completely.
>>>
>>> Thanks.
>>>
>>> ————————
>>> Qi Shu
>>>
>>>> 在 2017年8月24日，14:44，Jagadish Venkatraman <ja...@gmail.com> 写道：
>>>>
>>>> Is "Begin to close file" printed? Where exactly is your application
>>> stalled
>>>> ? I'd suggest you take a stack dump.
>>>>
>>>> On Wed, Aug 23, 2017 at 11:32 PM, 舒琦 <sh...@eefung.com> wrote:
>>>>
>>>>> Hi Jagadish,
>>>>>
>>>>> Below is part of the related code, log “"Begin to stop” is
>>> printed
>>>>> out.
>>>>>
>>>>> Thank you!
>>>>>
>>>>> public abstract class MyProducer implements SystemProducer {
>>>>>
>>>>> @Override
>>>>> public void stop() {
>>>>> LOGGER.info("Begin to stop");
>>>>>
>>>>> closeFiles();
>>>>>
>>>>> LOGGER.info("End to stop");
>>>>> }
>>>>>
>>>>> @Override
>>>>> protected void closeFiles() throws IOException {
>>>>> closeFiles(statusFiles);
>>>>> closeFiles(interactionFiles);
>>>>> }
>>>>>
>>>>> private void closeFiles(Map<String, FileInWriting> files) throws
>>>>> IOException {
>>>>> for (FileInWriting statusFile : files.values()) {
>>>>> LOGGER.info("Begin to close file: {}",
>>>>> statusFile.getFilePath());
>>>>> statusFile.getWriter().hflush();
>>>>> statusFile.getWriter().close();
>>>>> renameFile(statusFile.getFilePath());
>>>>> LOGGER.info("Successfully close file: {}",
>>>>> statusFile.getFilePath());
>>>>> }
>>>>>
>>>>> files.clear();
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> ————————
>>>>> Qi Shu
>>>>>
>>>>>> 在 2017年8月24日，14:21，Jagadish Venkatraman <ja...@gmail.com> 写道：
>>>>>>
>>>>>> Hi Qi,
>>>>>>
>>>>>>>> the stop method in SystemProducer is called, but the close files
>>>>>> operation(may need some time, there may be cache data to be flushed) in
>>>>>> stop method is not executed completely
>>>>>>
>>>>>> Are you seeing the *close()* method hang? SystemProducer.*close* is a
>>>>>> synchronous operation, and will block for the *close* method to finish.
>>>>>>
>>>>>> Best,
>>>>>> Jagadish
>>>>>>
>>>>>> On Wed, Aug 23, 2017 at 11:17 PM, 舒琦 <sh...@eefung.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I write a SystemProducer for HDFS and everything is fine. When
>>>>>>> samza job is shutdown, the stop method in SystemProducer is called,
>>> but
>>>>> the
>>>>>>> close files operation(may need some time, there may be cache data to
>>> be
>>>>>>> flushed) in stop method is not executed completely.
>>>>>>>
>>>>>>> How can I resolve this problem?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> ————————
>>>>>>> Qi Shu
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jagadish V,
>>>>>> Graduate Student,
>>>>>> Department of Computer Science,
>>>>>> Stanford University
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Jagadish V,
>>>> Graduate Student,
>>>> Department of Computer Science,
>>>> Stanford University
>>>
>>>
>>
>>
>> --
>> Jagadish V,
>> Graduate Student,
>> Department of Computer Science,
>> Stanford University
>
Re: Question about stop in SystemProducer
Posted by 舒琦 <sh...@eefung.com>.
Yes, in the same thread.
File list is not empty, because after samza job shutdown, all files in writing are not in complete state.
————————
Qi Shu
> 在 2017年8月24日，15:00，Jagadish Venkatraman <ja...@gmail.com> 写道：
>
>>> Now the problem is not program hang, but the container quit
> before the “closeFiles” method executed completely.
>
> It's unlikely that the container quit without returning from *close*. Are
> you sure there were filesList is not empty? I'd suggest adding more logging.
>
> I'm assuming you are calling *closeFiles* in the same thread?
>
> On Wed, Aug 23, 2017 at 11:53 PM, 舒琦 <sh...@eefung.com> wrote:
>
>> Hi Jagadish,
>>
>> When samza job is shutdown, “Begin to stop” is printed out in the
>> log file, but not the log in closeFiles.
>> Now the problem is not program hang, but the container quit before
>> the “closeFiles” method executed completely.
>>
>> Thanks.
>>
>> ————————
>> Qi Shu
>>
>>> 在 2017年8月24日，14:44，Jagadish Venkatraman <ja...@gmail.com> 写道：
>>>
>>> Is "Begin to close file" printed? Where exactly is your application
>> stalled
>>> ? I'd suggest you take a stack dump.
>>>
>>> On Wed, Aug 23, 2017 at 11:32 PM, 舒琦 <sh...@eefung.com> wrote:
>>>
>>>> Hi Jagadish,
>>>>
>>>> Below is part of the related code, log “"Begin to stop” is
>> printed
>>>> out.
>>>>
>>>> Thank you!
>>>>
>>>> public abstract class MyProducer implements SystemProducer {
>>>>
>>>> @Override
>>>> public void stop() {
>>>> LOGGER.info("Begin to stop");
>>>>
>>>> closeFiles();
>>>>
>>>> LOGGER.info("End to stop");
>>>> }
>>>>
>>>> @Override
>>>> protected void closeFiles() throws IOException {
>>>> closeFiles(statusFiles);
>>>> closeFiles(interactionFiles);
>>>> }
>>>>
>>>> private void closeFiles(Map<String, FileInWriting> files) throws
>>>> IOException {
>>>> for (FileInWriting statusFile : files.values()) {
>>>> LOGGER.info("Begin to close file: {}",
>>>> statusFile.getFilePath());
>>>> statusFile.getWriter().hflush();
>>>> statusFile.getWriter().close();
>>>> renameFile(statusFile.getFilePath());
>>>> LOGGER.info("Successfully close file: {}",
>>>> statusFile.getFilePath());
>>>> }
>>>>
>>>> files.clear();
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> ————————
>>>> Qi Shu
>>>>
>>>>> 在 2017年8月24日，14:21，Jagadish Venkatraman <ja...@gmail.com> 写道：
>>>>>
>>>>> Hi Qi,
>>>>>
>>>>>>> the stop method in SystemProducer is called, but the close files
>>>>> operation(may need some time, there may be cache data to be flushed) in
>>>>> stop method is not executed completely
>>>>>
>>>>> Are you seeing the *close()* method hang? SystemProducer.*close* is a
>>>>> synchronous operation, and will block for the *close* method to finish.
>>>>>
>>>>> Best,
>>>>> Jagadish
>>>>>
>>>>> On Wed, Aug 23, 2017 at 11:17 PM, 舒琦 <sh...@eefung.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I write a SystemProducer for HDFS and everything is fine. When
>>>>>> samza job is shutdown, the stop method in SystemProducer is called,
>> but
>>>> the
>>>>>> close files operation(may need some time, there may be cache data to
>> be
>>>>>> flushed) in stop method is not executed completely.
>>>>>>
>>>>>> How can I resolve this problem?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> ————————
>>>>>> Qi Shu
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jagadish V,
>>>>> Graduate Student,
>>>>> Department of Computer Science,
>>>>> Stanford University
>>>>
>>>>
>>>
>>>
>>> --
>>> Jagadish V,
>>> Graduate Student,
>>> Department of Computer Science,
>>> Stanford University
>>
>>
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
Re: Question about stop in SystemProducer
Posted by Jagadish Venkatraman <ja...@gmail.com>.
>> Now the problem is not program hang, but the container quit
before the “closeFiles” method executed completely.
It's unlikely that the container quit without returning from *close*. Are
you sure there were filesList is not empty? I'd suggest adding more logging.
I'm assuming you are calling *closeFiles* in the same thread?
On Wed, Aug 23, 2017 at 11:53 PM, 舒琦 <sh...@eefung.com> wrote:
> Hi Jagadish,
>
> When samza job is shutdown, “Begin to stop” is printed out in the
> log file, but not the log in closeFiles.
> Now the problem is not program hang, but the container quit before
> the “closeFiles” method executed completely.
>
> Thanks.
>
> ————————
> Qi Shu
>
> > 在 2017年8月24日，14:44，Jagadish Venkatraman <ja...@gmail.com> 写道：
> >
> > Is "Begin to close file" printed? Where exactly is your application
> stalled
> > ? I'd suggest you take a stack dump.
> >
> > On Wed, Aug 23, 2017 at 11:32 PM, 舒琦 <sh...@eefung.com> wrote:
> >
> >> Hi Jagadish,
> >>
> >> Below is part of the related code, log “"Begin to stop” is
> printed
> >> out.
> >>
> >> Thank you!
> >>
> >> public abstract class MyProducer implements SystemProducer {
> >>
> >> @Override
> >> public void stop() {
> >> LOGGER.info("Begin to stop");
> >>
> >> closeFiles();
> >>
> >> LOGGER.info("End to stop");
> >> }
> >>
> >> @Override
> >> protected void closeFiles() throws IOException {
> >> closeFiles(statusFiles);
> >> closeFiles(interactionFiles);
> >> }
> >>
> >> private void closeFiles(Map<String, FileInWriting> files) throws
> >> IOException {
> >> for (FileInWriting statusFile : files.values()) {
> >> LOGGER.info("Begin to close file: {}",
> >> statusFile.getFilePath());
> >> statusFile.getWriter().hflush();
> >> statusFile.getWriter().close();
> >> renameFile(statusFile.getFilePath());
> >> LOGGER.info("Successfully close file: {}",
> >> statusFile.getFilePath());
> >> }
> >>
> >> files.clear();
> >> }
> >>
> >> }
> >>
> >>
> >> ————————
> >> Qi Shu
> >>
> >>> 在 2017年8月24日，14:21，Jagadish Venkatraman <ja...@gmail.com> 写道：
> >>>
> >>> Hi Qi,
> >>>
> >>>>> the stop method in SystemProducer is called, but the close files
> >>> operation(may need some time, there may be cache data to be flushed) in
> >>> stop method is not executed completely
> >>>
> >>> Are you seeing the *close()* method hang? SystemProducer.*close* is a
> >>> synchronous operation, and will block for the *close* method to finish.
> >>>
> >>> Best,
> >>> Jagadish
> >>>
> >>> On Wed, Aug 23, 2017 at 11:17 PM, 舒琦 <sh...@eefung.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I write a SystemProducer for HDFS and everything is fine. When
> >>>> samza job is shutdown, the stop method in SystemProducer is called,
> but
> >> the
> >>>> close files operation(may need some time, there may be cache data to
> be
> >>>> flushed) in stop method is not executed completely.
> >>>>
> >>>> How can I resolve this problem?
> >>>>
> >>>> Thanks!
> >>>>
> >>>> ————————
> >>>> Qi Shu
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Jagadish V,
> >>> Graduate Student,
> >>> Department of Computer Science,
> >>> Stanford University
> >>
> >>
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
>
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
Re: Question about stop in SystemProducer
Posted by 舒琦 <sh...@eefung.com>.
Hi Jagadish,
When samza job is shutdown, “Begin to stop” is printed out in the log file, but not the log in closeFiles.
Now the problem is not program hang, but the container quit before the “closeFiles” method executed completely.
Thanks.
————————
Qi Shu
> 在 2017年8月24日，14:44，Jagadish Venkatraman <ja...@gmail.com> 写道：
>
> Is "Begin to close file" printed? Where exactly is your application stalled
> ? I'd suggest you take a stack dump.
>
> On Wed, Aug 23, 2017 at 11:32 PM, 舒琦 <sh...@eefung.com> wrote:
>
>> Hi Jagadish,
>>
>> Below is part of the related code, log “"Begin to stop” is printed
>> out.
>>
>> Thank you!
>>
>> public abstract class MyProducer implements SystemProducer {
>>
>> @Override
>> public void stop() {
>> LOGGER.info("Begin to stop");
>>
>> closeFiles();
>>
>> LOGGER.info("End to stop");
>> }
>>
>> @Override
>> protected void closeFiles() throws IOException {
>> closeFiles(statusFiles);
>> closeFiles(interactionFiles);
>> }
>>
>> private void closeFiles(Map<String, FileInWriting> files) throws
>> IOException {
>> for (FileInWriting statusFile : files.values()) {
>> LOGGER.info("Begin to close file: {}",
>> statusFile.getFilePath());
>> statusFile.getWriter().hflush();
>> statusFile.getWriter().close();
>> renameFile(statusFile.getFilePath());
>> LOGGER.info("Successfully close file: {}",
>> statusFile.getFilePath());
>> }
>>
>> files.clear();
>> }
>>
>> }
>>
>>
>> ————————
>> Qi Shu
>>
>>> 在 2017年8月24日，14:21，Jagadish Venkatraman <ja...@gmail.com> 写道：
>>>
>>> Hi Qi,
>>>
>>>>> the stop method in SystemProducer is called, but the close files
>>> operation(may need some time, there may be cache data to be flushed) in
>>> stop method is not executed completely
>>>
>>> Are you seeing the *close()* method hang? SystemProducer.*close* is a
>>> synchronous operation, and will block for the *close* method to finish.
>>>
>>> Best,
>>> Jagadish
>>>
>>> On Wed, Aug 23, 2017 at 11:17 PM, 舒琦 <sh...@eefung.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I write a SystemProducer for HDFS and everything is fine. When
>>>> samza job is shutdown, the stop method in SystemProducer is called, but
>> the
>>>> close files operation(may need some time, there may be cache data to be
>>>> flushed) in stop method is not executed completely.
>>>>
>>>> How can I resolve this problem?
>>>>
>>>> Thanks!
>>>>
>>>> ————————
>>>> Qi Shu
>>>>
>>>>
>>>
>>>
>>> --
>>> Jagadish V,
>>> Graduate Student,
>>> Department of Computer Science,
>>> Stanford University
>>
>>
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
Re: Question about stop in SystemProducer
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Is "Begin to close file" printed? Where exactly is your application stalled
? I'd suggest you take a stack dump.
On Wed, Aug 23, 2017 at 11:32 PM, 舒琦 <sh...@eefung.com> wrote:
> Hi Jagadish,
>
> Below is part of the related code, log “"Begin to stop” is printed
> out.
>
> Thank you!
>
> public abstract class MyProducer implements SystemProducer {
>
> @Override
> public void stop() {
> LOGGER.info("Begin to stop");
>
> closeFiles();
>
> LOGGER.info("End to stop");
> }
>
> @Override
> protected void closeFiles() throws IOException {
> closeFiles(statusFiles);
> closeFiles(interactionFiles);
> }
>
> private void closeFiles(Map<String, FileInWriting> files) throws
> IOException {
> for (FileInWriting statusFile : files.values()) {
> LOGGER.info("Begin to close file: {}",
> statusFile.getFilePath());
> statusFile.getWriter().hflush();
> statusFile.getWriter().close();
> renameFile(statusFile.getFilePath());
> LOGGER.info("Successfully close file: {}",
> statusFile.getFilePath());
> }
>
> files.clear();
> }
>
> }
>
>
> ————————
> Qi Shu
>
> > 在 2017年8月24日，14:21，Jagadish Venkatraman <ja...@gmail.com> 写道：
> >
> > Hi Qi,
> >
> >>> the stop method in SystemProducer is called, but the close files
> > operation(may need some time, there may be cache data to be flushed) in
> > stop method is not executed completely
> >
> > Are you seeing the *close()* method hang? SystemProducer.*close* is a
> > synchronous operation, and will block for the *close* method to finish.
> >
> > Best,
> > Jagadish
> >
> > On Wed, Aug 23, 2017 at 11:17 PM, 舒琦 <sh...@eefung.com> wrote:
> >
> >> Hi,
> >>
> >> I write a SystemProducer for HDFS and everything is fine. When
> >> samza job is shutdown, the stop method in SystemProducer is called, but
> the
> >> close files operation(may need some time, there may be cache data to be
> >> flushed) in stop method is not executed completely.
> >>
> >> How can I resolve this problem?
> >>
> >> Thanks!
> >>
> >> ————————
> >> Qi Shu
> >>
> >>
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
>
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
Re: Question about stop in SystemProducer
Posted by 舒琦 <sh...@eefung.com>.
Hi Jagadish,
Below is part of the related code, log “"Begin to stop” is printed out.
Thank you!
public abstract class MyProducer implements SystemProducer {
@Override
public void stop() {
LOGGER.info("Begin to stop");
closeFiles();
LOGGER.info("End to stop");
}
@Override
protected void closeFiles() throws IOException {
closeFiles(statusFiles);
closeFiles(interactionFiles);
}
private void closeFiles(Map<String, FileInWriting> files) throws IOException {
for (FileInWriting statusFile : files.values()) {
LOGGER.info("Begin to close file: {}", statusFile.getFilePath());
statusFile.getWriter().hflush();
statusFile.getWriter().close();
renameFile(statusFile.getFilePath());
LOGGER.info("Successfully close file: {}", statusFile.getFilePath());
}
files.clear();
}
}
————————
Qi Shu
> 在 2017年8月24日，14:21，Jagadish Venkatraman <ja...@gmail.com> 写道：
>
> Hi Qi,
>
>>> the stop method in SystemProducer is called, but the close files
> operation(may need some time, there may be cache data to be flushed) in
> stop method is not executed completely
>
> Are you seeing the *close()* method hang? SystemProducer.*close* is a
> synchronous operation, and will block for the *close* method to finish.
>
> Best,
> Jagadish
>
> On Wed, Aug 23, 2017 at 11:17 PM, 舒琦 <sh...@eefung.com> wrote:
>
>> Hi,
>>
>> I write a SystemProducer for HDFS and everything is fine. When
>> samza job is shutdown, the stop method in SystemProducer is called, but the
>> close files operation(may need some time, there may be cache data to be
>> flushed) in stop method is not executed completely.
>>
>> How can I resolve this problem?
>>
>> Thanks!
>>
>> ————————
>> Qi Shu
>>
>>
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
Re: Question about stop in SystemProducer
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Hi Qi,
>> the stop method in SystemProducer is called, but the close files
operation(may need some time, there may be cache data to be flushed) in
stop method is not executed completely
Are you seeing the *close()* method hang? SystemProducer.*close* is a
synchronous operation, and will block for the *close* method to finish.
Best,
Jagadish
On Wed, Aug 23, 2017 at 11:17 PM, 舒琦 <sh...@eefung.com> wrote:
> Hi,
>
> I write a SystemProducer for HDFS and everything is fine. When
> samza job is shutdown, the stop method in SystemProducer is called, but the
> close files operation(may need some time, there may be cache data to be
> flushed) in stop method is not executed completely.
>
> How can I resolve this problem?
>
> Thanks!
>
> ————————
> Qi Shu
>
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University