You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@batchee.apache.org by Mark Struberg <st...@yahoo.de> on 2015/03/17 10:27:54 UTC
shipping a new release?
Hi folks!
What do you think about shipping a new release?
We've fixed quite a few bugs.
LieGrue,
strub
Re: shipping a new release?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
that's even worse. Let me try to be more explicit.
You need to manage:
- checkpoint (file + line number I think but if your chunk is really the
folder then it can be something else)
- avoid concurrent reading. Common way is to add foo.txt.inprogress while
reading (or in another folder) and move foo.txt in done/ once done
Last point: this has no link with jsefa IMO but could be an extras
component (FolderReader) delegating to another reader.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-03-17 20:56 GMT+01:00 Mark Struberg <st...@yahoo.de>:
> I guess you mean the checkpoint we manage? Currently this is the line
> number, right?
>
> LieGrue,
> strub
>
>
>
>
>
> On Tuesday, 17 March 2015, 18:55, Romain Manni-Bucau <
> rmannibucau@gmail.com> wrote:
>
>
> >
> >
> >+1 but take care that's harder than a listFiles(), you need a inprogress
> and done markers
> >
> >
> >
> >Romain Manni-Bucau
> >@rmannibucau | Blog | Github | LinkedIn | Tomitriber
> >
> >2015-03-17 18:53 GMT+01:00 Reinhard Sandtner <reinhard.sandtner@gmail.com
> >:
> >
> >hey,
> >>
> >>it would be nice to have a jsefaReader which takes a list of files or a
> folder as parameter. currently you can only read a single file
> >>
> >>wdyt?
> >>
> >>lg
> >>reini
> >>
> >>> Am 17.03.2015 um 10:30 schrieb Romain Manni-Bucau <
> rmannibucau@gmail.com>:
> >>>
> >>> no time for it myself but +1
> >>>
> >>>
> >>> Romain Manni-Bucau
> >>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >>> <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> >>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> >>> <http://www.tomitribe.com>
> >>
> >>>
> >>> 2015-03-17 10:27 GMT+01:00 Mark Struberg <st...@yahoo.de>:
> >>>
> >>>> Hi folks!
> >>>>
> >>>> What do you think about shipping a new release?
> >>>> We've fixed quite a few bugs.
> >>>>
> >>>> LieGrue,
> >>>> strub
> >>>>
> >>
> >>
> >
> >
> >
>
Re: shipping a new release?
Posted by Mark Struberg <st...@yahoo.de>.
I guess you mean the checkpoint we manage? Currently this is the line number, right?
LieGrue,
strub
On Tuesday, 17 March 2015, 18:55, Romain Manni-Bucau <rm...@gmail.com> wrote:
>
>
>+1 but take care that's harder than a listFiles(), you need a inprogress and done markers
>
>
>
>Romain Manni-Bucau
>@rmannibucau | Blog | Github | LinkedIn | Tomitriber
>
>2015-03-17 18:53 GMT+01:00 Reinhard Sandtner <re...@gmail.com>:
>
>hey,
>>
>>it would be nice to have a jsefaReader which takes a list of files or a folder as parameter. currently you can only read a single file
>>
>>wdyt?
>>
>>lg
>>reini
>>
>>> Am 17.03.2015 um 10:30 schrieb Romain Manni-Bucau <rm...@gmail.com>:
>>>
>>> no time for it myself but +1
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
>>> <http://www.tomitribe.com>
>>
>>>
>>> 2015-03-17 10:27 GMT+01:00 Mark Struberg <st...@yahoo.de>:
>>>
>>>> Hi folks!
>>>>
>>>> What do you think about shipping a new release?
>>>> We've fixed quite a few bugs.
>>>>
>>>> LieGrue,
>>>> strub
>>>>
>>
>>
>
>
>
Re: shipping a new release?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
+1 but take care that's harder than a listFiles(), you need a inprogress
and done markers
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-03-17 18:53 GMT+01:00 Reinhard Sandtner <re...@gmail.com>:
> hey,
>
> it would be nice to have a jsefaReader which takes a list of files or a
> folder as parameter. currently you can only read a single file
>
> wdyt?
>
> lg
> reini
>
> > Am 17.03.2015 um 10:30 schrieb Romain Manni-Bucau <rmannibucau@gmail.com
> >:
> >
> > no time for it myself but +1
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> > <http://www.tomitribe.com>
> >
> > 2015-03-17 10:27 GMT+01:00 Mark Struberg <st...@yahoo.de>:
> >
> >> Hi folks!
> >>
> >> What do you think about shipping a new release?
> >> We've fixed quite a few bugs.
> >>
> >> LieGrue,
> >> strub
> >>
>
>
Re: shipping a new release?
Posted by Reinhard Sandtner <re...@gmail.com>.
hey,
it would be nice to have a jsefaReader which takes a list of files or a folder as parameter. currently you can only read a single file
wdyt?
lg
reini
> Am 17.03.2015 um 10:30 schrieb Romain Manni-Bucau <rm...@gmail.com>:
>
> no time for it myself but +1
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> <http://www.tomitribe.com>
>
> 2015-03-17 10:27 GMT+01:00 Mark Struberg <st...@yahoo.de>:
>
>> Hi folks!
>>
>> What do you think about shipping a new release?
>> We've fixed quite a few bugs.
>>
>> LieGrue,
>> strub
>>
Re: shipping a new release?
Posted by Romain Manni-Bucau <rm...@gmail.com>.
no time for it myself but +1
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2015-03-17 10:27 GMT+01:00 Mark Struberg <st...@yahoo.de>:
> Hi folks!
>
> What do you think about shipping a new release?
> We've fixed quite a few bugs.
>
> LieGrue,
> strub
>