You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Markus Weimer (JIRA)" <ji...@apache.org> on 2015/10/26 19:57:27 UTC
[jira] [Commented] (REEF-873) Fix DataSet id issue
[ https://issues.apache.org/jira/browse/REEF-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974832#comment-14974832 ]
Markus Weimer commented on REEF-873:
------------------------------------
I'm not sure I 100% understand the problem. Are long IDs breaking something in our code? Or are other systems unable to accept them?
I'm somewhat nervous about removing the file name from the ID. Having it in there makes for very nice debugging. Maybe we need to add a nice {{ToString}} method to the partition class such that we can still log the file name?
> Fix DataSet id issue
> --------------------
>
> Key: REEF-873
> URL: https://issues.apache.org/jira/browse/REEF-873
> Project: REEF
> Issue Type: Bug
> Reporter: Julia
> Assignee: Julia
>
> Currently the id of the dataset is formed from extract some string from the input file name. In reality, the input file name itself can be a generated file with some random numbers that makes the file name has no meaning at all. And since the file name can be very long, that can also break the current way to form the id.
> We should remove this id generation dependency on the input file names, instead, comes out a id with a prefix like "FileSystemDataSet-" plus a guid for example, as long as it is unique.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [jira] [Commented] (REEF-873) Fix DataSet id issue
Posted by Dhruv Mahajan <dh...@gmail.com>.
Agreed with this. Add ToString to the partition class and change Id ot what
Julia suggested.
Dhruv
On Mon, Oct 26, 2015 at 11:57 AM, Markus Weimer (JIRA) <ji...@apache.org>
wrote:
>
> [
> https://issues.apache.org/jira/browse/REEF-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974832#comment-14974832
> ]
>
> Markus Weimer commented on REEF-873:
> ------------------------------------
>
> I'm not sure I 100% understand the problem. Are long IDs breaking
> something in our code? Or are other systems unable to accept them?
>
> I'm somewhat nervous about removing the file name from the ID. Having it
> in there makes for very nice debugging. Maybe we need to add a nice
> {{ToString}} method to the partition class such that we can still log the
> file name?
>
> > Fix DataSet id issue
> > --------------------
> >
> > Key: REEF-873
> > URL: https://issues.apache.org/jira/browse/REEF-873
> > Project: REEF
> > Issue Type: Bug
> > Reporter: Julia
> > Assignee: Julia
> >
> > Currently the id of the dataset is formed from extract some string from
> the input file name. In reality, the input file name itself can be a
> generated file with some random numbers that makes the file name has no
> meaning at all. And since the file name can be very long, that can also
> break the current way to form the id.
> > We should remove this id generation dependency on the input file names,
> instead, comes out a id with a prefix like "FileSystemDataSet-" plus a guid
> for example, as long as it is unique.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>