You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Stu Hood (JIRA)" <ji...@apache.org> on 2011/02/04 10:32:27 UTC
[jira] Created: (AVRO-757) Allow a DataFile to be reopened without
reading the head
Allow a DataFile to be reopened without reading the head
--------------------------------------------------------
Key: AVRO-757
URL: https://issues.apache.org/jira/browse/AVRO-757
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Stu Hood
Assignee: Stu Hood
Fix For: 1.5.0
It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AVRO-757) Allow a DataFile to be reopened
without reading the head
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990718#comment-12990718 ]
Stu Hood commented on AVRO-757:
-------------------------------
> Perhaps we should implement this by adding the following methods to DataFileReader:
This is almost exactly what I prototyped last night: I'll post a patch over the weekend.
> ...Perhaps there should be a boolean 'sync' parameter to that method that defaults to true?
I was going to take a much lazier approach that assumed that the next step after reopening was to call seek or sync, but I think you are right that we need to be in some known position.
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-757) Allow a DataFile to be reopened without
reading the head
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated AVRO-757:
--------------------------
Attachment: 0002-Expose-Header-for-use-in-DataFileReader-and-test-usage.txt
0001-Move-header-state-into-a-Header-object.txt
> the Handle fields should not be public, but private, perhaps with accessor methods.
They are currently package protected... I don't think we can do any better than that though, because there are other members of the package that need access.
I can add accessors that are also package protected?
> the Handle constructor should be private or package-private, not protected, since it
> should not appear in javadoc.
Done.
> can we call it a Header instead of a Handle, to be consistent with the spec?
Done.
Thanks Doug!
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
> Attachments: 0001-Move-header-state-into-a-Header-object.txt, 0002-Expose-Header-for-use-in-DataFileReader-and-test-usage.txt
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AVRO-757) Allow a DataFile to be reopened
without reading the head
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990844#comment-12990844 ]
Doug Cutting commented on AVRO-757:
-----------------------------------
Overall this looks good. Some quick comments:
- the Handle fields should not be public, but private, perhaps with accessor methods.
- the Handle constructor should be private or package-private, not protected, since it should not appear in javadoc.
- can we call it a Header instead of a Handle, to be consistent with the spec?
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
> Attachments: 0001-Move-header-state-into-a-Handle-object.txt, 0002-Expose-Handle-for-use-in-DataFileReader-and-test-usage.txt
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-757) Allow a DataFile to be reopened without
reading the head
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated AVRO-757:
--------------------------
Attachment: (was: 0002-Expose-Handle-for-use-in-DataFileReader-and-test-usage.txt)
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AVRO-757) Allow a DataFile to be reopened
without reading the head
Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991595#comment-12991595 ]
Scott Carey commented on AVRO-757:
----------------------------------
+1, tests run and pass for me, code looks good.
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
> Attachments: 0001-Move-header-state-into-a-Header-object.txt, 0002-Expose-Header-for-use-in-DataFileReader-and-test-usage.txt, AVRO-757.patch
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-757) Allow a DataFile to be reopened without
reading the head
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated AVRO-757:
--------------------------
Attachment: (was: 0001-Move-header-state-into-a-Handle-object.txt)
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-757) Allow a DataFile to be reopened without
reading the head
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-757:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I committed this. Thanks, Stu!
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
> Attachments: 0001-Move-header-state-into-a-Header-object.txt, 0002-Expose-Header-for-use-in-DataFileReader-and-test-usage.txt, AVRO-757.patch
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AVRO-757) Allow a DataFile to be reopened
without reading the head
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990680#comment-12990680 ]
Doug Cutting commented on AVRO-757:
-----------------------------------
This sounds reasonable.
The specification includes a schema that corresponds to the file header.
http://avro.apache.org/docs/current/spec.html#Object+Container+Files
Perhaps we should implement this by adding the following methods to DataFileReader:
- public Header getHeader();
- public static FileReader<D> openReader(SeekableInput in, DatumReader<D>, Header);
The openReader() implementation should probably call sync(in.tell()), although there might be cases where you want it to instead just call seek(in.tell()). Perhaps there should be a boolean 'sync' parameter to that method that defaults to true?
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-757) Allow a DataFile to be reopened without
reading the head
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated AVRO-757:
--------------------------
Attachment: 0002-Expose-Handle-for-use-in-DataFileReader-and-test-usage.txt
0001-Move-header-state-into-a-Handle-object.txt
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
> Attachments: 0001-Move-header-state-into-a-Handle-object.txt, 0002-Expose-Handle-for-use-in-DataFileReader-and-test-usage.txt
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-757) Allow a DataFile to be reopened without
reading the head
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-757:
------------------------------
Attachment: AVRO-757.patch
Here's a new version of the patch with some mostly cosmetic changes:
- combines the two patch files into a single patch
- fixes a few places where changed lines exceeded 80-columns
- adds a javadoc comment to a protected method
- adds a transient declaration to 'metaKeyList' and a few 'private' declarations, where feasible.
- moved the Header definition to the front of the file
- fixed minor pre-existing whitespace and javadoc issue
Barring objections, I will commit this version soon.
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
> Attachments: 0001-Move-header-state-into-a-Header-object.txt, 0002-Expose-Header-for-use-in-DataFileReader-and-test-usage.txt, AVRO-757.patch
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-757) Allow a DataFile to be reopened without
reading the head
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated AVRO-757:
--------------------------
Release Note: Allow a DataFileReader to be reopened at an arbitrary position.
Status: Patch Available (was: Open)
> Allow a DataFile to be reopened without reading the head
> --------------------------------------------------------
>
> Key: AVRO-757
> URL: https://issues.apache.org/jira/browse/AVRO-757
> Project: Avro
> Issue Type: New Feature
> Components: java
> Reporter: Stu Hood
> Assignee: Stu Hood
> Labels: file
> Fix For: 1.5.0
>
> Attachments: 0001-Move-header-state-into-a-Handle-object.txt, 0002-Expose-Handle-for-use-in-DataFileReader-and-test-usage.txt
>
>
> It is useful to be able to re-open a file without needing to seek to the head of the stream to collect metadata and schema that you've read before.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira