You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Patrick Wendell (JIRA)" <ji...@apache.org> on 2010/08/05 00:22:18 UTC
[jira] Created: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Add File-Based Span Storage to TracePlugin
------------------------------------------
Key: AVRO-606
URL: https://issues.apache.org/jira/browse/AVRO-606
Project: Avro
Issue Type: Sub-task
Reporter: Patrick Wendell
Assignee: Patrick Wendell
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897386#action_12897386 ]
Philip Zeyliger commented on AVRO-606:
--------------------------------------
Looks good.
I don't see "writerEnabled" being used, except to set it to true once. Am I missing something? You probably need a volatile for it. I tend to use methods instead of fields for this sort of control (disableWriter() or what-not), so as to abstract away the implementation.
-- Philip
> Add File-Based Span Storage to TracePlugin
> ------------------------------------------
>
> Key: AVRO-606
> URL: https://issues.apache.org/jira/browse/AVRO-606
> Project: Avro
> Issue Type: Sub-task
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: AVRO-606.v1.txt, AVRO-606.v2.txt, AVRO-606.v3.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated AVRO-606:
---------------------------------
Attachment: AVRO-606.v1.txt
This is the full diff from origin/HEAD and includes outstanding code for AVRO-595.
The exact set of new files (but only a subset of all of the changes) are shown in this git hash:
http://github.com/pwendell/Avro/commit/c3127daa936a74db66b817ea97c6b9f5c1e24997
This patch is a preliminary file-based storage mechanism for Avro Trace.
> Add File-Based Span Storage to TracePlugin
> ------------------------------------------
>
> Key: AVRO-606
> URL: https://issues.apache.org/jira/browse/AVRO-606
> Project: Avro
> Issue Type: Sub-task
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: AVRO-606.v1.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895756#action_12895756 ]
Philip Zeyliger commented on AVRO-606:
--------------------------------------
Pressed "add" quite early.
Looks reasonable, for the most part. It feels a bit odd that the reader and writer of these log directories are the same class, still. That might be worth splitting up at some point. Instead of keeping a list of all previous files, you could also simply scan the directory and look around. Note that it's possible to add some metadata to AVRO data files when you open them, so you might store the open time in the header. (The close time would require a seek...). That way, you could retrieve spans across restarts.
> Add File-Based Span Storage to TracePlugin
> ------------------------------------------
>
> Key: AVRO-606
> URL: https://issues.apache.org/jira/browse/AVRO-606
> Project: Avro
> Issue Type: Sub-task
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: AVRO-606.v1.txt, AVRO-606.v2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated AVRO-606:
---------------------------------
Attachment: AVRO-606.v5.txt
Tweaked one test which was failing occasionally. Also, patched this on a clean checkout and tests/checksyle pass, FYI.
> Add File-Based Span Storage to TracePlugin
> ------------------------------------------
>
> Key: AVRO-606
> URL: https://issues.apache.org/jira/browse/AVRO-606
> Project: Avro
> Issue Type: Sub-task
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: AVRO-606.v1.txt, AVRO-606.v2.txt, AVRO-606.v3.txt, AVRO-606.v4.txt, AVRO-606.v5.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated AVRO-606:
---------------------------------
Attachment: AVRO-606.v2.txt
This is the same patch, but only the incremental diff from AVRO-595 which has been committed.
> Add File-Based Span Storage to TracePlugin
> ------------------------------------------
>
> Key: AVRO-606
> URL: https://issues.apache.org/jira/browse/AVRO-606
> Project: Avro
> Issue Type: Sub-task
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: AVRO-606.v1.txt, AVRO-606.v2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated AVRO-606:
---------------------------------
Attachment: AVRO-606.v4.txt
Yep I was actually in the process of deleting that variable but must have left a few references around. For now, we actually don't have any case where we need to explicitly disable the storage, so just keeping it out.
> Add File-Based Span Storage to TracePlugin
> ------------------------------------------
>
> Key: AVRO-606
> URL: https://issues.apache.org/jira/browse/AVRO-606
> Project: Avro
> Issue Type: Sub-task
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: AVRO-606.v1.txt, AVRO-606.v2.txt, AVRO-606.v3.txt, AVRO-606.v4.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Philip Zeyliger resolved AVRO-606.
----------------------------------
Resolution: Fixed
I just committed this as r985363. Thanks for your contribution, Patrick!
> Add File-Based Span Storage to TracePlugin
> ------------------------------------------
>
> Key: AVRO-606
> URL: https://issues.apache.org/jira/browse/AVRO-606
> Project: Avro
> Issue Type: Sub-task
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: AVRO-606.v1.txt, AVRO-606.v2.txt, AVRO-606.v3.txt, AVRO-606.v4.txt, AVRO-606.v5.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated AVRO-606:
---------------------------------
Attachment: AVRO-606.v3.txt
Hey Phil,
I addressed all of the small things.
For persistent storage - this gets very difficult to do if we don't enforce that there is only one TracePlugin running at a time. If multiple TracePlugin's are running at the same time, and they both are managing the files in that directory, we don't have any good way to coordinate the deletion of files. It turns into a consistency nightmare.
I have some code that does persistent storage in a local branch, but I'm going to punt it until we decide to make this into a Factory method, or figure out in more detail the context in which a TracePlugin is going to run.
- Patrick
> Add File-Based Span Storage to TracePlugin
> ------------------------------------------
>
> Key: AVRO-606
> URL: https://issues.apache.org/jira/browse/AVRO-606
> Project: Avro
> Issue Type: Sub-task
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: AVRO-606.v1.txt, AVRO-606.v2.txt, AVRO-606.v3.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896639#action_12896639 ]
Patrick Wendell commented on AVRO-606:
--------------------------------------
{quote}
this.spansSoFar -= spansPerFile.get(oldest);
Is this always the same as this.spansSoFar = 0?
{quote}
No. spansSoFar tracks spans stored in *all files* which is what the limit applies to. So this is just subtracting the spans of the most recently deleted file.
bq.Wouldn't you want to set spansSoFar to 0 if you're writing a new file?
Same answer.
{quote}
Boolean writerEnabled;
Why not primitive type?
{quote}
This is passed by reference to the thread so that the parent thread can signal the child to stop if it wishes. Is there a better way to do this?
{quote}
@param start UNIX time (in nanoseconds) as a long
Huh? I thought this was millis/1000, i.e., seconds and not nanos.
{quote}
I changed from milli to nanoseconds to support implementations which have better clock granularity. Currently I don't think any systems offer granularity better than microseconds (1*10^-6) but don't see any reason not to be more specific if there is no additional overhead.
> Add File-Based Span Storage to TracePlugin
> ------------------------------------------
>
> Key: AVRO-606
> URL: https://issues.apache.org/jira/browse/AVRO-606
> Project: Avro
> Issue Type: Sub-task
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: AVRO-606.v1.txt, AVRO-606.v2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-606) Add File-Based Span Storage to
TracePlugin
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895754#action_12895754 ]
Philip Zeyliger commented on AVRO-606:
--------------------------------------
bq. simply drop oldest file if
Nit: missing "the", and perhaps insert a "we" in there, to indicate that this is what the code does.
bq. private Long currentTimestamp
Why not "long"?
bq. private static int SECONDS_PER_FILE = 60 * 10; // ten minute chunks
Consider making this configurable.
bq. catch (InterruptedException e1) {
I think (http://www.ibm.com/developerworks/java/library/j-jtp05236.html) you're supposed to call Thread.currentThread().interrupt(); at this point. I kind of hate this part of Java.
You could also allow yourself to get interrupted, as a flow control mechanism.
Look around to see what other Hadoop worker threads do.
bq. } catch (IOException e) {
You should probably log the exception.
You should also probably catch all Throwables, and log those too. If you don't, your thread tends to die, and nobody tends to know where it went.
bq. this.spansSoFar -= spansPerFile.get(oldest);
Is this always the same as this.spansSoFar = 0?
bq. if (createNewFile) {
Wouldn't you want to set spansSoFar to 0 if you're writing a new file?
bq. + private static String TRACE_FILE_DIR = "/tmp";
This stuff ought to be configurable.
bq. Thread writer;
Why not private?
bq. + Boolean writerEnabled;
Why not primitive type?
bq. private static void addFileSpans
this feels like "readFileSpans()" more so than "add".
bq. private static void addFileSpans
You could do binary search here, but for now it's probably fine to just indicate that it's linear.
bq. public static void setFileGranularityForTesting(int granularity) {
Package-private (default) is probably good enough here.
bq. protected void finalize() {
Finalize is almost always a bad idea; why is this here?
bq. // TODO Auto-generated method stub
Throw UnsupportedOperationException() here.
bq. * @param start UNIX time (in nanoseconds) as a long
Huh? I thought this was millis/1000, i.e., seconds and not nanos.
> Add File-Based Span Storage to TracePlugin
> ------------------------------------------
>
> Key: AVRO-606
> URL: https://issues.apache.org/jira/browse/AVRO-606
> Project: Avro
> Issue Type: Sub-task
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: AVRO-606.v1.txt, AVRO-606.v2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.