You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@avro.apache.org by su...@apache.org on 2017/05/31 15:48:44 UTC
svn commit: r1797063 [2/5] - in /avro/site/publish/docs/1.8.2: ./ examples/
examples/java-example/ examples/java-example/src/
examples/java-example/src/main/ examples/java-example/src/main/java/
examples/java-example/src/main/java/example/ examples/mr-...
Propchange: avro/site/publish/docs/1.8.2/gettingstartedpython.pdf
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: avro/site/publish/docs/1.8.2/htmldocs/canonical-completeness.html
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/htmldocs/canonical-completeness.html?rev=1797063&view=auto
==============================================================================
--- avro/site/publish/docs/1.8.2/htmldocs/canonical-completeness.html (added)
+++ avro/site/publish/docs/1.8.2/htmldocs/canonical-completeness.html Wed May 31 15:48:43 2017
@@ -0,0 +1,204 @@
+<html>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+<title>Completeness of "Parsing Canonical Form"</title>
+</head>
+<body>
+
+<center><h1>Completeness of "Parsing Canonical Form"</h1></center>
+
+<h2>1.0 Introduction</h2>
+
+<p>One of the defining characteristics of Avro is that a reader is assumed to have the "same" schema used by the writer of the data the reader is reading. This assumption leads to a data format that's compact and amenable to many forms of schema evolution. However, there are nuances to defining exactly what it means for the reader to have "the same" schema used by the writer. We want to allow, for example, trivial transformations, such as the insertion of whitespace. But we can't allow transformations that change the real meaning of schemas, such as a reordering of fields in a record</p>
+
+<p>To clearly define what it means for a reader to have "the same" schema as a writer, the Avro specification defines <dfn>Parsing Canonical Form</dfn> (PCF), a set of transformations on Avro schemas that strip away irrelevencies (e.g., "doc" attributes) and normalize the JSON text (e.g., dealing with whitespace). Two schemas are defined to be "the same" as far as a reader is concerned if and only if their PCFs are textually equal.</p>
+
+<p>We believe that PCF is <em>sound</em> and <em>complete</em>. Soundness means that the PCF of a schema is logically equivalent to the original form, i.e., we can use the PCF in place of the original form without introducing bugs. Completeness is "maximal soundness:" if two schemas are logically equivalent, then their PFCs will be textually identical. The Avro specification claims that PCF is complete when it says: "[if two schemas have the same PCF, then] there is no serialized data that would allow a reader to distinguish data generated by a writer using one of the original schemas from data generated by a writing using the other original schema."</p>
+
+<p>We believe that the transformations that define PCF are "self-evidently" sound to people familiar with Avro. For example, fixing the order of fields in a JSON object, or eliminating irrelevant attributes like <code>doc</code>, or using the simple <code>int</code> in place of <code>{"type":"int"}</code> clearly don't change the meaning of a schema.</p>
+
+<p>Completeness, on the other hand, is much less obvious. How do we know that there aren't two logically equivalent schemas that happen to reduce to different canonical forms? All it takes is one such pair to foil our claim of completeness.</p>
+
+<p>In general, completeness properties like this can be tricky to prove. It turns out that, while soundness is critical to us, completeness is not. If two schemas are operationally equivalent (i.e., a reader can't tell their output apart), but we accidentally treat them as if they are different, then typically all that happens is that we'll do more work. For example, we might generate a decoder object to decode some incoming data when it turns out that we had already cached a decoder object that could do the job. This is not likely to happen often, and thus incompleteness isn't a huge problem.</p>
+
+<p>At the same time, if we knew that our canonical forms were complete, then we might take advantage of that fact in some circumstances (e.g., to serialize schemas). Also, the <code>Schema.equals(Object)</code> method provided in the Avro implementation makes many of the same assumptions made in the PCF definition. Thus, a completeness proof for our canonicalization would give us confidence in the correctness of this equality algorithm. So this issue is not entirely academic.</p>
+
+<p>We haven't worked out a full, formal proof (we hope someone from the community will step up to that task!). However, we've been thinking about it quite a bit, and we thought we'd share our thoughts so far.</p>
+
+
+<h2>2.0 Completeness argument for Parsing Canonical Form</h2>
+
+<p>Our formalization of Avro schemas would be based on interpreting them as grammars. In this interpretation, Avro schemas are grammars that generate tagged data streams. Consider, for example, the following schema for a linked-list:
+<pre>
+ {"type":"record", "name":"list", "fields":[
+ {"name":"value", "type":"int"},
+ {"name":"tail", "type":["null", "list"]}
+ ]}
+</pre>
+Interpreted as a grammar, it can generate a tagged data-stream that looks like this:
+<pre>
+ [record,"list"][field,"value"][int,10][field,"tail"][union,1]
+ [record,"list"][field,"value"][int,22][field,"tail"][union,0]
+</pre>
+(this is a two-record linked list whose first cell contains the value "10" and second cell the value "22"). Avro schemas can trivially be interpreted as grammars for such tagged data streams. Formal proofs involving Avro schemas can be carried out as proofs about languages and grammars.</p>
+
+<p>So what does it mean for the canonical form of a schema to be "complete?" Let <i>L(S)</i> denote the language generated by the Avro schema <code>S</code>, and <i>C(S)</i> denote the canonical form of the schema. The canonicalization is complete if:
+<blockquote>
+For all schemas <i>S<sub>1</sub></i> and <i>S<sub>2</sub></i>,<br>
+ <i>L(S<sub>1</sub>) = L(S<sub>2</sub>) ⇒ C(S<sub>1</sub>) = C(S<sub>2</sub>)</i>
+</blockquote>
+That is, for any two schemas that generate the same language, their canonicalizations are textually equivalent.
+
+<p>To prove this, we need to define some functions:
+<blockquote>
+<i>J</i> is a variable name we often use to denote a JSON expression representing an Avro schema<br>
+<i>C(J)</i> is the Parsing Canonical Form of <i>J</i> as defined in the Avro specification<br>
+<i>P(J)</i> is the ASG for an Avro schema generated by parsing <i>J</i> (think of <i>P(J)</i> as a <code>Schema</code> Java object)<br>
+<i>S</i> is a variable name we often use to denote such ASGs<br>
+<i>L(S)</i> is the language generated by a schema ASG
+</blockquote>
+<p>With all these symbols defined, our completeness criteria is now rendered as:
+<blockquote>
+∀ <i>J<sub>1</sub></i>, <i>J<sub>2</sub></i>:
+<i>L(P(J<sub>1</sub>)) = L(P(J<sub>2</sub>)) ⇒ C(J<sub>1</sub>) = C(J<sub>2</sub>)</i>
+</blockquote>
+We'll prove this by breaking it into two parts:
+<blockquote>
+(1): ∀ <i>S<sub>1</sub></i>, <i>S<sub>2</sub></i>:
+<i>L(S<sub>1</sub>) = L(S<sub>2</sub>) ⇒ S<sub>1</sub> ≅ S<sub>2</sub> <br>
+(2): ∀ <i>J<sub>1</sub></i>, <i>J<sub>2</sub></i>:
+<i>P(J<sub>1</sub>) ≅ P(J<sub>2</sub>) ⇒ C(J<sub>1</sub>) = C(J<sub>2</sub>)</i>
+</i>
+</blockquote>
+In this two-step decomposition, we've introduced a new operator ≅, which compares the ASGs of two Avro schemas. The ASG of an Avro schema can be viewed as a rooted, labeled, directed graph. Because Avro schemas can be recursive, these graphs can be cyclic. The ≅ operator is "true" between two ASGs when the set of minimal labeled paths (no cycles, starting from the root) on the two ASGs are the same. (The <code>Schema.equals(Object)</code> method in the Avro implementation computes something close to this ≅ relation, except that ≅ ignores "irrelevant" attributes like <code>doc</code> and <code>aliases</code>.)
+
+<p>It turns out that, implicit in the Avro Specification, there are "canonicalization" rules that are important to our proof of completeness. In particular, the Avro Specification says that a name must be defined "before" it is used, and that a name cannot be defined more than once in a schema. Consider the following redefinition of the linked-list schema, for example:
+<pre>
+ {"type":"record", "name":"list", "fields":[
+ {"name":"value", "type":"int"},
+ {"name":"tail",
+ "type":["null", {"type":"record", "name":"list", "fields":[
+ {"name":"value", "type":"int"},
+ {"name":"tail", "type":["null", "list"]}]}]}
+ ]}
+</pre>
+In this redefinition, we've "unpacked" the recursion in the linked list by one level. In some sense, this is a perfectly fine definition of a linked list, and is operationally equivalent to the more compact version given earlier. So it makes sense that our claim of completeness is dependent upon this kind of "unpacking" not occuring in real schemas.</p>
+
+<p>To deal with this issue in our proof, we pretend that the Avro specification does <em>not</em> require that named schemas be defined just once, and be defined "before" they are used. Rather, we treat this requirement as an additional transformation rule in the definition of Parsing Canonical Form:
+<ul>
+ <li> [MINIMIZE] Eliminate redundant definitions of named types (records, enums, and fixeds). That is, for each named type, have a defining instance that appears at first use, and then use just the name (rather than the full schema) everywhere else.</li>
+</ul>
+(As in the Avro spec, "first use" is defined as the first occurrence in a depth-first, left-to-right traversal of the schema abstract-syntax graph (ASG).)
+
+<p>Getting back to the proof of (1) and (2) from above, we need to introduce more functions:
+<blockquote>
+<i>P(J)=P<sub>A</sub>(P<sub>J</sub>(J))</i> - decompose parser into: <br>
+ <i>P<sub>J</sub></i> is the JSON parser<br>
+ <i>P<sub>A</sub></i> is the Avro parser (takes JSON ASTs as input)<br>
+<i>C(J)=C<sub>J</sub>(C<sub>A</sub>(C<sub>M</sub>(J)))</i> - decompose canonicalization into:<br>
+ <i>C<sub>M</sub>(J)</i> the MINIMIZE step<br>
+ <i>C<sub>A</sub>(J)</i> Avro normalizations<br>
+ <i>C<sub>J</sub>(J)</i> JSON normalizations<br>
+<i>M(S)</i> is the "named-schema NFA minimzation" of <i>S</i><br>
+</blockquote>
+"Named-schema NFA minimization" is similar to general NFA minimization, except that we only collapse nodes and edges related to named schema entities and not other nodes. For example, we would <em>not</em> collapse the nodes associated with <code>int</code> or <code>union</code> schemas.
+
+<p> Our proof of (1) looks like this (this proof refers to lemmas (3) and (4), which are defined later):
+<blockquote>
+<table>
+<tr><td>∀<i>S<sub>1</sub>,S<sub>2</sub></i>:</td><td><i>L(S<sub>1</sub>)=L(S<sub>2</sub>)</i></td><td></td></tr>
+<tr>
+<td></td><td>⇒<i>M(S<sub>1</sub>)=M(S<sub>2</sub>)</i></td>
+<td>by (3)</td>
+</tr>
+<tr>
+<td></td><td>⇒<i>S<sub>1</sub>≅S<sub>2</sub)</i></td>
+<td>by (4)</td>
+</tr>
+</table>
+</blockquote>
+Here's the proof of (2) (this proof refers to lemmas (4)-(7), which are defined later):
+<blockquote>
+<table>
+<tr><td>∀<i>J<sub>1</sub>,J<sub>2</sub></i>:</td><td><i>P(J<sub>1</sub>)≅P(J<sub>2</sub>)</i></td><td></td></tr>
+
+<tr>
+<td></td><td>⇒<i>M(P(J<sub>1</sub>))=M(P(J<sub>2</sub>))</i></td>
+<td>by (4)</td>
+</tr>
+
+<tr>
+<td></td><td>⇒<i>P(C<sub>M</sub>(J<sub>1</sub>))=P(C<sub>M</sub>(J<sub>2</sub>))</i></td>
+<td>by (5)</td>
+</tr>
+
+<tr>
+<td></td><td>⇒<i>P<sub>A</sub>(P<sub>J</sub>(C<sub>M</sub>(J<sub>1</sub>)))=P<sub>A</sub>(P<sub>J</sub>(C<sub>M</sub>(J<sub>2</sub>)))</i></td>
+<td>by definition of <i>P</i></td>
+</tr>
+
+<tr>
+<td></td><td>⇒<i>P<sub>J</sub>(C<sub>A</sub>(C<sub>M</sub>(J<sub>1</sub>)))=P<sub>J</sub>(C<sub>A</sub>(C<sub>M</sub>(J<sub>2</sub>)))</i></td>
+<td>by (6)</td>
+</tr>
+
+<tr>
+<td></td><td>⇒<i>C<sub>J</sub>(C<sub>A</sub>(C<sub>M</sub>(J<sub>1</sub>)))=C<sub>J</sub>(C<sub>A</sub>(C<sub>M</sub>(J<sub>2</sub>)))</i></td>
+<td>by (7)</td>
+</tr>
+
+<tr>
+<td></td><td>⇒<i>C(J<sub>1</sub>)=C(J<sub>2</sub>)</i></td>
+<td>by definition of <i>C</i></td>
+</tr>
+</table>
+</blockquote>
+
+Here are the lemmas needed above:
+<blockquote>
+(3): ∀ <i>S<sub>1</sub></i>, <i>S<sub>2</sub></i>:
+<i>L(S<sub>1</sub>) = L(S<sub>2</sub>) ⇒ M(S<sub>1</sub>) = M(S<sub>2</sub>)</i><br>
+
+(4): ∀ <i>S<sub>1</sub></i>, <i>S<sub>2</sub></i>:
+<i>M(S<sub>1</sub>) = M(S<sub>2</sub>) ⇔ S<sub>1</sub> ≅ S<sub>2</sub></i> <br>
+
+(5): ∀ <i>J</i>: <i>M(P(J)) = P(C<sub>M</sub>(J))</i><br>
+
+(6): ∀ <i>J<sub>1</sub></i>, <i>J<sub>2</sub></i>:
+<i>P<sub>A</sub>(P<sub>J</sub>(J<sub>1</sub>)) = P<sub>A</sub>(P<sub>J</sub>(J<sub>2</sub>)) ⇒ P<sub>J</sub>(C<sub>A</sub>(J<sub>1</sub>)) = P<sub>J</sub>(C<sub>A</sub>(J<sub>2</sub>))</i> <br>
+
+(7): ∀ <i>J<sub>1</sub></i>, <i>J<sub>2</sub></i>:
+<i>P<sub>J</sub>(J<sub>1</sub>) = P<sub>J</sub>(J<sub>2</sub>) ⇒ C<sub>J</sub>(J<sub>1</sub>) = C<sub>J</sub>(J<sub>2</sub>)</i> <br>
+</blockquote>
+
+<p>Proving the lemmas:
+<ol start="3">
+<li> This says that the language-related part of our canonicalization is complete, i.e., <i>M</i> finds the equivalence-classes of <i>L</i>. I would imagine one could prove this by modifying a proof that the equality of LL(1) grammars is a decidable problem. I haven't gotten very far in showing this, however.
+<li> The right-hand direction of this follows from the definition of minimization. The left-hand direction seems correct, but I'm not sure how to prove it (I think it also follows from the definition of minimization).
+<li> This is showing that the MINIMIZE step (which is done on JSON expressions) is equivalent to doing an named-schema NFA minimization on the ASG representation. This should follow pretty directly from a detailed definition of <i>M</i>, if we provided one.
+<li> This says that the Avro-related part of our canonicalization is complete, i.e., that <i>C<sub>A</sub></i> finds equivalence-classes of <i>P<sub>A</sub></i>.
+<li> This says that the JSON-related part of our canonicalization is complete, i.e., that <i>C<sub>J</sub></i> finds equivalence-classes of <i>P<sub>J</sub></i>. Note that, implicitly, this lemma ranges over only JSON expressions that are legal Avro schemas with no doc strings or default values, and thus (for example) doesn't need to worry about normalization of floating-point literals.
+</ol>
+
+
+<h2>3.0 Concluding remarks</h2>
+
+Engineers <a href="http://www.aps.org/publications/apsnews/201002/physicshistory.cfm">have a history</a> of running ahead of formal mathematical proofs, when things "seem correct" to them. In this case, it seems pretty obvious that Parsing Canonical Form is complete as well as sound, and we should go ahead and treat it as such. At the same time, formal proofs often turn up corner cases and exceptions that are valuable to document and account for. Thus, it'd nice if someone could provide a better completeness argument than we've been able to so far.
+
+</body>
+</html>
Added: avro/site/publish/docs/1.8.2/idl.html
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/idl.html?rev=1797063&view=auto
==============================================================================
--- avro/site/publish/docs/1.8.2/idl.html (added)
+++ avro/site/publish/docs/1.8.2/idl.html Wed May 31 15:48:43 2017
@@ -0,0 +1,744 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+<meta content="Apache Forrest" name="Generator">
+<meta name="Forrest-version" content="0.9">
+<meta name="Forrest-skin-name" content="pelt">
+<title>Apache Avro™ 1.8.2
+ IDL</title>
+<link type="text/css" href="skin/basic.css" rel="stylesheet">
+<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
+<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
+<link type="text/css" href="skin/profile.css" rel="stylesheet">
+<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
+<link rel="shortcut icon" href="images/favicon.ico">
+</head>
+<body onload="init()">
+<script type="text/javascript">ndeSetTextSize();</script>
+<div id="top">
+<!--+
+ |breadtrail
+ +-->
+<div class="breadtrail">
+<a href="http://www.apache.org/">Apache</a> > <a href="http://avro.apache.org/">Avro</a> > <a href="http://avro.apache.org/">Avro</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
+</div>
+<!--+
+ |header
+ +-->
+<div class="header">
+<!--+
+ |start group logo
+ +-->
+<div class="grouplogo">
+<a href="http://www.apache.org/"><img class="logoImage" alt="Apache" src="images/apache_feather.gif" title="The Apache Software Foundation"></a>
+</div>
+<!--+
+ |end group logo
+ +-->
+<!--+
+ |start Project Logo
+ +-->
+<div class="projectlogo">
+<a href="http://avro.apache.org/"><img class="logoImage" alt="Avro" src="images/avro-logo.png" title="Serialization System"></a>
+</div>
+<!--+
+ |end Project Logo
+ +-->
+<!--+
+ |start Search
+ +-->
+<div class="searchbox">
+<form action="http://www.google.com/search" method="get" class="roundtopsmall">
+<input value="avro.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">
+ <input name="Search" value="Search" type="submit">
+</form>
+</div>
+<!--+
+ |end search
+ +-->
+<!--+
+ |start Tabs
+ +-->
+<ul id="tabs">
+<li>
+<a class="unselected" href="http://avro.apache.org/">Project</a>
+</li>
+<li>
+<a class="unselected" href="http://wiki.apache.org/hadoop/Avro/">Wiki</a>
+</li>
+<li class="current">
+<a class="selected" href="index.html">Avro 1.8.2 Documentation</a>
+</li>
+</ul>
+<!--+
+ |end Tabs
+ +-->
+</div>
+</div>
+<div id="main">
+<div id="publishedStrip">
+<!--+
+ |start Subtabs
+ +-->
+<div id="level2tabs"></div>
+<!--+
+ |end Endtabs
+ +-->
+<script type="text/javascript"><!--
+document.write("Last Published: " + document.lastModified);
+// --></script>
+</div>
+<!--+
+ |breadtrail
+ +-->
+<div class="breadtrail">
+
+
+ </div>
+<!--+
+ |start Menu, mainarea
+ +-->
+<!--+
+ |start Menu
+ +-->
+<div id="menu">
+<div onclick="SwitchMenu('menu_selected_1.1', 'skin/')" id="menu_selected_1.1Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Documentation</div>
+<div id="menu_selected_1.1" class="selectedmenuitemgroup" style="display: block;">
+<div class="menuitem">
+<a href="index.html">Overview</a>
+</div>
+<div class="menuitem">
+<a href="gettingstartedjava.html">Getting started (Java)</a>
+</div>
+<div class="menuitem">
+<a href="gettingstartedpython.html">Getting started (Python)</a>
+</div>
+<div class="menuitem">
+<a href="spec.html">Specification</a>
+</div>
+<div class="menuitem">
+<a href="trevni/spec.html">Trevni</a>
+</div>
+<div class="menuitem">
+<a href="api/java/index.html">Java API</a>
+</div>
+<div class="menuitem">
+<a href="api/c/index.html">C API</a>
+</div>
+<div class="menuitem">
+<a href="api/cpp/html/index.html">C++ API</a>
+</div>
+<div class="menuitem">
+<a href="api/csharp/index.html">C# API</a>
+</div>
+<div class="menuitem">
+<a href="mr.html">MapReduce guide</a>
+</div>
+<div class="menupage">
+<div class="menupagetitle">IDL language</div>
+</div>
+<div class="menuitem">
+<a href="sasl.html">SASL profile</a>
+</div>
+<div class="menuitem">
+<a href="http://wiki.apache.org/hadoop/Avro/">Wiki</a>
+</div>
+<div class="menuitem">
+<a href="http://wiki.apache.org/hadoop/Avro/FAQ">FAQ</a>
+</div>
+</div>
+<div id="credit"></div>
+<div id="roundbottom">
+<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
+<!--+
+ |alternative credits
+ +-->
+<div id="credit2"></div>
+</div>
+<!--+
+ |end Menu
+ +-->
+<!--+
+ |start content
+ +-->
+<div id="content">
+<div title="Portable Document Format" class="pdflink">
+<a class="dida" href="idl.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
+ PDF</a>
+</div>
+<h1>Apache Avro™ 1.8.2
+ IDL</h1>
+<div id="front-matter">
+<div id="minitoc-area">
+<ul class="minitoc">
+<li>
+<a href="#preamble">Introduction</a>
+</li>
+<li>
+<a href="#overview">Overview</a>
+<ul class="minitoc">
+<li>
+<a href="#overview_purpose">Purpose</a>
+</li>
+<li>
+<a href="#overview_usage">Usage</a>
+</li>
+</ul>
+</li>
+<li>
+<a href="#defining_protocol">Defining a Protocol in Avro IDL</a>
+</li>
+<li>
+<a href="#imports">Imports</a>
+</li>
+<li>
+<a href="#format_enums">Defining an Enumeration</a>
+</li>
+<li>
+<a href="#format_fixed">Defining a Fixed Length Field</a>
+</li>
+<li>
+<a href="#format_records">Defining Records and Errors</a>
+<ul class="minitoc">
+<li>
+<a href="#primitive_types">Primitive Types</a>
+</li>
+<li>
+<a href="#logical_types">Logical Types</a>
+</li>
+<li>
+<a href="#schema_references">References to Named Schemata</a>
+</li>
+<li>
+<a href="#default_values">Default Values</a>
+</li>
+<li>
+<a href="#complex_types">Complex Types</a>
+<ul class="minitoc">
+<li>
+<a href="#arrays">Arrays</a>
+</li>
+<li>
+<a href="#maps">Maps</a>
+</li>
+<li>
+<a href="#unions">Unions</a>
+</li>
+</ul>
+</li>
+</ul>
+</li>
+<li>
+<a href="#define_messages">Defining RPC Messages</a>
+</li>
+<li>
+<a href="#minutiae">Other Language Features</a>
+<ul class="minitoc">
+<li>
+<a href="#minutiae_comments">Comments</a>
+</li>
+<li>
+<a href="#minutiae_escaping">Escaping Identifiers</a>
+</li>
+<li>
+<a href="#minutiae_annotations">Annotations for Ordering and Namespaces</a>
+</li>
+</ul>
+</li>
+<li>
+<a href="#example">Complete Example</a>
+</li>
+</ul>
+</div>
+</div>
+
+
+<a name="preamble"></a>
+<h2 class="h3">Introduction</h2>
+<div class="section">
+<p>This document defines Avro IDL, a higher-level language for authoring Avro schemata.
+ Before reading this document, you should have familiarity with the concepts of schemata and protocols,
+ as well as the various primitive and complex types available in Avro.
+ </p>
+</div>
+
+
+<a name="overview"></a>
+<h2 class="h3">Overview</h2>
+<div class="section">
+<a name="overview_purpose"></a>
+<h3 class="h4">Purpose</h3>
+<p>The aim of the Avro IDL language is to enable developers to author schemata in a way that
+ feels more similar to common programming languages like Java, C++, or Python. Additionally,
+ the Avro IDL language may feel more familiar for those users who have previously used the
+ interface description languages (IDLs) in other frameworks like Thrift, Protocol Buffers, or CORBA.
+ </p>
+<a name="overview_usage"></a>
+<h3 class="h4">Usage</h3>
+<p>
+ Each Avro IDL file defines a single Avro Protocol, and thus generates as its output a JSON-format
+ Avro Protocol file with extension <span class="codefrag">.avpr</span>.
+ </p>
+<p>
+ To convert a <span class="codefrag">.avdl</span> file into a <span class="codefrag">.avpr</span> file, it may be processed by the
+ <span class="codefrag">idl</span> tool. For example:
+ </p>
+<pre class="code">
+$ java -jar avroj-tools.jar idl src/test/idl/input/namespaces.avdl /tmp/namespaces.avpr
+$ head /tmp/namespaces.avpr
+{
+ "protocol" : "TestNamespace",
+ "namespace" : "avro.test.protocol",
+ </pre>
+<p>
+ The <span class="codefrag">idl</span> tool can also process input to and from <em>stdin</em> and <em>stdout</em>.
+ See <span class="codefrag">idl --help</span> for full usage information.
+ </p>
+<p>A Maven plugin is also provided to compile .avdl files. To
+ use it, add something like the following to your pom.xml:</p>
+<pre class="code">
+<build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <goals>
+ <goal>idl-protocol</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+</build>
+</pre>
+</div> <!-- end overview -->
+
+
+<a name="defining_protocol"></a>
+<h2 class="h3">Defining a Protocol in Avro IDL</h2>
+<div class="section">
+<p>An Avro IDL file consists of exactly one protocol definition. The minimal protocol is defined
+ by the following code:
+ </p>
+<pre class="code">
+protocol MyProtocol {
+}
+ </pre>
+<p>
+ This is equivalent to (and generates) the following JSON protocol definition:
+ </p>
+<pre class="code">
+{
+"protocol" : "MyProtocol",
+ "types" : [ ],
+ "messages" : {
+ }
+}
+ </pre>
+<p>
+ The namespace of the protocol may be changed using the <span class="codefrag">@namespace</span> annotation:
+ </p>
+<pre class="code">
+@namespace("mynamespace")
+protocol MyProtocol {
+}
+ </pre>
+<p>
+ This notation is used throughout Avro IDL as a way of specifying properties for the annotated element,
+ as will be described later in this document.
+ </p>
+<p>
+ Protocols in Avro IDL can contain the following items:
+ </p>
+<ul>
+
+<li>Imports of external protocol and schema files.</li>
+
+<li>Definitions of named schemata, including <em>record</em>s, <em>error</em>s, <em>enum</em>s, and <em>fixed</em>s.</li>
+
+<li>Definitions of RPC messages</li>
+
+</ul>
+</div>
+
+<a name="imports"></a>
+<h2 class="h3">Imports</h2>
+<div class="section">
+<p>Files may be imported in one of three formats: </p>
+<ul>
+
+<li>An IDL file may be imported with a statement like:
+ <pre class="code">import idl "foo.avdl";</pre>
+
+</li>
+
+<li>A JSON protocol file may be imported with a statement like:
+ <pre class="code">import protocol "foo.avpr";</pre>
+
+</li>
+
+<li>A JSON schema file may be imported with a statement like:
+ <pre class="code">import schema "foo.avsc";</pre>
+
+</li>
+
+</ul>
+<p>Messages and types in the imported file are added to this
+ file's protocol.</p>
+<p>Imported file names are resolved relative to the current IDL file.</p>
+</div>
+
+<a name="format_enums"></a>
+<h2 class="h3">Defining an Enumeration</h2>
+<div class="section">
+<p>
+ Enums are defined in Avro IDL using a syntax similar to C or Java:
+ </p>
+<pre class="code">
+enum Suit {
+ SPADES, DIAMONDS, CLUBS, HEARTS
+}
+ </pre>
+<p>
+ Note that, unlike the JSON format, anonymous enums cannot be defined.
+ </p>
+</div>
+
+<a name="format_fixed"></a>
+<h2 class="h3">Defining a Fixed Length Field</h2>
+<div class="section">
+<p>
+ Fixed fields are defined using the following syntax:
+ </p>
+<pre class="code">
+fixed MD5(16);
+ </pre>
+<p>This example defines a fixed-length type called <span class="codefrag">MD5</span> which contains 16 bytes.</p>
+</div>
+
+
+<a name="format_records"></a>
+<h2 class="h3">Defining Records and Errors</h2>
+<div class="section">
+<p>
+ Records are defined in Avro IDL using a syntax similar to a <span class="codefrag">struct</span> definition in C:
+ </p>
+<pre class="code">
+record Employee {
+ string name;
+ boolean active = true;
+ long salary;
+}
+ </pre>
+<p>
+ The above example defines a record with the name “Employee” with three fields.
+ </p>
+<p>
+ To define an error, simply use the keyword <span class="codefrag">error</span> instead of <span class="codefrag">record</span>.
+ For example:
+ </p>
+<pre class="code">
+error Kaboom {
+ string explanation;
+ int result_code = -1;
+}
+ </pre>
+<p>
+ Each field in a record or error consists of a type and a name,
+ optional property annotations and an optional default value.
+ </p>
+<p>A type reference in Avro IDL must be one of:</p>
+<ul>
+
+<li>A primitive type</li>
+
+<li>A logical type</li>
+
+<li>A named schema defined prior to this usage in the same Protocol</li>
+
+<li>A complex type (array, map, or union)</li>
+
+</ul>
+<a name="primitive_types"></a>
+<h3 class="h4">Primitive Types</h3>
+<p>The primitive types supported by Avro IDL are the same as those supported by Avro's JSON format.
+ This list includes <span class="codefrag">int</span>, <span class="codefrag">long</span>, <span class="codefrag">string</span>, <span class="codefrag">boolean</span>,
+ <span class="codefrag">float</span>, <span class="codefrag">double</span>, <span class="codefrag">null</span>, and <span class="codefrag">bytes</span>.
+ </p>
+<a name="logical_types"></a>
+<h3 class="h4">Logical Types</h3>
+<p>Some of the logical types supported by Avro's JSON format are also supported by Avro IDL.
+ The currently supported types are:
+ </p>
+<ul>
+
+<li>
+<span class="codefrag">decimal</span> (logical type <a href="spec.html#Decimal"><span class="codefrag">decimal</span></a>)</li>
+
+<li>
+<span class="codefrag">date</span> (logical type <a href="spec.html#Date"><span class="codefrag">date</span></a>)</li>
+
+<li>
+<span class="codefrag">time_ms</span> (logical type <a href="spec.html#Time+%28millisecond+precision%29"><span class="codefrag">time-millis</span></a>)</li>
+
+<li>
+<span class="codefrag">timestamp_ms</span> (logical type <a href="spec.html#Timestamp+%28millisecond+precision%29"><span class="codefrag">timestamp-millis</span></a>)</li>
+
+</ul>
+<p>For example:</p>
+<pre class="code">
+record Job {
+ string jobid;
+ date submitDate;
+ time_ms submitTime;
+ timestamp_ms finishTime;
+ decimal(9,2) finishRatio;
+}
+ </pre>
+<a name="schema_references"></a>
+<h3 class="h4">References to Named Schemata</h3>
+<p>If a named schema has already been defined in the same Avro IDL file, it may be referenced by name
+ as if it were a primitive type:
+ </p>
+<pre class="code">
+record Card {
+ Suit suit; // refers to the enum Card defined above
+ int number;
+}
+ </pre>
+<a name="default_values"></a>
+<h3 class="h4">Default Values</h3>
+<p>Default values for fields may be optionally
+ specified by using an equals sign after the field name
+ followed by a JSON expression indicating the default value.
+ This JSON is interpreted as described in
+ the <a href="spec.html#schema_record">spec</a>.</p>
+<a name="complex_types"></a>
+<h3 class="h4">Complex Types</h3>
+<a name="arrays"></a>
+<h4>Arrays</h4>
+<p>
+ Array types are written in a manner that will seem familiar to C++ or Java programmers. An array of
+ any type <span class="codefrag">t</span> is denoted <span class="codefrag">array<t></span>. For example, an array of strings is
+ denoted <span class="codefrag">array<string></span>, and a multidimensional array of <span class="codefrag">Foo</span> records
+ would be <span class="codefrag">array<array<Foo>></span>.
+ </p>
+<a name="maps"></a>
+<h4>Maps</h4>
+<p>Map types are written similarly to array types. An array that contains values of type
+ <span class="codefrag">t</span> is written <span class="codefrag">map<t></span>. As in the JSON schema format, all
+ maps contain <span class="codefrag">string</span>-type keys.</p>
+<a name="unions"></a>
+<h4>Unions</h4>
+<p>Union types are denoted as <span class="codefrag">union { typeA, typeB, typeC, ... }</span>. For example,
+ this record contains a string field that is optional (unioned with <span class="codefrag">null</span>):
+ </p>
+<pre class="code">
+record RecordWithUnion {
+ union { null, string } optionalString;
+}
+ </pre>
+<p>
+ Note that the same restrictions apply to Avro IDL unions as apply to unions defined in the
+ JSON format; namely, a record may not contain multiple elements of the same type.
+ </p>
+</div> <!-- how to define records -->
+
+<a name="define_messages"></a>
+<h2 class="h3">Defining RPC Messages</h2>
+<div class="section">
+<p>The syntax to define an RPC message within a Avro IDL protocol is similar to the syntax for
+ a method declaration within a C header file or a Java interface. To define an RPC message
+ <span class="codefrag">add</span> which takes two arguments named <span class="codefrag">foo</span> and <span class="codefrag">bar</span>,
+ returning an <span class="codefrag">int</span>, simply include the following definition within the protocol:
+ </p>
+<pre class="code">
+int add(int foo, int bar = 0);
+ </pre>
+<p>Message arguments, like record fields, may specify default
+ values.</p>
+<p>To define a message with no response, you may use the alias <span class="codefrag">void</span>, equivalent
+ to the Avro <span class="codefrag">null</span> type:
+ </p>
+<pre class="code">
+void logMessage(string message);
+ </pre>
+<p>
+ If you have previously defined an error type within the same protocol, you may declare that
+ a message can throw this error using the syntax:
+ </p>
+<pre class="code">
+void goKaboom() throws Kaboom;
+ </pre>
+<p>To define a one-way message, use the
+ keyword <span class="codefrag">oneway</span> after the parameter list, for example:
+ </p>
+<pre class="code">
+void fireAndForget(string message) oneway;
+ </pre>
+</div> <!-- define messages -->
+
+<a name="minutiae"></a>
+<h2 class="h3">Other Language Features</h2>
+<div class="section">
+<a name="minutiae_comments"></a>
+<h3 class="h4">Comments</h3>
+<p>All Java-style comments are supported within a Avro IDL file. Any text following
+ <span class="codefrag">//</span> on a line is ignored, as is any text between <span class="codefrag">/*</span> and
+ <span class="codefrag">*/</span>, possibly spanning multiple lines.</p>
+<p>Comments that begin with <span class="codefrag">/**</span> are used as the
+ documentation string for the type or field definition that
+ follows the comment.</p>
+<a name="minutiae_escaping"></a>
+<h3 class="h4">Escaping Identifiers</h3>
+<p>Occasionally, one will need to use a reserved language keyword as an identifier. In order
+ to do so, backticks (<span class="codefrag">`</span>) may be used to escape the identifier. For example, to define
+ a message with the literal name <em>error</em>, you may write:
+ </p>
+<pre class="code">
+void `error`();
+ </pre>
+<p>This syntax is allowed anywhere an identifier is expected.</p>
+<a name="minutiae_annotations"></a>
+<h3 class="h4">Annotations for Ordering and Namespaces</h3>
+<p>Java-style annotations may be used to add additional
+ properties to types and fields throughout Avro IDL.</p>
+<p>For example, to specify the sort order of a field within
+ a record, one may use the <span class="codefrag">@order</span> annotation
+ before the field name as follows:</p>
+<pre class="code">
+record MyRecord {
+ string @order("ascending") myAscendingSortField;
+ string @order("descending") myDescendingField;
+ string @order("ignore") myIgnoredField;
+}
+ </pre>
+<p>A field's type may also be preceded by annotations, e.g.: </p>
+<pre class="code">
+record MyRecord {
+ @java-class("java.util.ArrayList") array<string> myStrings;
+}
+ </pre>
+<p>This can be used to support java classes that can be
+ serialized/deserialized via their toString/String constructor, e.g.:</p>
+<pre class="code">
+record MyRecord {
+ @java-class("java.math.BigDecimal") string value;
+ @java-key-class("java.io.File") map<string> fileStates;
+ array<@java-class("java.math.BigDecimal") string> weights;
+}
+ </pre>
+<p>Similarly, a <span class="codefrag">@namespace</span> annotation may be used to modify the namespace
+ when defining a named schema. For example:
+ </p>
+<pre class="code">
+@namespace("org.apache.avro.firstNamespace")
+protocol MyProto {
+ @namespace("org.apache.avro.someOtherNamespace")
+ record Foo {}
+
+ record Bar {}
+}
+ </pre>
+<p>
+ will define a protocol in the <span class="codefrag">firstNamespace</span> namespace. The record <span class="codefrag">Foo</span> will be
+ defined in <span class="codefrag">someOtherNamespace</span> and <span class="codefrag">Bar</span> will be defined in <span class="codefrag">firstNamespace</span>
+ as it inherits its default from its container.
+ </p>
+<p>Type and field aliases are specified with
+ the <span class="codefrag">@aliases</span> annotation as follows:</p>
+<pre class="code">
+@aliases(["org.old.OldRecord", "org.ancient.AncientRecord"])
+record MyRecord {
+ string @aliases(["oldField", "ancientField"]) myNewField;
+}
+ </pre>
+<p>Some annotations like those listed above are handled
+ specially. All other annotations are added as properties to
+ the protocol, message, schema or field.</p>
+</div>
+
+<a name="example"></a>
+<h2 class="h3">Complete Example</h2>
+<div class="section">
+<p>The following is a complete example of a Avro IDL file that shows most of the above features:</p>
+<pre class="code">
+/**
+ * An example protocol in Avro IDL
+ */
+@namespace("org.apache.avro.test")
+protocol Simple {
+
+ @aliases(["org.foo.KindOf"])
+ enum Kind {
+ FOO,
+ BAR, // the bar enum value
+ BAZ
+ }
+
+ fixed MD5(16);
+
+ record TestRecord {
+ @order("ignore")
+ string name;
+
+ @order("descending")
+ Kind kind;
+
+ MD5 hash;
+
+ union { MD5, null} @aliases(["hash"]) nullableHash;
+
+ array<long> arrayOfLongs;
+ }
+
+ error TestError {
+ string message;
+ }
+
+ string hello(string greeting);
+ TestRecord echo(TestRecord `record`);
+ int add(int arg1, int arg2);
+ bytes echoBytes(bytes data);
+ void `error`() throws TestError;
+ void ping() oneway;
+}
+ </pre>
+<p>Additional examples may be found in the Avro source tree under the <span class="codefrag">src/test/idl/input</span> directory.</p>
+</div>
+
+
+<p>
+<em>Apache Avro, Avro, Apache, and the Avro and Apache logos are
+ trademarks of The Apache Software Foundation.</em>
+</p>
+
+
+</div>
+<!--+
+ |end content
+ +-->
+<div class="clearboth"> </div>
+</div>
+<div id="footer">
+<!--+
+ |start bottomstrip
+ +-->
+<div class="lastmodified">
+<script type="text/javascript"><!--
+document.write("Last Published: " + document.lastModified);
+// --></script>
+</div>
+<div class="copyright">
+ Copyright ©
+ 2012 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
+</div>
+<!--+
+ |end bottomstrip
+ +-->
+</div>
+</body>
+</html>
Added: avro/site/publish/docs/1.8.2/idl.pdf
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/idl.pdf?rev=1797063&view=auto
==============================================================================
Binary file - no diff available.
Propchange: avro/site/publish/docs/1.8.2/idl.pdf
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: avro/site/publish/docs/1.8.2/images/apache_feather.gif
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/images/apache_feather.gif?rev=1797063&view=auto
==============================================================================
Binary file - no diff available.
Propchange: avro/site/publish/docs/1.8.2/images/apache_feather.gif
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: avro/site/publish/docs/1.8.2/images/avro-logo.png
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/images/avro-logo.png?rev=1797063&view=auto
==============================================================================
Binary file - no diff available.
Propchange: avro/site/publish/docs/1.8.2/images/avro-logo.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: avro/site/publish/docs/1.8.2/images/built-with-forrest-button.png
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/images/built-with-forrest-button.png?rev=1797063&view=auto
==============================================================================
Binary file - no diff available.
Propchange: avro/site/publish/docs/1.8.2/images/built-with-forrest-button.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: avro/site/publish/docs/1.8.2/images/favicon.ico
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/images/favicon.ico?rev=1797063&view=auto
==============================================================================
Binary file - no diff available.
Propchange: avro/site/publish/docs/1.8.2/images/favicon.ico
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: avro/site/publish/docs/1.8.2/images/instruction_arrow.png
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/images/instruction_arrow.png?rev=1797063&view=auto
==============================================================================
Binary file - no diff available.
Propchange: avro/site/publish/docs/1.8.2/images/instruction_arrow.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: avro/site/publish/docs/1.8.2/index.html
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/index.html?rev=1797063&view=auto
==============================================================================
--- avro/site/publish/docs/1.8.2/index.html (added)
+++ avro/site/publish/docs/1.8.2/index.html Wed May 31 15:48:43 2017
@@ -0,0 +1,304 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+<meta content="Apache Forrest" name="Generator">
+<meta name="Forrest-version" content="0.9">
+<meta name="Forrest-skin-name" content="pelt">
+<title>Apache Avro™ 1.8.2
+ Documentation</title>
+<link type="text/css" href="skin/basic.css" rel="stylesheet">
+<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
+<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
+<link type="text/css" href="skin/profile.css" rel="stylesheet">
+<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
+<link rel="shortcut icon" href="images/favicon.ico">
+</head>
+<body onload="init()">
+<script type="text/javascript">ndeSetTextSize();</script>
+<div id="top">
+<!--+
+ |breadtrail
+ +-->
+<div class="breadtrail">
+<a href="http://www.apache.org/">Apache</a> > <a href="http://avro.apache.org/">Avro</a> > <a href="http://avro.apache.org/">Avro</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
+</div>
+<!--+
+ |header
+ +-->
+<div class="header">
+<!--+
+ |start group logo
+ +-->
+<div class="grouplogo">
+<a href="http://www.apache.org/"><img class="logoImage" alt="Apache" src="images/apache_feather.gif" title="The Apache Software Foundation"></a>
+</div>
+<!--+
+ |end group logo
+ +-->
+<!--+
+ |start Project Logo
+ +-->
+<div class="projectlogo">
+<a href="http://avro.apache.org/"><img class="logoImage" alt="Avro" src="images/avro-logo.png" title="Serialization System"></a>
+</div>
+<!--+
+ |end Project Logo
+ +-->
+<!--+
+ |start Search
+ +-->
+<div class="searchbox">
+<form action="http://www.google.com/search" method="get" class="roundtopsmall">
+<input value="avro.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">
+ <input name="Search" value="Search" type="submit">
+</form>
+</div>
+<!--+
+ |end search
+ +-->
+<!--+
+ |start Tabs
+ +-->
+<ul id="tabs">
+<li>
+<a class="unselected" href="http://avro.apache.org/">Project</a>
+</li>
+<li>
+<a class="unselected" href="http://wiki.apache.org/hadoop/Avro/">Wiki</a>
+</li>
+<li class="current">
+<a class="selected" href="index.html">Avro 1.8.2 Documentation</a>
+</li>
+</ul>
+<!--+
+ |end Tabs
+ +-->
+</div>
+</div>
+<div id="main">
+<div id="publishedStrip">
+<!--+
+ |start Subtabs
+ +-->
+<div id="level2tabs"></div>
+<!--+
+ |end Endtabs
+ +-->
+<script type="text/javascript"><!--
+document.write("Last Published: " + document.lastModified);
+// --></script>
+</div>
+<!--+
+ |breadtrail
+ +-->
+<div class="breadtrail">
+
+
+ </div>
+<!--+
+ |start Menu, mainarea
+ +-->
+<!--+
+ |start Menu
+ +-->
+<div id="menu">
+<div onclick="SwitchMenu('menu_selected_1.1', 'skin/')" id="menu_selected_1.1Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Documentation</div>
+<div id="menu_selected_1.1" class="selectedmenuitemgroup" style="display: block;">
+<div class="menupage">
+<div class="menupagetitle">Overview</div>
+</div>
+<div class="menuitem">
+<a href="gettingstartedjava.html">Getting started (Java)</a>
+</div>
+<div class="menuitem">
+<a href="gettingstartedpython.html">Getting started (Python)</a>
+</div>
+<div class="menuitem">
+<a href="spec.html">Specification</a>
+</div>
+<div class="menuitem">
+<a href="trevni/spec.html">Trevni</a>
+</div>
+<div class="menuitem">
+<a href="api/java/index.html">Java API</a>
+</div>
+<div class="menuitem">
+<a href="api/c/index.html">C API</a>
+</div>
+<div class="menuitem">
+<a href="api/cpp/html/index.html">C++ API</a>
+</div>
+<div class="menuitem">
+<a href="api/csharp/index.html">C# API</a>
+</div>
+<div class="menuitem">
+<a href="mr.html">MapReduce guide</a>
+</div>
+<div class="menuitem">
+<a href="idl.html">IDL language</a>
+</div>
+<div class="menuitem">
+<a href="sasl.html">SASL profile</a>
+</div>
+<div class="menuitem">
+<a href="http://wiki.apache.org/hadoop/Avro/">Wiki</a>
+</div>
+<div class="menuitem">
+<a href="http://wiki.apache.org/hadoop/Avro/FAQ">FAQ</a>
+</div>
+</div>
+<div id="credit">
+<hr>
+<a href="http://forrest.apache.org/"><img border="0" title="Built with Apache Forrest" alt="Built with Apache Forrest - logo" src="images/built-with-forrest-button.png" style="width: 88px;height: 31px;"></a>
+</div>
+<div id="roundbottom">
+<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
+<!--+
+ |alternative credits
+ +-->
+<div id="credit2"></div>
+</div>
+<!--+
+ |end Menu
+ +-->
+<!--+
+ |start content
+ +-->
+<div id="content">
+<div title="Portable Document Format" class="pdflink">
+<a class="dida" href="index.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
+ PDF</a>
+</div>
+<h1>Apache Avro™ 1.8.2
+ Documentation</h1>
+<div id="front-matter">
+<div id="minitoc-area">
+<ul class="minitoc">
+<li>
+<a href="#intro">Introduction</a>
+</li>
+<li>
+<a href="#schemas">Schemas</a>
+</li>
+<li>
+<a href="#compare">Comparison with other systems</a>
+</li>
+</ul>
+</div>
+</div>
+
+<a name="intro"></a>
+<h2 class="h3">Introduction</h2>
+<div class="section">
+<p>Apache Avro™ is a data serialization system.</p>
+<p>Avro provides:</p>
+<ul>
+
+<li>Rich data structures.</li>
+
+<li>A compact, fast, binary data format.</li>
+
+<li>A container file, to store persistent data.</li>
+
+<li>Remote procedure call (RPC).</li>
+
+<li>Simple integration with dynamic languages. Code
+ generation is not required to read or write data files nor
+ to use or implement RPC protocols. Code generation as an
+ optional optimization, only worth implementing for
+ statically typed languages.</li>
+
+</ul>
+</div>
+
+<a name="schemas"></a>
+<h2 class="h3">Schemas</h2>
+<div class="section">
+<p>Avro relies on <em>schemas</em>. When Avro data is read, the
+ schema used when writing it is always present. This permits
+ each datum to be written with no per-value overheads, making
+ serialization both fast and small. This also facilitates use
+ with dynamic, scripting languages, since data, together with
+ its schema, is fully self-describing.</p>
+<p>When Avro data is stored in a file, its schema is stored with
+ it, so that files may be processed later by any program. If
+ the program reading the data expects a different schema this
+ can be easily resolved, since both schemas are present.</p>
+<p>When Avro is used in RPC, the client and server exchange
+ schemas in the connection handshake. (This can be optimized
+ so that, for most calls, no schemas are actually transmitted.)
+ Since both client and server both have the other's full
+ schema, correspondence between same named fields, missing
+ fields, extra fields, etc. can all be easily resolved.</p>
+<p>Avro schemas are defined with
+ <a href="http://www.json.org/">JSON</a> . This
+ facilitates implementation in languages that already have
+ JSON libraries.</p>
+</div>
+
+<a name="compare"></a>
+<h2 class="h3">Comparison with other systems</h2>
+<div class="section">
+<p>Avro provides functionality similar to systems such
+ as <a href="http://thrift.apache.org/">Thrift</a>,
+ <a href="http://code.google.com/p/protobuf/">Protocol
+ Buffers</a>, etc. Avro differs from these systems in the
+ following fundamental aspects.</p>
+<ul>
+
+<li>
+<em>Dynamic typing</em>: Avro does not require that code
+ be generated. Data is always accompanied by a schema that
+ permits full processing of that data without code
+ generation, static datatypes, etc. This facilitates
+ construction of generic data-processing systems and
+ languages.</li>
+
+<li>
+<em>Untagged data</em>: Since the schema is present when
+ data is read, considerably less type information need be
+ encoded with data, resulting in smaller serialization size.</li>
+
+<li>
+<em>No manually-assigned field IDs</em>: When a schema
+ changes, both the old and new schema are always present when
+ processing data, so differences may be resolved
+ symbolically, using field names.</li>
+
+</ul>
+</div>
+
+
+<p>
+<em>Apache Avro, Avro, Apache, and the Avro and Apache logos are
+ trademarks of The Apache Software Foundation.</em>
+</p>
+
+
+</div>
+<!--+
+ |end content
+ +-->
+<div class="clearboth"> </div>
+</div>
+<div id="footer">
+<!--+
+ |start bottomstrip
+ +-->
+<div class="lastmodified">
+<script type="text/javascript"><!--
+document.write("Last Published: " + document.lastModified);
+// --></script>
+</div>
+<div class="copyright">
+ Copyright ©
+ 2012 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
+</div>
+<div id="logos"></div>
+<!--+
+ |end bottomstrip
+ +-->
+</div>
+</body>
+</html>
Added: avro/site/publish/docs/1.8.2/index.pdf
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/index.pdf?rev=1797063&view=auto
==============================================================================
Binary file - no diff available.
Propchange: avro/site/publish/docs/1.8.2/index.pdf
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: avro/site/publish/docs/1.8.2/linkmap.html
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/linkmap.html?rev=1797063&view=auto
==============================================================================
--- avro/site/publish/docs/1.8.2/linkmap.html (added)
+++ avro/site/publish/docs/1.8.2/linkmap.html Wed May 31 15:48:43 2017
@@ -0,0 +1,303 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<html>
+<head>
+<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+<meta content="Apache Forrest" name="Generator">
+<meta name="Forrest-version" content="0.9">
+<meta name="Forrest-skin-name" content="pelt">
+<title>Site Linkmap Table of Contents</title>
+<link type="text/css" href="skin/basic.css" rel="stylesheet">
+<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
+<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
+<link type="text/css" href="skin/profile.css" rel="stylesheet">
+<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
+<link rel="shortcut icon" href="images/favicon.ico">
+</head>
+<body onload="init()">
+<script type="text/javascript">ndeSetTextSize();</script>
+<div id="top">
+<!--+
+ |breadtrail
+ +-->
+<div class="breadtrail">
+<a href="http://www.apache.org/">Apache</a> > <a href="http://avro.apache.org/">Avro</a> > <a href="http://avro.apache.org/">Avro</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
+</div>
+<!--+
+ |header
+ +-->
+<div class="header">
+<!--+
+ |start group logo
+ +-->
+<div class="grouplogo">
+<a href="http://www.apache.org/"><img class="logoImage" alt="Apache" src="images/apache_feather.gif" title="The Apache Software Foundation"></a>
+</div>
+<!--+
+ |end group logo
+ +-->
+<!--+
+ |start Project Logo
+ +-->
+<div class="projectlogo">
+<a href="http://avro.apache.org/"><img class="logoImage" alt="Avro" src="images/avro-logo.png" title="Serialization System"></a>
+</div>
+<!--+
+ |end Project Logo
+ +-->
+<!--+
+ |start Search
+ +-->
+<div class="searchbox">
+<form action="http://www.google.com/search" method="get" class="roundtopsmall">
+<input value="avro.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">
+ <input name="Search" value="Search" type="submit">
+</form>
+</div>
+<!--+
+ |end search
+ +-->
+<!--+
+ |start Tabs
+ +-->
+<ul id="tabs">
+<li>
+<a class="unselected" href="http://avro.apache.org/">Project</a>
+</li>
+<li>
+<a class="unselected" href="http://wiki.apache.org/hadoop/Avro/">Wiki</a>
+</li>
+<li class="current">
+<a class="selected" href="index.html">Avro 1.8.2 Documentation</a>
+</li>
+</ul>
+<!--+
+ |end Tabs
+ +-->
+</div>
+</div>
+<div id="main">
+<div id="publishedStrip">
+<!--+
+ |start Subtabs
+ +-->
+<div id="level2tabs"></div>
+<!--+
+ |end Endtabs
+ +-->
+<script type="text/javascript"><!--
+document.write("Last Published: " + document.lastModified);
+// --></script>
+</div>
+<!--+
+ |breadtrail
+ +-->
+<div class="breadtrail">
+
+
+ </div>
+<!--+
+ |start Menu, mainarea
+ +-->
+<!--+
+ |start Menu
+ +-->
+<div id="menu">
+<div onclick="SwitchMenu('menu_1.1', 'skin/')" id="menu_1.1Title" class="menutitle">Documentation</div>
+<div id="menu_1.1" class="menuitemgroup">
+<div class="menuitem">
+<a href="index.html">Overview</a>
+</div>
+<div class="menuitem">
+<a href="gettingstartedjava.html">Getting started (Java)</a>
+</div>
+<div class="menuitem">
+<a href="gettingstartedpython.html">Getting started (Python)</a>
+</div>
+<div class="menuitem">
+<a href="spec.html">Specification</a>
+</div>
+<div class="menuitem">
+<a href="trevni/spec.html">Trevni</a>
+</div>
+<div class="menuitem">
+<a href="api/java/index.html">Java API</a>
+</div>
+<div class="menuitem">
+<a href="api/c/index.html">C API</a>
+</div>
+<div class="menuitem">
+<a href="api/cpp/html/index.html">C++ API</a>
+</div>
+<div class="menuitem">
+<a href="api/csharp/index.html">C# API</a>
+</div>
+<div class="menuitem">
+<a href="mr.html">MapReduce guide</a>
+</div>
+<div class="menuitem">
+<a href="idl.html">IDL language</a>
+</div>
+<div class="menuitem">
+<a href="sasl.html">SASL profile</a>
+</div>
+<div class="menuitem">
+<a href="http://wiki.apache.org/hadoop/Avro/">Wiki</a>
+</div>
+<div class="menuitem">
+<a href="http://wiki.apache.org/hadoop/Avro/FAQ">FAQ</a>
+</div>
+</div>
+<div id="credit"></div>
+<div id="roundbottom">
+<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
+<!--+
+ |alternative credits
+ +-->
+<div id="credit2"></div>
+</div>
+<!--+
+ |end Menu
+ +-->
+<!--+
+ |start content
+ +-->
+<div id="content">
+<div title="Portable Document Format" class="pdflink">
+<a class="dida" href="linkmap.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
+ PDF</a>
+</div>
+<h1>Site Linkmap Table of Contents</h1>
+<div id="front-matter"></div>
+<p>
+ This is a map of the complete site and its structure.
+ </p>
+<ul>
+<li>
+<a>Avro</a> ___________________ <em>site</em>
+</li>
+<ul>
+
+
+<ul>
+<li>
+<a>Documentation</a> ___________________ <em>docs</em>
+</li>
+<ul>
+
+<ul>
+<li>
+<a href="index.html">Overview</a> ___________________ <em>overview</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="gettingstartedjava.html">Getting started (Java)</a> ___________________ <em>gettingstartedjava</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="gettingstartedpython.html">Getting started (Python)</a> ___________________ <em>gettingstartedpython</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="spec.html">Specification</a> ___________________ <em>spec</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="trevni/spec.html">Trevni</a> ___________________ <em>trevni</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="api/java/index.html">Java API</a> ___________________ <em>java-api</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="api/c/index.html">C API</a> ___________________ <em>c-api</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="api/cpp/html/index.html">C++ API</a> ___________________ <em>cpp-api</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="api/csharp/index.html">C# API</a> ___________________ <em>csharp-api</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="mr.html">MapReduce guide</a> ___________________ <em>mr</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="idl.html">IDL language</a> ___________________ <em>idl</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="sasl.html">SASL profile</a> ___________________ <em>sasl</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="http://wiki.apache.org/hadoop/Avro/">Wiki</a> ___________________ <em>wiki</em>
+</li>
+</ul>
+
+<ul>
+<li>
+<a href="http://wiki.apache.org/hadoop/Avro/FAQ">FAQ</a> ___________________ <em>faq</em>
+</li>
+</ul>
+
+</ul>
+</ul>
+
+
+
+
+</ul>
+</ul>
+</div>
+<!--+
+ |end content
+ +-->
+<div class="clearboth"> </div>
+</div>
+<div id="footer">
+<!--+
+ |start bottomstrip
+ +-->
+<div class="lastmodified">
+<script type="text/javascript"><!--
+document.write("Last Published: " + document.lastModified);
+// --></script>
+</div>
+<div class="copyright">
+ Copyright ©
+ 2012 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
+</div>
+<!--+
+ |end bottomstrip
+ +-->
+</div>
+</body>
+</html>
Added: avro/site/publish/docs/1.8.2/linkmap.pdf
URL: http://svn.apache.org/viewvc/avro/site/publish/docs/1.8.2/linkmap.pdf?rev=1797063&view=auto
==============================================================================
Binary file - no diff available.
Propchange: avro/site/publish/docs/1.8.2/linkmap.pdf
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream