questions

[Pere Ferrera <fe...@gmail.com>]

Hi,
I'm evaluating Avro for using it in a big project. I have a few questions:

- What's the equivalent of NullWritable in Avro?
- I generate an entity A. I save some instances of it in file F. I remove
one field from A's definition and re-generate the code of it. If I read file
F, I get "org.apache.avro.AvroRuntimeException("Bad index");". Isn't this
behavior breaking the statement: "*if the writer's record contains a field
with a name not present in the reader's record, the writer's value for that
field is ignored.*"
(here<http://avro.apache.org/docs/current/spec.html#Schema+Resolution>)
? (I'm using a DataFileReader wrapping a SpecificDatumWriter to test this).

Thanks,

Pere.

Re: questions

[Pere Ferrera <fe...@gmail.com>]

On Fri, Jun 24, 2011 at 4:19 PM, Doug Cutting <cu...@apache.org> wrote:

> On 06/24/2011 11:27 AM, Pere Ferrera wrote:
> > </>Hi,
> > I'm evaluating Avro for using it in a big project. I have a few
> questions:
> >
> > - What's the equivalent of NullWritable in Avro?
>
> The schema "null".
>
> > - I generate an entity A. I save some instances of it in file F. I
> > remove one field from A's definition and re-generate the code of it. If
> > I read file F, I get "org.apache.avro.AvroRuntimeException("Bad
> > index");". Isn't this behavior breaking the statement: "*if the writer's
> > record contains a field with a name not present in the reader's record,
> > the writer's value for that field is ignored.*" (here
> > <http://avro.apache.org/docs/current/spec.html#Schema+Resolution>) ?
> > (I'm using a DataFileReader wrapping a SpecificDatumWriter to test this).
>
> That should work.  What version of Avro are you using?  Can you provide
> a complete example that illustrates the problem?


I cloned the project "avro-rpc-quickstart" in github and modified its
ivy.xml to use Avro 1.5.1 . I am playing with the mail.avpr example.
The following is the code I use to test writing / reading from a file:

---

public static void writeOneMessage(Message message, String file) throws
IOException {
 Schema schema = ReflectData.get().getSchema(Message.class);
SpecificDatumWriter<Message> dWriter = new SpecificDatumWriter<Message>();
 DataFileWriter<Message> writer = new DataFileWriter<Message>(dWriter);
writer = writer.create(schema, new File(file));
 writer.append(message);
writer.close();
}
 public static Message readOneMessage(String file) throws IOException {
SpecificDatumReader<Message> dReader = new SpecificDatumReader<Message>();
 DataFileReader<Message> reader = new DataFileReader<Message>(new
File(file), dReader);
Message msg = reader.next();
 reader.close();
return msg;
}
 public static void main(String[] args) throws IOException {
Message message = new Message();
 message.from = new Utf8("From");
message.to   = new Utf8("To");
 writeOneMessage(message, "test");
Message back = readOneMessage("test");
 }

---

I am using a class "Message" that is generated and it looks like this:

---

/**
 * Autogenerated by Avro
 *
 * DO NOT EDIT DIRECTLY
 */
package example.proto;
@SuppressWarnings("all")
public class Message extends org.apache.avro.specific.SpecificRecordBase
implements org.apache.avro.specific.SpecificRecord {
  public static final org.apache.avro.Schema SCHEMA$ =
org.apache.avro.Schema.parse("{\"type\":\"record\",\"name\":\"Message\",\"namespace\":\"example.proto\",\"fields\":[{\"name\":\"to\",\"type\":\"string\"},{\"name\":\"from\",\"type\":\"string\"}]}");
  public java.lang.CharSequence to;
  public java.lang.CharSequence from;
  public org.apache.avro.Schema getSchema() { return SCHEMA$; }
  // Used by DatumWriter.  Applications should not call.
  public java.lang.Object get(int field$) {
    switch (field$) {
    case 0: return to;
    case 1: return from;
    default: throw new org.apache.avro.AvroRuntimeException("Bad index");
    }
  }
  // Used by DatumReader.  Applications should not call.
  @SuppressWarnings(value="unchecked")
  public void put(int field$, java.lang.Object value$) {
    switch (field$) {
    case 0: to = (java.lang.CharSequence)value$; break;
    case 1: from = (java.lang.CharSequence)value$; break;
    default: throw new org.apache.avro.AvroRuntimeException("Bad index");
    }
  }
}

---

It looks like if I remove a field from the message schema and re-generate
the bean then the switch by field index will lead to throwing that
AvroRuntimeException. However, I thought that conflicts in Avro were
resolved by name, not by index.



 Doug
>



-- 
Pere Ferrera
CTO & Co-founder
www.datasalt.com

Re: questions

[Doug Cutting <cu...@apache.org>]

On 06/24/2011 11:27 AM, Pere Ferrera wrote:
> </>Hi,
> I'm evaluating Avro for using it in a big project. I have a few questions:
> 
> - What's the equivalent of NullWritable in Avro?

The schema "null".

> - I generate an entity A. I save some instances of it in file F. I
> remove one field from A's definition and re-generate the code of it. If
> I read file F, I get "org.apache.avro.AvroRuntimeException("Bad
> index");". Isn't this behavior breaking the statement: "*if the writer's
> record contains a field with a name not present in the reader's record,
> the writer's value for that field is ignored.*" (here
> <http://avro.apache.org/docs/current/spec.html#Schema+Resolution>) ?
> (I'm using a DataFileReader wrapping a SpecificDatumWriter to test this).

That should work.  What version of Avro are you using?  Can you provide
a complete example that illustrates the problem?

Doug