You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Jonathan Hsieh (JIRA)" <ji...@apache.org> on 2010/01/08 01:28:54 UTC
[jira] Created: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
JsonEncoder is not flushed after writing using ReflectDatumWriter
------------------------------------------------------------------
Key: AVRO-295
URL: https://issues.apache.org/jira/browse/AVRO-295
Project: Avro
Issue Type: New Feature
Components: java
Affects Versions: 1.3.0
Reporter: Jonathan Hsieh
JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
class A {
long timestamp;
}
public void testEventSchemaSerializeBinary() throws IOException {
A e = new A();
e.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
System.out.println(schm);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder json = new BinaryEncoder(out);
writer.write(e, json); // only one calls
byte[] bs = out.toByteArray();
int len = bs.length; // length is 2, which is reasonable.
System.out.println("output size: " + len);
}
public void testSerializeJson() throws IOException {
A a = new A();
a.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
JsonEncoder json = new JsonEncoder(schm, out);
writer.write(e, json); /// only one call
// did not flush
byte[] bs = out.toByteArray();
int len = bs.length; // len == 0; this is unexpected!
System.out.println("output size: " + len);
// flushed this time.
json.flush();
bs = out.toByteArray();
len = bs.length; // len == 18; this is better!
System.out.println("output size: " + len);
}
One way to deal with it is to have either all Encoders have flush method (so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated AVRO-295:
--------------------------------
Description:
JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
{code}
class A {
long timestamp;
}
public void testEventSchemaSerializeBinary() throws IOException {
A e = new A();
e.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
System.out.println(schm);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder json = new BinaryEncoder(out);
writer.write(e, json); // only one calls
byte[] bs = out.toByteArray();
int len = bs.length; // length is 2, which is reasonable.
System.out.println("output size: " + len);
}
public void testSerializeJson() throws IOException {
A a = new A();
a.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
JsonEncoder json = new JsonEncoder(schm, out);
writer.write(e, json); /// only one call
// did not flush
byte[] bs = out.toByteArray();
int len = bs.length; // len == 0; this is unexpected!
System.out.println("output size: " + len);
// flushed this time. this is a bit unwieldy
json.flush();
bs = out.toByteArray();
len = bs.length; // len == 18; this is better!
System.out.println("output size: " + len);
}
{code}
One way to deal with it is to have either all Encoders have flush method so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
was:
JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
{code}
class A {
long timestamp;
}
public void testEventSchemaSerializeBinary() throws IOException {
A e = new A();
e.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
System.out.println(schm);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder json = new BinaryEncoder(out);
writer.write(e, json); // only one calls
byte[] bs = out.toByteArray();
int len = bs.length; // length is 2, which is reasonable.
System.out.println("output size: " + len);
}
public void testSerializeJson() throws IOException {
A a = new A();
a.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
JsonEncoder json = new JsonEncoder(schm, out);
writer.write(e, json); /// only one call
// did not flush
byte[] bs = out.toByteArray();
int len = bs.length; // len == 0; this is unexpected!
System.out.println("output size: " + len);
// flushed this time. this is a bit unwieldy
json.flush();
bs = out.toByteArray();
len = bs.length; // len == 18; this is better!
System.out.println("output size: " + len);
}
{code}
One way to deal with it is to have either all Encoders have flush method (so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
> JsonEncoder is not flushed after writing using ReflectDatumWriter
> ------------------------------------------------------------------
>
> Key: AVRO-295
> URL: https://issues.apache.org/jira/browse/AVRO-295
> Project: Avro
> Issue Type: New Feature
> Components: java
> Affects Versions: 1.3.0
> Reporter: Jonathan Hsieh
>
> JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
> {code}
> class A {
> long timestamp;
> }
> public void testEventSchemaSerializeBinary() throws IOException {
> A e = new A();
> e.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> System.out.println(schm);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder json = new BinaryEncoder(out);
> writer.write(e, json); // only one calls
> byte[] bs = out.toByteArray();
> int len = bs.length; // length is 2, which is reasonable.
> System.out.println("output size: " + len);
> }
> public void testSerializeJson() throws IOException {
> A a = new A();
> a.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> JsonEncoder json = new JsonEncoder(schm, out);
> writer.write(e, json); /// only one call
> // did not flush
> byte[] bs = out.toByteArray();
> int len = bs.length; // len == 0; this is unexpected!
> System.out.println("output size: " + len);
>
> // flushed this time. this is a bit unwieldy
> json.flush();
> bs = out.toByteArray();
> len = bs.length; // len == 18; this is better!
> System.out.println("output size: " + len);
> }
> {code}
> One way to deal with it is to have either all Encoders have flush method so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-295:
-------------------------------------
Assignee: Thiruvalluvan M. G.
Status: Patch Available (was: Open)
> JsonEncoder is not flushed after writing using ReflectDatumWriter
> ------------------------------------------------------------------
>
> Key: AVRO-295
> URL: https://issues.apache.org/jira/browse/AVRO-295
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.3.0
> Reporter: Jonathan Hsieh
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-295-test.patch, AVRO-295.patch
>
>
> JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
> {code}
> class A {
> long timestamp;
> }
> public void testEventSchemaSerializeBinary() throws IOException {
> A e = new A();
> e.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> System.out.println(schm);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder json = new BinaryEncoder(out);
> writer.write(e, json); // only one calls
> byte[] bs = out.toByteArray();
> int len = bs.length; // length is 2, which is reasonable.
> System.out.println("output size: " + len);
> }
> public void testSerializeJson() throws IOException {
> A a = new A();
> a.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> JsonEncoder json = new JsonEncoder(schm, out);
> writer.write(e, json); /// only one call
> // did not flush
> byte[] bs = out.toByteArray();
> int len = bs.length; // len == 0; this is unexpected!
> System.out.println("output size: " + len);
>
> // flushed this time. this is a bit unwieldy
> json.flush();
> bs = out.toByteArray();
> len = bs.length; // len == 18; this is better!
> System.out.println("output size: " + len);
> }
> {code}
> One way to deal with it is to have either all Encoders have flush method so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831568#action_12831568 ]
Doug Cutting commented on AVRO-295:
-----------------------------------
I am not convinced that we should automatically flush after each object is serialized, as this may adversely affect performance when writing many small objects. Mightn't it be better to add a flush() method to DatumWriter and leave control of when things are flushed to the application?
> JsonEncoder is not flushed after writing using ReflectDatumWriter
> ------------------------------------------------------------------
>
> Key: AVRO-295
> URL: https://issues.apache.org/jira/browse/AVRO-295
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.3.0
> Reporter: Jonathan Hsieh
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-295-test.patch, AVRO-295.patch
>
>
> JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
> {code}
> class A {
> long timestamp;
> }
> public void testEventSchemaSerializeBinary() throws IOException {
> A e = new A();
> e.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> System.out.println(schm);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder json = new BinaryEncoder(out);
> writer.write(e, json); // only one calls
> byte[] bs = out.toByteArray();
> int len = bs.length; // length is 2, which is reasonable.
> System.out.println("output size: " + len);
> }
> public void testSerializeJson() throws IOException {
> A a = new A();
> a.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> JsonEncoder json = new JsonEncoder(schm, out);
> writer.write(e, json); /// only one call
> // did not flush
> byte[] bs = out.toByteArray();
> int len = bs.length; // len == 0; this is unexpected!
> System.out.println("output size: " + len);
>
> // flushed this time. this is a bit unwieldy
> json.flush();
> bs = out.toByteArray();
> len = bs.length; // len == 18; this is better!
> System.out.println("output size: " + len);
> }
> {code}
> One way to deal with it is to have either all Encoders have flush method so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831594#action_12831594 ]
Doug Cutting commented on AVRO-295:
-----------------------------------
> The DatumWriter doesn't own an encoder.
Right. Forgot that!
> Shall we resolve this JIRA as "Not a problem" then? Perhaps, I'll just check-in the test.
+1 That test looks like a fine addition. Thanks!
> JsonEncoder is not flushed after writing using ReflectDatumWriter
> ------------------------------------------------------------------
>
> Key: AVRO-295
> URL: https://issues.apache.org/jira/browse/AVRO-295
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.3.0
> Reporter: Jonathan Hsieh
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-295-test-2.patch, AVRO-295-test.patch, AVRO-295.patch
>
>
> JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
> {code}
> class A {
> long timestamp;
> }
> public void testEventSchemaSerializeBinary() throws IOException {
> A e = new A();
> e.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> System.out.println(schm);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder json = new BinaryEncoder(out);
> writer.write(e, json); // only one calls
> byte[] bs = out.toByteArray();
> int len = bs.length; // length is 2, which is reasonable.
> System.out.println("output size: " + len);
> }
> public void testSerializeJson() throws IOException {
> A a = new A();
> a.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> JsonEncoder json = new JsonEncoder(schm, out);
> writer.write(e, json); /// only one call
> // did not flush
> byte[] bs = out.toByteArray();
> int len = bs.length; // len == 0; this is unexpected!
> System.out.println("output size: " + len);
>
> // flushed this time. this is a bit unwieldy
> json.flush();
> bs = out.toByteArray();
> len = bs.length; // len == 18; this is better!
> System.out.println("output size: " + len);
> }
> {code}
> One way to deal with it is to have either all Encoders have flush method so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-295:
-------------------------------------
Resolution: Not A Problem
Status: Resolved (was: Patch Available)
Committed revision 908366 with the test showing how to flush() from the client code.
> JsonEncoder is not flushed after writing using ReflectDatumWriter
> ------------------------------------------------------------------
>
> Key: AVRO-295
> URL: https://issues.apache.org/jira/browse/AVRO-295
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.3.0
> Reporter: Jonathan Hsieh
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-295-test-2.patch, AVRO-295-test.patch, AVRO-295.patch
>
>
> JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
> {code}
> class A {
> long timestamp;
> }
> public void testEventSchemaSerializeBinary() throws IOException {
> A e = new A();
> e.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> System.out.println(schm);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder json = new BinaryEncoder(out);
> writer.write(e, json); // only one calls
> byte[] bs = out.toByteArray();
> int len = bs.length; // length is 2, which is reasonable.
> System.out.println("output size: " + len);
> }
> public void testSerializeJson() throws IOException {
> A a = new A();
> a.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> JsonEncoder json = new JsonEncoder(schm, out);
> writer.write(e, json); /// only one call
> // did not flush
> byte[] bs = out.toByteArray();
> int len = bs.length; // len == 0; this is unexpected!
> System.out.println("output size: " + len);
>
> // flushed this time. this is a bit unwieldy
> json.flush();
> bs = out.toByteArray();
> len = bs.length; // len == 18; this is better!
> System.out.println("output size: " + len);
> }
> {code}
> One way to deal with it is to have either all Encoders have flush method so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated AVRO-295:
--------------------------------
Description:
JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
{code}
class A {
long timestamp;
}
public void testEventSchemaSerializeBinary() throws IOException {
A e = new A();
e.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
System.out.println(schm);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder json = new BinaryEncoder(out);
writer.write(e, json); // only one calls
byte[] bs = out.toByteArray();
int len = bs.length; // length is 2, which is reasonable.
System.out.println("output size: " + len);
}
public void testSerializeJson() throws IOException {
A a = new A();
a.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
JsonEncoder json = new JsonEncoder(schm, out);
writer.write(e, json); /// only one call
// did not flush
byte[] bs = out.toByteArray();
int len = bs.length; // len == 0; this is unexpected!
System.out.println("output size: " + len);
// flushed this time. this is a bit unwieldy
json.flush();
bs = out.toByteArray();
len = bs.length; // len == 18; this is better!
System.out.println("output size: " + len);
}
{code}
One way to deal with it is to have either all Encoders have flush method (so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
was:
JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
{code}
class A {
long timestamp;
}
public void testEventSchemaSerializeBinary() throws IOException {
A e = new A();
e.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
System.out.println(schm);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder json = new BinaryEncoder(out);
writer.write(e, json); // only one calls
byte[] bs = out.toByteArray();
int len = bs.length; // length is 2, which is reasonable.
System.out.println("output size: " + len);
}
public void testSerializeJson() throws IOException {
A a = new A();
a.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
JsonEncoder json = new JsonEncoder(schm, out);
writer.write(e, json); /// only one call
// did not flush
byte[] bs = out.toByteArray();
int len = bs.length; // len == 0; this is unexpected!
System.out.println("output size: " + len);
// flushed this time.
json.flush();
bs = out.toByteArray();
len = bs.length; // len == 18; this is better!
System.out.println("output size: " + len);
}
{code}
One way to deal with it is to have either all Encoders have flush method (so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
> JsonEncoder is not flushed after writing using ReflectDatumWriter
> ------------------------------------------------------------------
>
> Key: AVRO-295
> URL: https://issues.apache.org/jira/browse/AVRO-295
> Project: Avro
> Issue Type: New Feature
> Components: java
> Affects Versions: 1.3.0
> Reporter: Jonathan Hsieh
>
> JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
> {code}
> class A {
> long timestamp;
> }
> public void testEventSchemaSerializeBinary() throws IOException {
> A e = new A();
> e.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> System.out.println(schm);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder json = new BinaryEncoder(out);
> writer.write(e, json); // only one calls
> byte[] bs = out.toByteArray();
> int len = bs.length; // length is 2, which is reasonable.
> System.out.println("output size: " + len);
> }
> public void testSerializeJson() throws IOException {
> A a = new A();
> a.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> JsonEncoder json = new JsonEncoder(schm, out);
> writer.write(e, json); /// only one call
> // did not flush
> byte[] bs = out.toByteArray();
> int len = bs.length; // len == 0; this is unexpected!
> System.out.println("output size: " + len);
>
> // flushed this time. this is a bit unwieldy
> json.flush();
> bs = out.toByteArray();
> len = bs.length; // len == 18; this is better!
> System.out.println("output size: " + len);
> }
> {code}
> One way to deal with it is to have either all Encoders have flush method (so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-295:
-------------------------------------
Attachment: AVRO-295-test.patch
AVRO-295.patch
Encoder does have a flush() method since it implements java.io.Flushable.
The attached patch takes care of the bug, by flushing in the outermost write() method. The test patch has a test which brings out the bug.
> JsonEncoder is not flushed after writing using ReflectDatumWriter
> ------------------------------------------------------------------
>
> Key: AVRO-295
> URL: https://issues.apache.org/jira/browse/AVRO-295
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.3.0
> Reporter: Jonathan Hsieh
> Attachments: AVRO-295-test.patch, AVRO-295.patch
>
>
> JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
> {code}
> class A {
> long timestamp;
> }
> public void testEventSchemaSerializeBinary() throws IOException {
> A e = new A();
> e.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> System.out.println(schm);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder json = new BinaryEncoder(out);
> writer.write(e, json); // only one calls
> byte[] bs = out.toByteArray();
> int len = bs.length; // length is 2, which is reasonable.
> System.out.println("output size: " + len);
> }
> public void testSerializeJson() throws IOException {
> A a = new A();
> a.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> JsonEncoder json = new JsonEncoder(schm, out);
> writer.write(e, json); /// only one call
> // did not flush
> byte[] bs = out.toByteArray();
> int len = bs.length; // len == 0; this is unexpected!
> System.out.println("output size: " + len);
>
> // flushed this time. this is a bit unwieldy
> json.flush();
> bs = out.toByteArray();
> len = bs.length; // len == 18; this is better!
> System.out.println("output size: " + len);
> }
> {code}
> One way to deal with it is to have either all Encoders have flush method so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated AVRO-295:
--------------------------------
Issue Type: Improvement (was: New Feature)
> JsonEncoder is not flushed after writing using ReflectDatumWriter
> ------------------------------------------------------------------
>
> Key: AVRO-295
> URL: https://issues.apache.org/jira/browse/AVRO-295
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.3.0
> Reporter: Jonathan Hsieh
>
> JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
> {code}
> class A {
> long timestamp;
> }
> public void testEventSchemaSerializeBinary() throws IOException {
> A e = new A();
> e.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> System.out.println(schm);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder json = new BinaryEncoder(out);
> writer.write(e, json); // only one calls
> byte[] bs = out.toByteArray();
> int len = bs.length; // length is 2, which is reasonable.
> System.out.println("output size: " + len);
> }
> public void testSerializeJson() throws IOException {
> A a = new A();
> a.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> JsonEncoder json = new JsonEncoder(schm, out);
> writer.write(e, json); /// only one call
> // did not flush
> byte[] bs = out.toByteArray();
> int len = bs.length; // len == 0; this is unexpected!
> System.out.println("output size: " + len);
>
> // flushed this time. this is a bit unwieldy
> json.flush();
> bs = out.toByteArray();
> len = bs.length; // len == 18; this is better!
> System.out.println("output size: " + len);
> }
> {code}
> One way to deal with it is to have either all Encoders have flush method so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-295:
-------------------------------------
Attachment: AVRO-295-test-2.patch
For the BinaryDecoder, flush() is a no-op. But, for BlockingBinaryEncoder and JsonEncoder flush does some useful things. Still, flushing in DatumWriter could be a performance problem for small objects.
The DatumWriter doesn't own an encoder. The encoder gets passed in the write() method. So it doesn't make sense to add a flush() method in DatumWriter.
Of course, the client can flush the encoder whenever it wants. test-2 patch demonstrates this. If we take this route, there is no fix required.
Shall we resolve this JIRA as "Not a problem" then? Perhaps, I'll just check-in the test.
> JsonEncoder is not flushed after writing using ReflectDatumWriter
> ------------------------------------------------------------------
>
> Key: AVRO-295
> URL: https://issues.apache.org/jira/browse/AVRO-295
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.3.0
> Reporter: Jonathan Hsieh
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-295-test-2.patch, AVRO-295-test.patch, AVRO-295.patch
>
>
> JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
> {code}
> class A {
> long timestamp;
> }
> public void testEventSchemaSerializeBinary() throws IOException {
> A e = new A();
> e.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> System.out.println(schm);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder json = new BinaryEncoder(out);
> writer.write(e, json); // only one calls
> byte[] bs = out.toByteArray();
> int len = bs.length; // length is 2, which is reasonable.
> System.out.println("output size: " + len);
> }
> public void testSerializeJson() throws IOException {
> A a = new A();
> a.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> JsonEncoder json = new JsonEncoder(schm, out);
> writer.write(e, json); /// only one call
> // did not flush
> byte[] bs = out.toByteArray();
> int len = bs.length; // len == 0; this is unexpected!
> System.out.println("output size: " + len);
>
> // flushed this time. this is a bit unwieldy
> json.flush();
> bs = out.toByteArray();
> len = bs.length; // len == 18; this is better!
> System.out.println("output size: " + len);
> }
> {code}
> One way to deal with it is to have either all Encoders have flush method so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-295) JsonEncoder is not flushed after
writing using ReflectDatumWriter
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated AVRO-295:
--------------------------------
Description:
JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
{code}
class A {
long timestamp;
}
public void testEventSchemaSerializeBinary() throws IOException {
A e = new A();
e.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
System.out.println(schm);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder json = new BinaryEncoder(out);
writer.write(e, json); // only one calls
byte[] bs = out.toByteArray();
int len = bs.length; // length is 2, which is reasonable.
System.out.println("output size: " + len);
}
public void testSerializeJson() throws IOException {
A a = new A();
a.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
JsonEncoder json = new JsonEncoder(schm, out);
writer.write(e, json); /// only one call
// did not flush
byte[] bs = out.toByteArray();
int len = bs.length; // len == 0; this is unexpected!
System.out.println("output size: " + len);
// flushed this time.
json.flush();
bs = out.toByteArray();
len = bs.length; // len == 18; this is better!
System.out.println("output size: " + len);
}
{code}
One way to deal with it is to have either all Encoders have flush method (so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
was:
JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
class A {
long timestamp;
}
public void testEventSchemaSerializeBinary() throws IOException {
A e = new A();
e.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
System.out.println(schm);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder json = new BinaryEncoder(out);
writer.write(e, json); // only one calls
byte[] bs = out.toByteArray();
int len = bs.length; // length is 2, which is reasonable.
System.out.println("output size: " + len);
}
public void testSerializeJson() throws IOException {
A a = new A();
a.timestamp = 1234;
ReflectData reflectData = ReflectData.get();
Schema schm = reflectData.getSchema(A.class);
ReflectDatumWriter writer = new ReflectDatumWriter(schm);
ByteArrayOutputStream out = new ByteArrayOutputStream();
JsonEncoder json = new JsonEncoder(schm, out);
writer.write(e, json); /// only one call
// did not flush
byte[] bs = out.toByteArray();
int len = bs.length; // len == 0; this is unexpected!
System.out.println("output size: " + len);
// flushed this time.
json.flush();
bs = out.toByteArray();
len = bs.length; // len == 18; this is better!
System.out.println("output size: " + len);
}
One way to deal with it is to have either all Encoders have flush method (so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
> JsonEncoder is not flushed after writing using ReflectDatumWriter
> ------------------------------------------------------------------
>
> Key: AVRO-295
> URL: https://issues.apache.org/jira/browse/AVRO-295
> Project: Avro
> Issue Type: New Feature
> Components: java
> Affects Versions: 1.3.0
> Reporter: Jonathan Hsieh
>
> JsonEncoder needs to be flushed otherwise data may be left in its buffers. Ideally behavior should be the same regardless of what kind of Encoder passed in. Here is some example code:
> {code}
> class A {
> long timestamp;
> }
> public void testEventSchemaSerializeBinary() throws IOException {
> A e = new A();
> e.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> System.out.println(schm);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder json = new BinaryEncoder(out);
> writer.write(e, json); // only one calls
> byte[] bs = out.toByteArray();
> int len = bs.length; // length is 2, which is reasonable.
> System.out.println("output size: " + len);
> }
> public void testSerializeJson() throws IOException {
> A a = new A();
> a.timestamp = 1234;
> ReflectData reflectData = ReflectData.get();
> Schema schm = reflectData.getSchema(A.class);
> ReflectDatumWriter writer = new ReflectDatumWriter(schm);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> JsonEncoder json = new JsonEncoder(schm, out);
> writer.write(e, json); /// only one call
> // did not flush
> byte[] bs = out.toByteArray();
> int len = bs.length; // len == 0; this is unexpected!
> System.out.println("output size: " + len);
>
> // flushed this time.
> json.flush();
> bs = out.toByteArray();
> len = bs.length; // len == 18; this is better!
> System.out.println("output size: " + len);
> }
> {code}
> One way to deal with it is to have either all Encoders have flush method (so the DatumWriter can always flush it, and potentially add a flush method to DatumWriter as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.