You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Roshan Naik <ro...@hortonworks.com> on 2014/02/26 23:54:06 UTC
Review Request 18544: Hive Streaming sink
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/
-----------------------------------------------------------
Review request for Flume.
Bugs: FLUME-1734
https://issues.apache.org/jira/browse/FLUME-1734
Repository: flume-git
Description
-------
Hive streaming sink.
Diffs
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst 8390cd2
flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties PRE-CREATION
flume-ng-sinks/pom.xml 6ac2b4d
pom.xml 362fb45
Diff: https://reviews.apache.org/r/18544/diff/
Testing
-------
This version lacks unit tests.
Thanks,
Roshan Naik
Re: Review Request 18544: Hive Streaming sink
Posted by Roshan Naik <ro...@hortonworks.com>.
> On Aug. 21, 2014, 7:24 p.m., Brock Noland wrote:
> > conf/log4j.properties, line 35
> > <https://reviews.apache.org/r/18544/diff/4/?file=609917#file609917line35>
> >
> > Hmm, are we doing this?
had to put it in as there was just too much noise coming from the hive libraries during normal flume operation.
- Roshan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/#review51208
-----------------------------------------------------------
On Sept. 3, 2014, 3:19 a.m., Roshan Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18544/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2014, 3:19 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-1734
> https://issues.apache.org/jira/browse/FLUME-1734
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Hive streaming sink.
>
>
> Diffs
> -----
>
> bin/flume-ng e09e26b
> conf/log4j.properties 3918511
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac11558
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java 0a1cd7a
> flume-ng-dist/pom.xml 8c18af6
> flume-ng-doc/sphinx/FlumeUserGuide.rst a718fbf
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java ff32c45
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java 8e08f22
> flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftSourceProtocol.java 7f966b0
> flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/Config.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveJsonSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveSink.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveWriter.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties PRE-CREATION
> flume-ng-sinks/pom.xml 3381bde
> flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java eba8d2e
> pom.xml 4bdfcac
>
> Diff: https://reviews.apache.org/r/18544/diff/
>
>
> Testing
> -------
>
> includes unit tests.
>
>
> Thanks,
>
> Roshan Naik
>
>
Re: Review Request 18544: Hive Streaming sink
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/#review51208
-----------------------------------------------------------
Hi,
Thank you very mcuh Roshan! I have some comments.
conf/log4j.properties
<https://reviews.apache.org/r/18544/#comment89249>
Hmm, are we doing this?
flume-ng-doc/sphinx/FlumeUserGuide.rst
<https://reviews.apache.org/r/18544/#comment89250>
trailing ws
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java
<https://reviews.apache.org/r/18544/#comment89251>
this isn't upgraded to 0.9.1
but if we did remove this we'd need to remove this comments
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java
<https://reviews.apache.org/r/18544/#comment89252>
why is this not private? also both of these will default to null
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java
<https://reviews.apache.org/r/18544/#comment89253>
let's put all these constants in a constant
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java
<https://reviews.apache.org/r/18544/#comment89254>
The rest of this file has many spacing mistakes
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment89260>
constants should be in ALL CAPS
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment89261>
why is this not private?
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment89262>
why is this visible?
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment89263>
pls move constants to a constants class
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment89264>
space on each side of ==
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment89265>
more space issues (all over) this file
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment89267>
comemnted out code should be removed
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment89270>
more space issues
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment89271>
why are we doing a for loop on a single item?
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java
<https://reviews.apache.org/r/18544/#comment89272>
if cause is an Error it should be re-thrown as Error.
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java
<https://reviews.apache.org/r/18544/#comment89273>
remove commented out code
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java
<https://reviews.apache.org/r/18544/#comment89274>
nest exception
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java
<https://reviews.apache.org/r/18544/#comment89269>
all of these should end with Exception
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java
<https://reviews.apache.org/r/18544/#comment89275>
commented code should be removed
- Brock Noland
On June 14, 2014, 4:19 a.m., Roshan Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18544/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 4:19 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-1734
> https://issues.apache.org/jira/browse/FLUME-1734
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Hive streaming sink.
>
>
> Diffs
> -----
>
> bin/flume-ng e09e26b
> conf/log4j.properties 3918511
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac11558
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java 0a1cd7a
> flume-ng-dist/pom.xml 690fec5
> flume-ng-doc/sphinx/FlumeUserGuide.rst b2058f5
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java ff32c45
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java 8e08f22
> flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftSourceProtocol.java 7f966b0
> flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveJsonSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveSink.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveWriter.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties PRE-CREATION
> flume-ng-sinks/pom.xml 56c4dee
> flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java eba8d2e
> pom.xml 4dae2c9
>
> Diff: https://reviews.apache.org/r/18544/diff/
>
>
> Testing
> -------
>
> This version lacks unit tests.
>
>
> Thanks,
>
> Roshan Naik
>
>
Re: Review Request 18544: Hive Streaming sink
Posted by Roshan Naik <ro...@hortonworks.com>.
> On Oct. 14, 2014, 11:05 p.m., Brock Noland wrote:
> > flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java, line 136
> > <https://reviews.apache.org/r/18544/diff/5/?file=674780#file674780line136>
> >
> > We should log exceptions like this.
loogging not yet been added in for unit tests. this class is used for unit tests.
> On Oct. 14, 2014, 11:05 p.m., Brock Noland wrote:
> > flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java, line 212
> > <https://reviews.apache.org/r/18544/diff/5/?file=674768#file674768line212>
> >
> > are these changes due to the rev in the thrift version?
yes. my notes suggest a race condition in TThreadSeverPool
- Roshan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/#review56592
-----------------------------------------------------------
On Sept. 3, 2014, 3:19 a.m., Roshan Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18544/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2014, 3:19 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-1734
> https://issues.apache.org/jira/browse/FLUME-1734
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Hive streaming sink.
>
>
> Diffs
> -----
>
> bin/flume-ng e09e26b
> conf/log4j.properties 3918511
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac11558
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java 0a1cd7a
> flume-ng-dist/pom.xml 8c18af6
> flume-ng-doc/sphinx/FlumeUserGuide.rst a718fbf
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java ff32c45
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java 8e08f22
> flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftSourceProtocol.java 7f966b0
> flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/Config.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveJsonSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveSink.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveWriter.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties PRE-CREATION
> flume-ng-sinks/pom.xml 3381bde
> flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java eba8d2e
> pom.xml 4bdfcac
>
> Diff: https://reviews.apache.org/r/18544/diff/
>
>
> Testing
> -------
>
> includes unit tests.
>
>
> Thanks,
>
> Roshan Naik
>
>
Re: Review Request 18544: Hive Streaming sink
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/#review56592
-----------------------------------------------------------
Hi Roshan,
It looks good and I think we should commit this soon. There are just a couple things we should resolve. There is a bunch of code which is commented out. We should remove that. Also there is a bunch of lint/whitepsace issues which we should resolve. Can you go throught the patch and find any if(x==y) and make it if (x == y)
flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java
<https://reviews.apache.org/r/18544/#comment96959>
are these changes due to the rev in the thrift version?
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java
<https://reviews.apache.org/r/18544/#comment96958>
extra line
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java
<https://reviews.apache.org/r/18544/#comment96957>
extra space
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java
<https://reviews.apache.org/r/18544/#comment96956>
spaces
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment96955>
map on LHS
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment96952>
why is this here and then the same thing a few lines below not commented
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment96953>
why is heartBeatTimer not private
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment96954>
does this really need hashmap on LHS? Sould be map
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment96951>
this
context.getString(Config.SERIALIZER);
should be
context.getString(Config.SERIALIZER, "").trim();
and then you can check for s.isEmpty() in the next line and not worry about nulls.
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java
<https://reviews.apache.org/r/18544/#comment96950>
Why do we have HashMap on LHS, it should be Map
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java
<https://reviews.apache.org/r/18544/#comment96949>
All of these classes below should end with Exception
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveWriter.java
<https://reviews.apache.org/r/18544/#comment96948>
remove commented code
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java
<https://reviews.apache.org/r/18544/#comment96946>
We should log exceptions like this.
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java
<https://reviews.apache.org/r/18544/#comment96947>
missing spaces
if(partKeys.size()!=partVals.size())
should be:
if (partKeys.size() != partVals.size())
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java
<https://reviews.apache.org/r/18544/#comment96945>
We should remove commented code
- Brock Noland
On Sept. 3, 2014, 3:19 a.m., Roshan Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18544/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2014, 3:19 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-1734
> https://issues.apache.org/jira/browse/FLUME-1734
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Hive streaming sink.
>
>
> Diffs
> -----
>
> bin/flume-ng e09e26b
> conf/log4j.properties 3918511
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac11558
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java 0a1cd7a
> flume-ng-dist/pom.xml 8c18af6
> flume-ng-doc/sphinx/FlumeUserGuide.rst a718fbf
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java ff32c45
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java 8e08f22
> flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftSourceProtocol.java 7f966b0
> flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/Config.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveJsonSerializer.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveSink.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveWriter.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java PRE-CREATION
> flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties PRE-CREATION
> flume-ng-sinks/pom.xml 3381bde
> flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java eba8d2e
> pom.xml 4bdfcac
>
> Diff: https://reviews.apache.org/r/18544/diff/
>
>
> Testing
> -------
>
> includes unit tests.
>
>
> Thanks,
>
> Roshan Naik
>
>
Re: Review Request 18544: Hive Streaming sink
Posted by Roshan Naik <ro...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/
-----------------------------------------------------------
(Updated Nov. 13, 2014, 3:05 a.m.)
Review request for Flume.
Bugs: FLUME-1734
https://issues.apache.org/jira/browse/FLUME-1734
Repository: flume-git
Description
-------
Hive streaming sink.
Diffs (updated)
-----
bin/flume-ng 4b323a6
conf/log4j.properties 3918511
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac11558
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java 0a1cd7a
flume-ng-dist/pom.xml a5db0c7
flume-ng-doc/sphinx/FlumeUserGuide.rst 0ab23fd
flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java ff32c45
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java 8e08f22
flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftSourceProtocol.java 7f966b0
flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/Config.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveJsonSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveSink.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveWriter.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties PRE-CREATION
flume-ng-sinks/pom.xml 4bac019
flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java eba8d2e
pom.xml 4f550d3
Diff: https://reviews.apache.org/r/18544/diff/
Testing
-------
includes unit tests.
Thanks,
Roshan Naik
Re: Review Request 18544: Hive Streaming sink
Posted by Roshan Naik <ro...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/
-----------------------------------------------------------
(Updated Sept. 3, 2014, 3:19 a.m.)
Review request for Flume.
Changes
-------
adressing brock's review comments
Bugs: FLUME-1734
https://issues.apache.org/jira/browse/FLUME-1734
Repository: flume-git
Description
-------
Hive streaming sink.
Diffs (updated)
-----
bin/flume-ng e09e26b
conf/log4j.properties 3918511
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac11558
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java 0a1cd7a
flume-ng-dist/pom.xml 8c18af6
flume-ng-doc/sphinx/FlumeUserGuide.rst a718fbf
flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java ff32c45
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java 8e08f22
flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftSourceProtocol.java 7f966b0
flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/Config.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveJsonSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveSink.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveWriter.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties PRE-CREATION
flume-ng-sinks/pom.xml 3381bde
flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java eba8d2e
pom.xml 4bdfcac
Diff: https://reviews.apache.org/r/18544/diff/
Testing (updated)
-------
includes unit tests.
Thanks,
Roshan Naik
Re: Review Request 18544: Hive Streaming sink
Posted by Roshan Naik <ro...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/
-----------------------------------------------------------
(Updated June 14, 2014, 4:19 a.m.)
Review request for Flume.
Changes
-------
Uploading new patch v2 :
addressed loose ends in unit tests
fixed issues related to recovering from intermittent HDFS and Hive failures.
Bugs: FLUME-1734
https://issues.apache.org/jira/browse/FLUME-1734
Repository: flume-git
Description
-------
Hive streaming sink.
Diffs (updated)
-----
bin/flume-ng e09e26b
conf/log4j.properties 3918511
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac11558
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java 0a1cd7a
flume-ng-dist/pom.xml 690fec5
flume-ng-doc/sphinx/FlumeUserGuide.rst b2058f5
flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java ff32c45
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java 8e08f22
flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftSourceProtocol.java 7f966b0
flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveJsonSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveSink.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveWriter.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties PRE-CREATION
flume-ng-sinks/pom.xml 56c4dee
flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java eba8d2e
pom.xml 4dae2c9
Diff: https://reviews.apache.org/r/18544/diff/
Testing
-------
This version lacks unit tests.
Thanks,
Roshan Naik
Re: Review Request 18544: Hive Streaming sink
Posted by Roshan Naik <ro...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/
-----------------------------------------------------------
(Updated April 10, 2014, 8:17 a.m.)
Review request for Flume.
Changes
-------
Fully functional patch
Bugs: FLUME-1734
https://issues.apache.org/jira/browse/FLUME-1734
Repository: flume-git
Description
-------
Hive streaming sink.
Diffs (updated)
-----
bin/flume-ng e09e26b
conf/log4j.properties 3918511
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac11558
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java 0a1cd7a
flume-ng-dist/pom.xml 338e108
flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
flume-ng-legacy-sources/flume-thrift-source/src/main/java/com/cloudera/flume/handlers/thrift/ThriftFlumeEventServer.java ff32c45
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java 8e08f22
flume-ng-sdk/src/main/java/org/apache/flume/thrift/ThriftSourceProtocol.java 7f966b0
flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveJsonSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveSink.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestHiveWriter.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/java/org/apache/flume/sink/hive/TestUtil.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties PRE-CREATION
flume-ng-sinks/pom.xml 6ac2b4d
flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java eba8d2e
pom.xml 2aa0ad1
Diff: https://reviews.apache.org/r/18544/diff/
Testing
-------
This version lacks unit tests.
Thanks,
Roshan Naik
Re: Review Request 18544: Hive Streaming sink
Posted by Roshan Naik <ro...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18544/
-----------------------------------------------------------
(Updated Feb. 27, 2014, 8:54 a.m.)
Review request for Flume.
Changes
-------
Updating patch with Serializer support
Bugs: FLUME-1734
https://issues.apache.org/jira/browse/FLUME-1734
Repository: flume-git
Description
-------
Hive streaming sink.
Diffs (updated)
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst 8390cd2
flume-ng-sinks/flume-hive-sink/pom.xml PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveDelimitedTextSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveEventSerializer.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveSink.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/main/java/org/apache/flume/sink/hive/HiveWriter.java PRE-CREATION
flume-ng-sinks/flume-hive-sink/src/test/resources/log4j.properties PRE-CREATION
flume-ng-sinks/pom.xml 6ac2b4d
pom.xml 362fb45
Diff: https://reviews.apache.org/r/18544/diff/
Testing
-------
This version lacks unit tests.
Thanks,
Roshan Naik