You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Dave Oshinsky <do...@commvault.com> on 2016/02/05 17:33:14 UTC
strange build problem with generated sources
Hi Drill-ers,
I am looking into fixing JIRA https://issues.apache.org/jira/browse/DRILL-4184. I've encountered a number of strange build problems along the way with my drill 1.4 snapshot, including inability to rebuild after running "mvn clean", no matter what I try. So, I'm building from scratch for the second time, at least. The latest problem really has me stumped at the moment. I added a "forceBits(int,int)" method that I see in generated source file NullableDecimal28SparseVector.java (and NullableDecimal38SparseVector.java), but somehow this doesn't get compiled properly into the *.class and my build keeps failing as if the new forceBits method isn't there:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project drill-java-exec: Compilation failure: Compilation failure:
[ERROR] C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\main\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthColumnReaders.java:[108,29] error: cannot find symbol
[ERROR] symbol: method forceBits(int,int)
[ERROR] location: variable nullableDecimal28Vector of type NullableDecimal28SparseVector
[ERROR] C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\main\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthColumnReaders.java:[179,29] error: cannot find symbol
Can anyone suggest how to fix this without starting over from scratch in a new build node (again)? Any advice would be greatly appreciated.
I will send a separate email eventually regarding the design of my fix, which I know is only a short-term solution to the problem of handling variable width decimal fields in Parquet files. To make a long story short, all the decimal vectors are fixed width vectors, which don't have the ability to "remember" varying sizes from one decimal field to the next. I've hacked up something to "remember" the varying field sizes (BigDecimal array sizes) in NullableVarLengthValuesColumn and VarLengthValuesColumn, not in the decimal vectors. This seems to work, though it's admittedly ugly. However, I ran into a problem with nullable varying width decimal columns where the "isSet" always returns 0, as if the column is null, when it is not, and the sparse decimal data is present in the vector (but Drill won't send the decimal value, because it thinks it's null). Hence the "forceBits" hack to try to work around this. It seemed like I was close to running a successful Drill query on the varying width decimal Parquet data, but alas, I ran into (another) build problem. I do have a LOT of questions as to why the decimal stuff was designed the way it is, but that's for another email....
Thanks,
Dave Oshinsky
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
RE: strange build problem with generated sources
Posted by Dave Oshinsky <do...@commvault.com>.
While the build was successful, Drill fails at runtime. I built and copied these jars:
cp exec/java-exec/target/drill-java-exec-1.4.0.jar ../../jars/drill-java-exec-1.4.0.jar
cp exec/vector/target/vector-1.4.0.jar ../../jars
The exception from Drill:
Exception in thread "drill-executor-2" java.lang.Error: Unresolved compilation problem:
The method forceBits(int, int) is undefined for the type NullableDecimal28SparseVector
at org.apache.drill.exec.store.parquet.columnreaders.VarLengthColumnReaders$NullableDecimal28Column.setSafe(VarLengthColumnReaders.java:108)
at org.apache.drill.exec.store.parquet.columnreaders.NullableVarLengthValuesColumn.readField(NullableVarLengthValuesColumn.java:160)
at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readRecords(ColumnReader.java:163)
at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readPage(ColumnReader.java:194)
at org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:141)
This is the NullableDecimal28SparseVector.java code showing forceBits() is implemented:
public void forceBits(int index, int value){ // DAO
bits.getMutator().setSafe(index, value);
}
Do I need to copy any other jars? I copied everything that implements or refers to NullableDecimal28SparseVector, or so I thought.
-----Original Message-----
From: Dave Oshinsky [mailto:doshinsky@commvault.com]
Sent: Friday, February 05, 2016 1:23 PM
To: dev@drill.apache.org
Subject: RE: strange build problem with generated sources
Thanks Abdel! That fixed the build (adding it to FixedValueVectors.java, throwing UnsupportedOperationException). Why was that necessary? Is it some complicated aspect to how NullableDecimal38SparseVector.class gets compiled from NullableDecimal38SparseVector.java? I actually saw the forceBits() method in the generated NullableDecimal38SparseVector.java code before, yet the build behaved as if it were not there.
The error I saw earlier was from these two calls to forceBits() in VarLengthColumnReaders.java:
@Override
public boolean setSafe(int index, BigDecimal intermediate) { // DAO
int width = Decimal28SparseHolder.WIDTH;
if (index >= nullableDecimal28Vector.getValueCapacity()) {
return false;
}
DecimalUtility.getSparseFromBigDecimal(intermediate, nullableDecimal28Vector.getBuffer(), index * width, schemaElement.getScale(),
schemaElement.getPrecision(), Decimal28SparseHolder.nDecimalDigits);
nullableDecimal28Vector.forceBits(index, 1);
return true;
}
........
@Override
public boolean setSafe(int index, BigDecimal intermediate) { // DAO
int width = Decimal38SparseHolder.WIDTH;
if (index >= nullableDecimal38Vector.getValueCapacity()) {
return false;
}
DecimalUtility.getSparseFromBigDecimal(intermediate, nullableDecimal38Vector.getBuffer(), index * width, schemaElement.getScale(),
schemaElement.getPrecision(), Decimal38SparseHolder.nDecimalDigits);
nullableDecimal38Vector.forceBits(index, 1);
return true;
}
-----Original Message-----
From: Abdel Hakim Deneche [mailto:adeneche@maprtech.com]
Sent: Friday, February 05, 2016 12:41 PM
To: dev@drill.apache.org
Subject: Re: strange build problem with generated sources
Do you also need to add forceBits to the template FixedValueVectors.java ?
Can you copy the line from VarLengthColumnReaders .java where you hit a compile error ?
On Fri, Feb 5, 2016 at 9:26 AM, Dave Oshinsky <do...@commvault.com>
wrote:
> Hi Abdel,
> Thanks for reply. I added it to
> exec/vector/target/codegen/templates/NullableValueVectors.java. I
> then see it propagated by Freemarker to generated sources, as shown
> below signature. In Eclipse, I'm able to navigate from where I'm
> calling
> forceBits() to e.g. NullableDecimal28SparseVector.java, and I see
> forceBits() implemented there. However, Eclipse has the dreaded "red
> X" in VarLengthColumnReaders.java indicating an error, and the build also fails.
> Weird, huh?
>
> I do have to say, this whole "generated sources" business is a pain in
> the you know where ;-) In all seriousness, I would implement the
> whole decimal business MUCH more simply if writing it from scratch.
> Why the gazillion special cases for different widths (all with
> separate generated sources), for nullable and not, sparse or dense?
> Just treat them all the same, at least from where I'm sitting with a Parquet-centric perspective.
> And the memory usage would be MUCH lower by treating decimals ALWAYS
> as variable width. Most actual decimal numbers are way smaller than
> the precision would indicate, in many of my test cases.
>
> Dave Oshinsky
>
> $ find . -name "*.java" | xargs grep forceBits
> ./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal28Vector.forceBits(index, 1);
> ./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal38Vector.forceBits(index, 1);
> ./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal28Vector.forceBits(index, 1);
> ./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal38Vector.forceBits(index, 1);
> ./exec/vector/src/main/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/classes/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBigIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBitVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDateVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal18Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28DenseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28SparseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38DenseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38SparseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal9Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat4Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat8Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalDayVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalYearVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableSmallIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeStampVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTinyIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt1Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt2Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt4Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt8Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVar16CharVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarBinaryVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarCharVector.java:
> public void forceBits(int index, int value){ // DAO
>
> -----Original Message-----
> From: Abdel Hakim Deneche [mailto:adeneche@maprtech.com]
> Sent: Friday, February 05, 2016 12:10 PM
> To: dev@drill.apache.org
> Subject: Re: strange build problem with generated sources
>
> Hey Dave,
>
> Which file did you add the "forceBits()" method to ?
>
> On Fri, Feb 5, 2016 at 8:33 AM, Dave Oshinsky
> <do...@commvault.com>
> wrote:
>
> > Hi Drill-ers,
> > I am looking into fixing JIRA
> > https://issues.apache.org/jira/browse/DRILL-4184. I've encountered
> > a number of strange build problems along the way with my drill 1.4
> > snapshot, including inability to rebuild after running "mvn clean",
> > no matter what I try. So, I'm building from scratch for the second
> > time, at least. The latest problem really has me stumped at the
> > moment. I added a "forceBits(int,int)" method that I see in
> > generated source file NullableDecimal28SparseVector.java (and
> > NullableDecimal38SparseVector.java), but somehow this doesn't get
> > compiled properly into the *.class and my build keeps failing as if
> > the new forceBits method isn't there:
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.2:compile
> > (default-compile) on project drill-java-exec: Compilation failure:
> > Compilation failure:
> > [ERROR]
> > C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src
> > \m
> > ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLength
> > Co
> > lumnReaders.java:[108,29]
> > error: cannot find symbol
> > [ERROR] symbol: method forceBits(int,int)
> > [ERROR] location: variable nullableDecimal28Vector of type
> > NullableDecimal28SparseVector [ERROR]
> > C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src
> > \m
> > ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLength
> > Co
> > lumnReaders.java:[179,29]
> > error: cannot find symbol
> >
> > Can anyone suggest how to fix this without starting over from
> > scratch in a new build node (again)? Any advice would be greatly appreciated.
> >
> > I will send a separate email eventually regarding the design of my
> > fix, which I know is only a short-term solution to the problem of
> > handling variable width decimal fields in Parquet files. To make a
> > long story short, all the decimal vectors are fixed width vectors,
> > which don't have the ability to "remember" varying sizes from one
> > decimal field to the next. I've hacked up something to "remember"
> > the varying field sizes (BigDecimal array sizes) in
> > NullableVarLengthValuesColumn and VarLengthValuesColumn, not in the
> > decimal vectors. This seems to work, though it's admittedly ugly.
> > However, I ran into a problem with nullable varying width decimal
> > columns where the "isSet" always returns 0, as if the column is
> > null, when it is not, and the sparse decimal data is present in the
> > vector (but Drill won't send the decimal value, because it thinks it's null).
> > Hence the "forceBits" hack to try to work around this. It seemed
> > like I was close to running a successful Drill query on the varying
> > width decimal Parquet data, but alas, I ran into (another) build
> > problem. I do have a LOT of questions as to why the decimal stuff
> > was designed the
> way it is, but that's for another email....
> >
> > Thanks,
> > Dave Oshinsky
> >
> >
> >
> >
> >
> > ***************************Legal
> > Disclaimer***************************
> > "This communication may contain confidential and privileged material
> > for the sole use of the intended recipient. Any unauthorized review,
> > use or distribution by others is strictly prohibited. If you have
> > received the message by mistake, please advise the sender by reply
> > email and delete the message. Thank you."
> > ********************************************************************
> > **
>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm
> _campaign=Free%20available
> >
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material
> for the sole use of the intended recipient. Any unauthorized review,
> use or distribution by others is strictly prohibited. If you have
> received the message by mistake, please advise the sender by reply
> email and delete the message. Thank you."
> **********************************************************************
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the sole use of the intended recipient. Any unauthorized review, use or distribution by others is strictly prohibited. If you have received the message by mistake, please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
RE: strange build problem with generated sources
Posted by Dave Oshinsky <do...@commvault.com>.
Thanks Abdel! That fixed the build (adding it to FixedValueVectors.java, throwing UnsupportedOperationException). Why was that necessary? Is it some complicated aspect to how NullableDecimal38SparseVector.class gets compiled from NullableDecimal38SparseVector.java? I actually saw the forceBits() method in the generated NullableDecimal38SparseVector.java code before, yet the build behaved as if it were not there.
The error I saw earlier was from these two calls to forceBits() in VarLengthColumnReaders.java:
@Override
public boolean setSafe(int index, BigDecimal intermediate) { // DAO
int width = Decimal28SparseHolder.WIDTH;
if (index >= nullableDecimal28Vector.getValueCapacity()) {
return false;
}
DecimalUtility.getSparseFromBigDecimal(intermediate, nullableDecimal28Vector.getBuffer(), index * width, schemaElement.getScale(),
schemaElement.getPrecision(), Decimal28SparseHolder.nDecimalDigits);
nullableDecimal28Vector.forceBits(index, 1);
return true;
}
........
@Override
public boolean setSafe(int index, BigDecimal intermediate) { // DAO
int width = Decimal38SparseHolder.WIDTH;
if (index >= nullableDecimal38Vector.getValueCapacity()) {
return false;
}
DecimalUtility.getSparseFromBigDecimal(intermediate, nullableDecimal38Vector.getBuffer(), index * width, schemaElement.getScale(),
schemaElement.getPrecision(), Decimal38SparseHolder.nDecimalDigits);
nullableDecimal38Vector.forceBits(index, 1);
return true;
}
-----Original Message-----
From: Abdel Hakim Deneche [mailto:adeneche@maprtech.com]
Sent: Friday, February 05, 2016 12:41 PM
To: dev@drill.apache.org
Subject: Re: strange build problem with generated sources
Do you also need to add forceBits to the template FixedValueVectors.java ?
Can you copy the line from VarLengthColumnReaders .java where you hit a compile error ?
On Fri, Feb 5, 2016 at 9:26 AM, Dave Oshinsky <do...@commvault.com>
wrote:
> Hi Abdel,
> Thanks for reply. I added it to
> exec/vector/target/codegen/templates/NullableValueVectors.java. I
> then see it propagated by Freemarker to generated sources, as shown
> below signature. In Eclipse, I'm able to navigate from where I'm
> calling
> forceBits() to e.g. NullableDecimal28SparseVector.java, and I see
> forceBits() implemented there. However, Eclipse has the dreaded "red
> X" in VarLengthColumnReaders.java indicating an error, and the build also fails.
> Weird, huh?
>
> I do have to say, this whole "generated sources" business is a pain in
> the you know where ;-) In all seriousness, I would implement the
> whole decimal business MUCH more simply if writing it from scratch.
> Why the gazillion special cases for different widths (all with
> separate generated sources), for nullable and not, sparse or dense?
> Just treat them all the same, at least from where I'm sitting with a Parquet-centric perspective.
> And the memory usage would be MUCH lower by treating decimals ALWAYS
> as variable width. Most actual decimal numbers are way smaller than
> the precision would indicate, in many of my test cases.
>
> Dave Oshinsky
>
> $ find . -name "*.java" | xargs grep forceBits
> ./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal28Vector.forceBits(index, 1);
> ./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal38Vector.forceBits(index, 1);
> ./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal28Vector.forceBits(index, 1);
> ./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal38Vector.forceBits(index, 1);
> ./exec/vector/src/main/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/classes/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBigIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBitVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDateVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal18Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28DenseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28SparseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38DenseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38SparseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal9Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat4Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat8Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalDayVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalYearVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableSmallIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeStampVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTinyIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt1Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt2Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt4Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt8Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVar16CharVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarBinaryVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarCharVector.java:
> public void forceBits(int index, int value){ // DAO
>
> -----Original Message-----
> From: Abdel Hakim Deneche [mailto:adeneche@maprtech.com]
> Sent: Friday, February 05, 2016 12:10 PM
> To: dev@drill.apache.org
> Subject: Re: strange build problem with generated sources
>
> Hey Dave,
>
> Which file did you add the "forceBits()" method to ?
>
> On Fri, Feb 5, 2016 at 8:33 AM, Dave Oshinsky
> <do...@commvault.com>
> wrote:
>
> > Hi Drill-ers,
> > I am looking into fixing JIRA
> > https://issues.apache.org/jira/browse/DRILL-4184. I've encountered
> > a number of strange build problems along the way with my drill 1.4
> > snapshot, including inability to rebuild after running "mvn clean",
> > no matter what I try. So, I'm building from scratch for the second
> > time, at least. The latest problem really has me stumped at the
> > moment. I added a "forceBits(int,int)" method that I see in
> > generated source file NullableDecimal28SparseVector.java (and
> > NullableDecimal38SparseVector.java), but somehow this doesn't get
> > compiled properly into the *.class and my build keeps failing as if
> > the new forceBits method isn't there:
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.2:compile
> > (default-compile) on project drill-java-exec: Compilation failure:
> > Compilation failure:
> > [ERROR]
> > C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src
> > \m
> > ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLength
> > Co
> > lumnReaders.java:[108,29]
> > error: cannot find symbol
> > [ERROR] symbol: method forceBits(int,int)
> > [ERROR] location: variable nullableDecimal28Vector of type
> > NullableDecimal28SparseVector [ERROR]
> > C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src
> > \m
> > ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLength
> > Co
> > lumnReaders.java:[179,29]
> > error: cannot find symbol
> >
> > Can anyone suggest how to fix this without starting over from
> > scratch in a new build node (again)? Any advice would be greatly appreciated.
> >
> > I will send a separate email eventually regarding the design of my
> > fix, which I know is only a short-term solution to the problem of
> > handling variable width decimal fields in Parquet files. To make a
> > long story short, all the decimal vectors are fixed width vectors,
> > which don't have the ability to "remember" varying sizes from one
> > decimal field to the next. I've hacked up something to "remember"
> > the varying field sizes (BigDecimal array sizes) in
> > NullableVarLengthValuesColumn and VarLengthValuesColumn, not in the
> > decimal vectors. This seems to work, though it's admittedly ugly.
> > However, I ran into a problem with nullable varying width decimal
> > columns where the "isSet" always returns 0, as if the column is
> > null, when it is not, and the sparse decimal data is present in the
> > vector (but Drill won't send the decimal value, because it thinks it's null).
> > Hence the "forceBits" hack to try to work around this. It seemed
> > like I was close to running a successful Drill query on the varying
> > width decimal Parquet data, but alas, I ran into (another) build
> > problem. I do have a LOT of questions as to why the decimal stuff
> > was designed the
> way it is, but that's for another email....
> >
> > Thanks,
> > Dave Oshinsky
> >
> >
> >
> >
> >
> > ***************************Legal
> > Disclaimer***************************
> > "This communication may contain confidential and privileged material
> > for the sole use of the intended recipient. Any unauthorized review,
> > use or distribution by others is strictly prohibited. If you have
> > received the message by mistake, please advise the sender by reply
> > email and delete the message. Thank you."
> > ********************************************************************
> > **
>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm
> _campaign=Free%20available
> >
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material
> for the sole use of the intended recipient. Any unauthorized review,
> use or distribution by others is strictly prohibited. If you have
> received the message by mistake, please advise the sender by reply
> email and delete the message. Thank you."
> **********************************************************************
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
Re: strange build problem with generated sources
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
Do you also need to add forceBits to the template FixedValueVectors.java ?
Can you copy the line from VarLengthColumnReaders .java where you hit a
compile error ?
On Fri, Feb 5, 2016 at 9:26 AM, Dave Oshinsky <do...@commvault.com>
wrote:
> Hi Abdel,
> Thanks for reply. I added it to
> exec/vector/target/codegen/templates/NullableValueVectors.java. I then see
> it propagated by Freemarker to generated sources, as shown below
> signature. In Eclipse, I'm able to navigate from where I'm calling
> forceBits() to e.g. NullableDecimal28SparseVector.java, and I see
> forceBits() implemented there. However, Eclipse has the dreaded "red X" in
> VarLengthColumnReaders.java indicating an error, and the build also fails.
> Weird, huh?
>
> I do have to say, this whole "generated sources" business is a pain in the
> you know where ;-) In all seriousness, I would implement the whole
> decimal business MUCH more simply if writing it from scratch. Why the
> gazillion special cases for different widths (all with separate generated
> sources), for nullable and not, sparse or dense? Just treat them all the
> same, at least from where I'm sitting with a Parquet-centric perspective.
> And the memory usage would be MUCH lower by treating decimals ALWAYS as
> variable width. Most actual decimal numbers are way smaller than the
> precision would indicate, in many of my test cases.
>
> Dave Oshinsky
>
> $ find . -name "*.java" | xargs grep forceBits
> ./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal28Vector.forceBits(index, 1);
> ./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal38Vector.forceBits(index, 1);
> ./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal28Vector.forceBits(index, 1);
> ./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java:
> nullableDecimal38Vector.forceBits(index, 1);
> ./exec/vector/src/main/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/classes/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/codegen/templates/NullableValueVectors.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBigIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBitVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDateVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal18Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28DenseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28SparseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38DenseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38SparseVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal9Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat4Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat8Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalDayVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalYearVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableSmallIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeStampVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTinyIntVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt1Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt2Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt4Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt8Vector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVar16CharVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarBinaryVector.java:
> public void forceBits(int index, int value){ // DAO
> ./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarCharVector.java:
> public void forceBits(int index, int value){ // DAO
>
> -----Original Message-----
> From: Abdel Hakim Deneche [mailto:adeneche@maprtech.com]
> Sent: Friday, February 05, 2016 12:10 PM
> To: dev@drill.apache.org
> Subject: Re: strange build problem with generated sources
>
> Hey Dave,
>
> Which file did you add the "forceBits()" method to ?
>
> On Fri, Feb 5, 2016 at 8:33 AM, Dave Oshinsky <do...@commvault.com>
> wrote:
>
> > Hi Drill-ers,
> > I am looking into fixing JIRA
> > https://issues.apache.org/jira/browse/DRILL-4184. I've encountered a
> > number of strange build problems along the way with my drill 1.4
> > snapshot, including inability to rebuild after running "mvn clean", no
> > matter what I try. So, I'm building from scratch for the second time,
> > at least. The latest problem really has me stumped at the moment. I
> > added a "forceBits(int,int)" method that I see in generated source
> > file NullableDecimal28SparseVector.java (and
> > NullableDecimal38SparseVector.java), but somehow this doesn't get
> > compiled properly into the *.class and my build keeps failing as if
> > the new forceBits method isn't there:
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.2:compile
> > (default-compile) on project drill-java-exec: Compilation failure:
> > Compilation failure:
> > [ERROR]
> > C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\m
> > ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthCo
> > lumnReaders.java:[108,29]
> > error: cannot find symbol
> > [ERROR] symbol: method forceBits(int,int)
> > [ERROR] location: variable nullableDecimal28Vector of type
> > NullableDecimal28SparseVector [ERROR]
> > C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\m
> > ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthCo
> > lumnReaders.java:[179,29]
> > error: cannot find symbol
> >
> > Can anyone suggest how to fix this without starting over from scratch
> > in a new build node (again)? Any advice would be greatly appreciated.
> >
> > I will send a separate email eventually regarding the design of my
> > fix, which I know is only a short-term solution to the problem of
> > handling variable width decimal fields in Parquet files. To make a
> > long story short, all the decimal vectors are fixed width vectors,
> > which don't have the ability to "remember" varying sizes from one
> > decimal field to the next. I've hacked up something to "remember" the
> > varying field sizes (BigDecimal array sizes) in
> > NullableVarLengthValuesColumn and VarLengthValuesColumn, not in the
> > decimal vectors. This seems to work, though it's admittedly ugly.
> > However, I ran into a problem with nullable varying width decimal
> > columns where the "isSet" always returns 0, as if the column is null,
> > when it is not, and the sparse decimal data is present in the vector
> > (but Drill won't send the decimal value, because it thinks it's null).
> > Hence the "forceBits" hack to try to work around this. It seemed like
> > I was close to running a successful Drill query on the varying width
> > decimal Parquet data, but alas, I ran into (another) build problem. I
> > do have a LOT of questions as to why the decimal stuff was designed the
> way it is, but that's for another email....
> >
> > Thanks,
> > Dave Oshinsky
> >
> >
> >
> >
> >
> > ***************************Legal Disclaimer***************************
> > "This communication may contain confidential and privileged material
> > for the sole use of the intended recipient. Any unauthorized review,
> > use or distribution by others is strictly prohibited. If you have
> > received the message by mistake, please advise the sender by reply
> > email and delete the message. Thank you."
> > **********************************************************************
>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **********************************************************************
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
RE: strange build problem with generated sources
Posted by Dave Oshinsky <do...@commvault.com>.
Hi Abdel,
Thanks for reply. I added it to exec/vector/target/codegen/templates/NullableValueVectors.java. I then see it propagated by Freemarker to generated sources, as shown below signature. In Eclipse, I'm able to navigate from where I'm calling forceBits() to e.g. NullableDecimal28SparseVector.java, and I see forceBits() implemented there. However, Eclipse has the dreaded "red X" in VarLengthColumnReaders.java indicating an error, and the build also fails. Weird, huh?
I do have to say, this whole "generated sources" business is a pain in the you know where ;-) In all seriousness, I would implement the whole decimal business MUCH more simply if writing it from scratch. Why the gazillion special cases for different widths (all with separate generated sources), for nullable and not, sparse or dense? Just treat them all the same, at least from where I'm sitting with a Parquet-centric perspective. And the memory usage would be MUCH lower by treating decimals ALWAYS as variable width. Most actual decimal numbers are way smaller than the precision would indicate, in many of my test cases.
Dave Oshinsky
$ find . -name "*.java" | xargs grep forceBits
./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java: nullableDecimal28Vector.forceBits(index, 1);
./exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java: nullableDecimal38Vector.forceBits(index, 1);
./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java: nullableDecimal28Vector.forceBits(index, 1);
./exec/java-exec/target/classes/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java: nullableDecimal38Vector.forceBits(index, 1);
./exec/vector/src/main/codegen/templates/NullableValueVectors.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/classes/codegen/templates/NullableValueVectors.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/codegen/templates/NullableValueVectors.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBigIntVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableBitVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDateVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal18Vector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28DenseVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal28SparseVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38DenseVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal38SparseVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableDecimal9Vector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat4Vector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableFloat8Vector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalDayVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntervalYearVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableIntVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableSmallIntVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeStampVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTimeVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableTinyIntVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt1Vector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt2Vector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt4Vector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableUInt8Vector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVar16CharVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarBinaryVector.java: public void forceBits(int index, int value){ // DAO
./exec/vector/target/generated-sources/org/apache/drill/exec/vector/NullableVarCharVector.java: public void forceBits(int index, int value){ // DAO
-----Original Message-----
From: Abdel Hakim Deneche [mailto:adeneche@maprtech.com]
Sent: Friday, February 05, 2016 12:10 PM
To: dev@drill.apache.org
Subject: Re: strange build problem with generated sources
Hey Dave,
Which file did you add the "forceBits()" method to ?
On Fri, Feb 5, 2016 at 8:33 AM, Dave Oshinsky <do...@commvault.com>
wrote:
> Hi Drill-ers,
> I am looking into fixing JIRA
> https://issues.apache.org/jira/browse/DRILL-4184. I've encountered a
> number of strange build problems along the way with my drill 1.4
> snapshot, including inability to rebuild after running "mvn clean", no
> matter what I try. So, I'm building from scratch for the second time,
> at least. The latest problem really has me stumped at the moment. I
> added a "forceBits(int,int)" method that I see in generated source
> file NullableDecimal28SparseVector.java (and
> NullableDecimal38SparseVector.java), but somehow this doesn't get
> compiled properly into the *.class and my build keeps failing as if
> the new forceBits method isn't there:
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.2:compile
> (default-compile) on project drill-java-exec: Compilation failure:
> Compilation failure:
> [ERROR]
> C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\m
> ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthCo
> lumnReaders.java:[108,29]
> error: cannot find symbol
> [ERROR] symbol: method forceBits(int,int)
> [ERROR] location: variable nullableDecimal28Vector of type
> NullableDecimal28SparseVector [ERROR]
> C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\m
> ain\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthCo
> lumnReaders.java:[179,29]
> error: cannot find symbol
>
> Can anyone suggest how to fix this without starting over from scratch
> in a new build node (again)? Any advice would be greatly appreciated.
>
> I will send a separate email eventually regarding the design of my
> fix, which I know is only a short-term solution to the problem of
> handling variable width decimal fields in Parquet files. To make a
> long story short, all the decimal vectors are fixed width vectors,
> which don't have the ability to "remember" varying sizes from one
> decimal field to the next. I've hacked up something to "remember" the
> varying field sizes (BigDecimal array sizes) in
> NullableVarLengthValuesColumn and VarLengthValuesColumn, not in the
> decimal vectors. This seems to work, though it's admittedly ugly.
> However, I ran into a problem with nullable varying width decimal
> columns where the "isSet" always returns 0, as if the column is null,
> when it is not, and the sparse decimal data is present in the vector
> (but Drill won't send the decimal value, because it thinks it's null).
> Hence the "forceBits" hack to try to work around this. It seemed like
> I was close to running a successful Drill query on the varying width
> decimal Parquet data, but alas, I ran into (another) build problem. I
> do have a LOT of questions as to why the decimal stuff was designed the way it is, but that's for another email....
>
> Thanks,
> Dave Oshinsky
>
>
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material
> for the sole use of the intended recipient. Any unauthorized review,
> use or distribution by others is strictly prohibited. If you have
> received the message by mistake, please advise the sender by reply
> email and delete the message. Thank you."
> **********************************************************************
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
Re: strange build problem with generated sources
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
Hey Dave,
Which file did you add the "forceBits()" method to ?
On Fri, Feb 5, 2016 at 8:33 AM, Dave Oshinsky <do...@commvault.com>
wrote:
> Hi Drill-ers,
> I am looking into fixing JIRA
> https://issues.apache.org/jira/browse/DRILL-4184. I've encountered a
> number of strange build problems along the way with my drill 1.4 snapshot,
> including inability to rebuild after running "mvn clean", no matter what I
> try. So, I'm building from scratch for the second time, at least. The
> latest problem really has me stumped at the moment. I added a
> "forceBits(int,int)" method that I see in generated source file
> NullableDecimal28SparseVector.java (and
> NullableDecimal38SparseVector.java), but somehow this doesn't get compiled
> properly into the *.class and my build keeps failing as if the new
> forceBits method isn't there:
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.2:compile
> (default-compile) on project drill-java-exec: Compilation failure:
> Compilation failure:
> [ERROR]
> C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\main\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthColumnReaders.java:[108,29]
> error: cannot find symbol
> [ERROR] symbol: method forceBits(int,int)
> [ERROR] location: variable nullableDecimal28Vector of type
> NullableDecimal28SparseVector
> [ERROR]
> C:\apache\apache-drill-1.4.0\rebuild2\drill-1.4.0\exec\java-exec\src\main\java\org\apache\drill\exec\store\parquet\columnreaders\VarLengthColumnReaders.java:[179,29]
> error: cannot find symbol
>
> Can anyone suggest how to fix this without starting over from scratch in a
> new build node (again)? Any advice would be greatly appreciated.
>
> I will send a separate email eventually regarding the design of my fix,
> which I know is only a short-term solution to the problem of handling
> variable width decimal fields in Parquet files. To make a long story
> short, all the decimal vectors are fixed width vectors, which don't have
> the ability to "remember" varying sizes from one decimal field to the
> next. I've hacked up something to "remember" the varying field sizes
> (BigDecimal array sizes) in NullableVarLengthValuesColumn and
> VarLengthValuesColumn, not in the decimal vectors. This seems to work,
> though it's admittedly ugly. However, I ran into a problem with nullable
> varying width decimal columns where the "isSet" always returns 0, as if the
> column is null, when it is not, and the sparse decimal data is present in
> the vector (but Drill won't send the decimal value, because it thinks it's
> null). Hence the "forceBits" hack to try to work around this. It seemed
> like I was close to running a successful Drill query on the varying width
> decimal Parquet data, but alas, I ran into (another) build problem. I do
> have a LOT of questions as to why the decimal stuff was designed the way it
> is, but that's for another email....
>
> Thanks,
> Dave Oshinsky
>
>
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **********************************************************************
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>