You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datafu.apache.org by "jian wang (JIRA)" <ji...@apache.org> on 2014/03/02 13:37:19 UTC
[jira] [Commented] (DATAFU-33) Can't instantiate AliasableEvalFunc
for unit-testing
[ https://issues.apache.org/jira/browse/DATAFU-33?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917387#comment-13917387 ]
jian wang commented on DATAFU-33:
---------------------------------
In the test() method, please change this line:
udf.getOutputSchema(schemaBag)
to
udf.outputSchema(schemaBag).
and try again.
The outputSchema() function calls storeFieldAliases() to set up the alias map.
I have also created a unit test of the AliasEvalFunc based on your use case, please check: https://reviews.apache.org/r/18670/
> Can't instantiate AliasableEvalFunc for unit-testing
> ----------------------------------------------------
>
> Key: DATAFU-33
> URL: https://issues.apache.org/jira/browse/DATAFU-33
> Project: DataFu
> Issue Type: Bug
> Affects Versions: 1.3.0
> Environment: CDH 4.4
> Reporter: Sergey
>
> Hi, I can't make it work for 1.2 version too. Works perfectly inside script, can't instantiate in for unit-testing.
> {code}
> @Test
> void test(){
> def udf = new ReportBuilder('/any/path/to/2013/10/21', getBasePath('Partition_dict.csv'))
> udf.setUDFContextSignature('test')
> def schemaTuple = new Schema([
> new Schema.FieldSchema('msisdn', DataType.LONG),
> new Schema.FieldSchema('ts', DataType.INTEGER),
> new Schema.FieldSchema('center_lon', DataType.DOUBLE),
> new Schema.FieldSchema('center_lat', DataType.DOUBLE),
> ])
> def schemaBag = new Schema(new Schema.FieldSchema('orderedRoutes', schemaTuple, DataType.BAG))
> udf.getOutputSchema(schemaBag)
> udf.exec(TupleFactory.instance.newTuple([71230000000L, 1382351612, 10.697D, 20.713D]))
> }
> /** A stub for testing files taken from distcache */
> def getBasePath(String fileName){
> FileUtils.toFile(this.class.classLoader.getResource(fileName)).parentFile.absolutePath
> }
> {code}
> Script usage:
> {code}
> orderedRoutes = ORDER routes BY ts;
> GENERATE FLATTEN(ReportBuilder(orderedRoutes)) as (list_of_fields:anytype);
> {code}
> fails here:
> {code}
> @Override
> DataBag exec(Tuple input) throws IOException {
> def pivots = getBag(input, ORDERED_ROUTES).toList() //error happens here
> def outputBag = BagFactory.instance.newDefaultBag()
> //some code goes here
> outputBag
> }
> {code}
> {code}
> 14/02/24 17:55:40 ERROR udf.ReportBuilder: Class: class pig.udf.ReportBuilder
> 14/02/24 17:55:40 ERROR udf.ReportBuilder: Instance name: test
> 14/02/24 17:55:40 ERROR udf.ReportBuilder: Properties: {test={}}
> java.lang.RuntimeException: Could not retrieve aliases from properties using aliasMap
> at datafu.pig.util.AliasableEvalFunc.getFieldAliases(AliasableEvalFunc.java:164)
> at datafu.pig.util.AliasableEvalFunc.getPosition(AliasableEvalFunc.java:171)
> at datafu.pig.util.AliasableEvalFunc.getBag(AliasableEvalFunc.java:253)
> at datafu.pig.util.AliasableEvalFunc$getBag.callCurrent(Unknown Source)
> at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
> at udf.ReportBuilder.exec(ReportBuilder.groovy:26)
> at pig.udf.ReportBuilder$exec.call(Unknown Source)
> at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
> at udf.ReportBuilderTest.test(ReportBuilderTest.groovy:28)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)