You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Julien Le Dem (JIRA)" <ji...@apache.org> on 2012/06/20 19:00:44 UTC
[jira] [Commented] (PIG-2478) Using Illustrate with Javascript UDF
throws NEP
[ https://issues.apache.org/jira/browse/PIG-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397629#comment-13397629 ]
Julien Le Dem commented on PIG-2478:
------------------------------------
The problem is that the input schema is not set in that case.
i.e.: Pig does not seem to call outputSchema in that case.
> Using Illustrate with Javascript UDF throws NEP
> -----------------------------------------------
>
> Key: PIG-2478
> URL: https://issues.apache.org/jira/browse/PIG-2478
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.11
> Environment: Mac OSX Lion 10.7.2, Hadoop (0.20.205.0), Pig (Apache Pig version 0.11.0-SNAPSHOT (r1230844).
> Reporter: dan young
>
> When I try to run illustrate while using a Javascript UDF, I always get a NEP....but if I do a dump/store, everything works fine.
> dans-MacBook-Pro:pig danoyoung$ pig -x local
> 2012-01-12 22:16:57,131 [main] INFO org.apache.pig.Main - Logging error messages to: /Users/danoyoung/projects/pig/pig_1326431817128.log
> 2012-01-12 22:16:57,386 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
> 2012-01-12 22:16:57.483 java[27809:1903] Unable to load realm info from SCDynamicStore
> grunt> set io.sort.mb 500;
> grunt> register '/Users/danoyoung/projects/pig/udf/udf.js' using org.apache.pig.scripting.js.JsScriptEngine as myfuncs;
> 2012-01-12 22:17:24,803 [main] INFO org.apache.pig.scripting.js.JsScriptEngine - Register scripting UDF: get_date_marker
> 2012-01-12 22:17:24,804 [main] INFO org.apache.pig.scripting.js.JsScriptEngine - Register scripting UDF: get_record
> grunt> register '/usr/local/pig/piggybank.jar';
> grunt>
> grunt> a = LOAD '/Users/danoyoung/Downloads/adwords_KeywordDailyReport_1-1-2012_daily' USING org.apache.pig.piggybank.storage.XMLLoader('row') AS (doc:chararray);
> grunt>
> grunt> b = FOREACH a GENERATE FLATTEN(myfuncs.get_record(doc)) AS (dw_date_marker:int,ad_network_ad_group_key:long,ad_network_keyword_key:long,firstpagecpc:int,qualityscore:int,cost:float,position:float);
> grunt> describe b;
> b: {dw_date_marker: int,ad_network_ad_group_key: long,ad_network_keyword_key: long,firstpagecpc: int,qualityscore: int,cost: float,position: float}
> grunt>
> grunt> illustrate b;
> 2012-01-12 22:17:25,642 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
> 2012-01-12 22:17:25,846 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2012-01-12 22:17:25,860 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
> 2012-01-12 22:17:25,860 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
> 2012-01-12 22:17:25,870 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
> 2012-01-12 22:17:25,887 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-01-12 22:17:26,161 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
> 2012-01-12 22:17:26,721 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2012-01-12 22:17:26,723 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
> 2012-01-12 22:17:26,723 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
> 2012-01-12 22:17:26,724 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
> 2012-01-12 22:17:26,725 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> java.lang.NullPointerException
> at org.apache.pig.scripting.js.JsFunction.exec(JsFunction.java:200)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:225)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:262)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:332)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:284)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:271)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:266)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:194)
> at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
> at org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
> at org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:98)
> at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
> at org.apache.pig.PigServer.getExamples(PigServer.java:1202)
> at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> at org.apache.pig.Main.run(Main.java:523)
> at org.apache.pig.Main.main(Main.java:148)
> 2012-01-12 22:17:26,767 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered IOException. Exception : null
> Details at logfile: /Users/danoyoung/projects/pig/pig_1326431817128.log
> grunt>
> Here's the log details:
> Pig Stack Trace
> ---------------
> ERROR 2997: Encountered IOException. Exception : null
> java.io.IOException: Exception : null
> at org.apache.pig.PigServer.getExamples(PigServer.java:1208)
> at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> at org.apache.pig.Main.run(Main.java:523)
> at org.apache.pig.Main.main(Main.java:148)
> ================================================================================
> Here's the Javascript UDF:
> get_record.outputSchema = "dw_date_marker:int,ad_network_ad_group_key:long,ad_network_keyword_key:long,firstpagecpc:int,qualityscore:int,cost:float,position:float";
> function get_record(doc){
> var rec = new Object;
> rec.dt = doc.substring(doc.search(/date=/ig)+6,doc.search(/\" campaignid=/ig));
> //rec.campaign_id = doc.substring(doc.search(/campaignid=/ig)+12,doc.search(/\" adgroupid=/ig));
> rec.ad_network_ad_group_key = doc.substring(doc.search(/adgroupid=/ig)+11,doc.search(/\" keywordid=/ig));
> rec.ad_network_keyword_key = doc.substring(doc.search(/keywordid=/ig)+11,doc.search(/\" keyword=/ig));
> rec.firstpagecpc = doc.substring(doc.search(/firstpagecpc=/ig)+14,doc.search(/\" qualityscore=/ig))/1000000;
> rec.qualityscore = doc.substring(doc.search(/qualityscore=/ig)+14,doc.search(/\" imps=/ig));
> rec.cost = doc.substring(doc.search(/cost=/ig)+6,doc.search(/\" pos=/ig))/1000000;
> rec.position = doc.substring(doc.search(/pos=/ig)+5,doc.search(/\"><\/row/ig));
> rec.dw_date_marker = get_date_marker(rec.dt);
>
> return {dw_date_marker:rec.dw_date_marker,
> ad_network_ad_group_key:rec.ad_network_ad_group_key,
> ad_network_keyword_key:rec.ad_network_keyword_key,
> firstpagecpc:rec.firstpagecpc,
> qualityscore:rec.qualityscore,
> cost:rec.cost,
> position:rec.position};
> }
> function get_date_marker(dt){
> var dateMarkers = {
> _2012_01_01:1204,
> _2012_01_02:1205,
> _2012_01_03:1206,
> _2012_01_04:1207,
> _2012_01_05:1208,
> _2012_01_06:1209,
> _2012_01_07:1210,
> _2012_01_08:1211,
> _2012_01_09:1212,
> _2012_01_10:1213,
> _2012_01_11:1214,
> _2012_01_12:1215,
> _2012_01_13:1216,
> _2012_01_14:1217,
> _2012_01_15:1218,
> _2012_01_16:1219,
> _2012_01_17:1220,
> _2012_01_18:1221,
> _2012_01_19:1222,
> _2012_01_20:1223,
> _2012_01_21:1224,
> _2012_01_22:1225,
> _2012_01_23:1226,
> _2012_01_24:1227,
> _2012_01_25:1228,
> _2012_01_26:1229,
> _2012_01_27:1230,
> _2012_01_28:1231,
> _2012_01_29:1232,
> _2012_01_30:1233,
> _2012_01_31:1234,
> _2012_02_01:1235,
> _2012_02_02:1236,
> _2012_02_03:1237,
> _2012_02_04:1238,
> _2012_02_05:1239,
> _2012_02_06:1240,
> _2012_02_07:1241,
> _2012_02_08:1242,
> _2012_02_09:1243,
> _2012_02_10:1244,
> _2012_02_11:1245,
> _2012_02_12:1246,
> _2012_02_13:1247,
> _2012_02_14:1248,
> _2012_02_15:1249,
> _2012_02_16:1250,
> _2012_02_17:1251,
> _2012_02_18:1252,
> _2012_02_19:1253,
> _2012_02_20:1254,
> _2012_02_21:1255,
> _2012_02_22:1256,
> _2012_02_23:1257,
> _2012_02_24:1258,
> _2012_02_25:1259,
> _2012_02_26:1260,
> _2012_02_27:1261,
> _2012_02_28:1262,
> _2012_02_29:1263,
> _2012_03_01:1264,
> _2012_03_02:1265,
> _2012_03_03:1266,
> _2012_03_04:1267,
> _2012_03_05:1268,
> _2012_03_06:1269,
> _2012_03_07:1270,
> _2012_03_08:1271,
> _2012_03_09:1272,
> _2012_03_10:1273,
> _2012_03_11:1274,
> _2012_03_12:1275,
> _2012_03_13:1276,
> _2012_03_14:1277,
> _2012_03_15:1278,
> _2012_03_16:1279,
> _2012_03_17:1280,
> _2012_03_18:1281,
> _2012_03_19:1282,
> _2012_03_20:1283,
> _2012_03_21:1284,
> _2012_03_22:1285,
> _2012_03_23:1286,
> _2012_03_24:1287,
> _2012_03_25:1288,
> _2012_03_26:1289,
> _2012_03_27:1290,
> _2012_03_28:1291,
> _2012_03_29:1292,
> _2012_03_30:1293,
> _2012_03_31:1294,
> _2012_04_01:1295,
> _2012_04_02:1296,
> _2012_04_03:1297,
> _2012_04_04:1298,
> _2012_04_05:1299,
> _2012_04_06:1300,
> _2012_04_07:1301,
> _2012_04_08:1302,
> _2012_04_09:1303,
> _2012_04_10:1304,
> _2012_04_11:1305,
> _2012_04_12:1306,
> _2012_04_13:1307,
> _2012_04_14:1308,
> _2012_04_15:1309,
> _2012_04_16:1310,
> _2012_04_17:1311,
> _2012_04_18:1312,
> _2012_04_19:1313,
> _2012_04_20:1314,
> _2012_04_21:1315,
> _2012_04_22:1316,
> _2012_04_23:1317,
> _2012_04_24:1318,
> _2012_04_25:1319,
> _2012_04_26:1320,
> _2012_04_27:1321,
> _2012_04_28:1322,
> _2012_04_29:1323,
> _2012_04_30:1324,
> _2012_05_01:1325,
> _2012_05_02:1326,
> _2012_05_03:1327,
> _2012_05_04:1328,
> _2012_05_05:1329,
> _2012_05_06:1330,
> _2012_05_07:1331,
> _2012_05_08:1332,
> _2012_05_09:1333,
> _2012_05_10:1334,
> _2012_05_11:1335,
> _2012_05_12:1336,
> _2012_05_13:1337,
> _2012_05_14:1338,
> _2012_05_15:1339,
> _2012_05_16:1340,
> _2012_05_17:1341,
> _2012_05_18:1342,
> _2012_05_19:1343,
> _2012_05_20:1344,
> _2012_05_21:1345,
> _2012_05_22:1346,
> _2012_05_23:1347,
> _2012_05_24:1348,
> _2012_05_25:1349,
> _2012_05_26:1350,
> _2012_05_27:1351,
> _2012_05_28:1352,
> _2012_05_29:1353,
> _2012_05_30:1354,
> _2012_05_31:1355,
> _2012_06_01:1356,
> _2012_06_02:1357,
> _2012_06_03:1358,
> _2012_06_04:1359,
> _2012_06_05:1360,
> _2012_06_06:1361,
> _2012_06_07:1362,
> _2012_06_08:1363,
> _2012_06_09:1364,
> _2012_06_10:1365,
> _2012_06_11:1366,
> _2012_06_12:1367,
> _2012_06_13:1368,
> _2012_06_14:1369,
> _2012_06_15:1370,
> _2012_06_16:1371,
> _2012_06_17:1372,
> _2012_06_18:1373,
> _2012_06_19:1374,
> _2012_06_20:1375,
> _2012_06_21:1376,
> _2012_06_22:1377,
> _2012_06_23:1378,
> _2012_06_24:1379,
> _2012_06_25:1380,
> _2012_06_26:1381,
> _2012_06_27:1382,
> _2012_06_28:1383,
> _2012_06_29:1384,
> _2012_06_30:1385,
> _2012_07_01:1386,
> _2012_07_02:1387,
> _2012_07_03:1388,
> _2012_07_04:1389,
> _2012_07_05:1390,
> _2012_07_06:1391,
> _2012_07_07:1392,
> _2012_07_08:1393,
> _2012_07_09:1394,
> _2012_07_10:1395,
> _2012_07_11:1396,
> _2012_07_12:1397,
> _2012_07_13:1398,
> _2012_07_14:1399,
> _2012_07_15:1400,
> _2012_07_16:1401,
> _2012_07_17:1402,
> _2012_07_18:1403,
> _2012_07_19:1404,
> _2012_07_20:1405,
> _2012_07_21:1406,
> _2012_07_22:1407,
> _2012_07_23:1408,
> _2012_07_24:1409,
> _2012_07_25:1410,
> _2012_07_26:1411,
> _2012_07_27:1412,
> _2012_07_28:1413,
> _2012_07_29:1414,
> _2012_07_30:1415,
> _2012_07_31:1416,
> _2012_08_01:1417,
> _2012_08_02:1418,
> _2012_08_03:1419,
> _2012_08_04:1420,
> _2012_08_05:1421,
> _2012_08_06:1422,
> _2012_08_07:1423,
> _2012_08_08:1424,
> _2012_08_09:1425,
> _2012_08_10:1426,
> _2012_08_11:1427,
> _2012_08_12:1428,
> _2012_08_13:1429,
> _2012_08_14:1430,
> _2012_08_15:1431,
> _2012_08_16:1432,
> _2012_08_17:1433,
> _2012_08_18:1434,
> _2012_08_19:1435,
> _2012_08_20:1436,
> _2012_08_21:1437,
> _2012_08_22:1438,
> _2012_08_23:1439,
> _2012_08_24:1440,
> _2012_08_25:1441,
> _2012_08_26:1442,
> _2012_08_27:1443,
> _2012_08_28:1444,
> _2012_08_29:1445,
> _2012_08_30:1446,
> _2012_08_31:1447,
> _2012_09_01:1448,
> _2012_09_02:1449,
> _2012_09_03:1450,
> _2012_09_04:1451,
> _2012_09_05:1452,
> _2012_09_06:1453,
> _2012_09_07:1454,
> _2012_09_08:1455,
> _2012_09_09:1456,
> _2012_09_10:1457,
> _2012_09_11:1458,
> _2012_09_12:1459,
> _2012_09_13:1460,
> _2012_09_14:1461,
> _2012_09_15:1462,
> _2012_09_16:1463,
> _2012_09_17:1464,
> _2012_09_18:1465,
> _2012_09_19:1466,
> _2012_09_20:1467,
> _2012_09_21:1468,
> _2012_09_22:1469,
> _2012_09_23:1470,
> _2012_09_24:1471,
> _2012_09_25:1472,
> _2012_09_26:1473,
> _2012_09_27:1474,
> _2012_09_28:1475,
> _2012_09_29:1476,
> _2012_09_30:1477,
> _2012_10_01:1478,
> _2012_10_02:1479,
> _2012_10_03:1480,
> _2012_10_04:1481,
> _2012_10_05:1482,
> _2012_10_06:1483,
> _2012_10_07:1484,
> _2012_10_08:1485,
> _2012_10_09:1486,
> _2012_10_10:1487,
> _2012_10_11:1488,
> _2012_10_12:1489,
> _2012_10_13:1490,
> _2012_10_14:1491,
> _2012_10_15:1492,
> _2012_10_16:1493,
> _2012_10_17:1494,
> _2012_10_18:1495,
> _2012_10_19:1496,
> _2012_10_20:1497,
> _2012_10_21:1498,
> _2012_10_22:1499,
> _2012_10_23:1500,
> _2012_10_24:1501,
> _2012_10_25:1502,
> _2012_10_26:1503,
> _2012_10_27:1504,
> _2012_10_28:1505,
> _2012_10_29:1506,
> _2012_10_30:1507,
> _2012_10_31:1508,
> _2012_11_01:1509,
> _2012_11_02:1510,
> _2012_11_03:1511,
> _2012_11_04:1512,
> _2012_11_05:1513,
> _2012_11_06:1514,
> _2012_11_07:1515,
> _2012_11_08:1516,
> _2012_11_09:1517,
> _2012_11_10:1518,
> _2012_11_11:1519,
> _2012_11_12:1520,
> _2012_11_13:1521,
> _2012_11_14:1522,
> _2012_11_15:1523,
> _2012_11_16:1524,
> _2012_11_17:1525,
> _2012_11_18:1526,
> _2012_11_19:1527,
> _2012_11_20:1528,
> _2012_11_21:1529,
> _2012_11_22:1530,
> _2012_11_23:1531,
> _2012_11_24:1532,
> _2012_11_25:1533,
> _2012_11_26:1534,
> _2012_11_27:1535,
> _2012_11_28:1536,
> _2012_11_29:1537,
> _2012_11_30:1538,
> _2012_12_01:1539,
> _2012_12_02:1540,
> _2012_12_03:1541,
> _2012_12_04:1542,
> _2012_12_05:1543,
> _2012_12_06:1544,
> _2012_12_07:1545,
> _2012_12_08:1546,
> _2012_12_09:1547,
> _2012_12_10:1548,
> _2012_12_11:1549,
> _2012_12_12:1550,
> _2012_12_13:1551,
> _2012_12_14:1552,
> _2012_12_15:1553,
> _2012_12_16:1554,
> _2012_12_17:1555,
> _2012_12_18:1556,
> _2012_12_19:1557,
> _2012_12_20:1558,
> _2012_12_21:1559,
> _2012_12_22:1560,
> _2012_12_23:1561,
> _2012_12_24:1562,
> _2012_12_25:1563,
> _2012_12_26:1564,
> _2012_12_27:1565,
> _2012_12_28:1566,
> _2012_12_29:1567,
> _2012_12_30:1568,
> _2012_12_31:1569};
> var rec = new Object;
> rec.dt = dt.replace(/-/g, "_");
> rec.date_marker = dateMarkers["_" + rec.dt];
> return rec.date_marker;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira