You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/12/16 11:16:30 UTC
[GitHub] [incubator-druid] Maplejw opened a new issue #9048: druid index on
dataproc
Maplejw opened a new issue #9048: druid index on dataproc
URL: https://github.com/apache/incubator-druid/issues/9048
My yarn runs on dataproc,and storage by gcs.
When I post an index task, The log shows the mapreduce job completed,but hash fail,it says the
```code
Path[var/druid/hadoop-tmp/cm_client_behavior/2019-12-16T104735.560Z_00c53a90fdb54ae2b6315981f4afd9fd/intervals.json] didn't exist
```
But the file is generated by druid,not me.so why?
The index task config
```code
{
"type": "index_hadoop",
"spec": {
"dataSchema": {
"dataSource": "cm-client-behavior",
"parser": {
"type": "hadoopyString",
"parseSpec": {
"format": "json",
"dimensionsSpec": {
"dimensions": [
"device_id",
"os",
"country",
"device_model",
"sys_lang",
"network_type",
"sys_version",
"app_version",
"behavior_id",
"behavior_desc"
]
},
"timestampSpec": {
"column": "proc_time"
}
}
},
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "day",
"rollup": true,
"queryGranularity": "none"
},
"metricsSpec": [
{"type" : "count","name":"count"}
]
},
"ioConfig": {
"type": "hadoop",
"inputSpec": {
"type":"static",
"paths": "gs://xxxxxxx/cm/dw/dws/client_behavior/proc_date=20191215/*"
},
"appendToExisting": false
},
"tuningConfig": {
"type" : "hadoop",
"partitionsSpec" : {
"type" : "hashed",
"targetPartitionSize" : 5000000
},
"forceExtendableShardSpecs" : true,
"jobProperties" : {
}
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org