You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@chukwa.apache.org by 华凯 <hu...@163.com> on 2010/03/02 02:45:01 UTC
why mysql db no data
hi,
I deployed chukwa+hadoop+hicc, and use chukwa collect /var/message log, not hadoop log
execute start-data-process.sh, CHUKWA_HOME/bin/downSampling.sh
and it looks like process normally
mysql db has extented tables like chukwaagent_2097_week ..etc
But there is no data in mysql db. Is this mysql db only store the data from hadoop log?
Or i missed something?
In dbAdmin.log
2010-03-02 01:34:29,788 INFO main Aggregator - Aggregator started.
2010-03-02 01:34:30,872 INFO main DataExpiration - DataExpiration for: 2010-03-02 365 finished: (1.217 seconds)
2010-03-02 01:34:31,431 INFO main DataExpiration - DataExpiration for: 2010-03-02 91 finished: (1.148 seconds)
2010-03-02 01:34:31,770 INFO main DataExpiration - DataExpiration for: 2010-03-02 3650 finished: (2.443 seconds)
2010-03-02 01:34:31,954 INFO main DataExpiration - DataExpiration for: 2010-03-02 7 finished: (1.796 seconds)
2010-03-02 01:34:32,028 INFO main DataExpiration - DataExpiration for: 2010-03-02 30 finished: (1.627 seconds)
2010-03-02 01:34:32,556 INFO main Aggregator - Longest running query: replace into [cluster_system_metrics] (select [avg(system_metrics)] from [system_metrics] where timestamp between '[past_15_minutes]' and '[past_5_minutes]' group by timestamp); (0.283 seconds)
2010-03-02 01:34:32,556 INFO main Aggregator - Total running time: (2.767 seconds)
2010-03-02 01:34:32,556 INFO main Aggregator - Aggregator finished.
in postProcess.log
2010-03-02 01:42:29,956 INFO main PostProcessorManager - PostProcess Start, directory:/chukwa/postProcess/demuxOutputDir_1267494140727
2010-03-02 01:42:29,957 INFO main PostProcessorManager - org.apache.hadoop.chukwa.dataloader.MetricDataLoaderPool processing: /chukwa/postProcess/demuxOutputDir_1267494140727
2010-03-02 01:42:29,959 INFO main MetricDataLoaderPool - Processing: hdfs://logminerdb:9000/chukwa/postProcess/demuxOutputDir_1267494140727/chukwa/SysLog/SysLog_20090302.D.evt
2010-03-02 01:42:29,959 INFO pool-1-thread-2 MetricDataLoader - StreamName: SysLog_20090302.D.evt
2010-03-02 01:42:30,093 INFO pool-1-thread-2 MetricDataLoader - Batch mode inserted=0records.
2010-03-02 01:42:30,093 INFO pool-1-thread-2 MetricDataLoader - batchMode commit done
2010-03-02 01:42:30,093 INFO pool-1-thread-2 MetricDataLoader - Saved (syslog,chukwa) 0 sec. numOfRecords: 10
2010-03-02 01:42:30,093 INFO main PostProcessorManager - loadData Duration:136
2010-03-02 01:42:30,093 INFO main ChukwaConfiguration - chukwaConf is /opt/chukwainstall/chukwaconf
2010-03-02 01:42:30,103 INFO main MoveToRepository - Start MoveToRepository main()
2010-03-02 01:42:30,105 INFO main MoveToRepository - main procesing Cluster (chukwa)
2010-03-02 01:42:30,105 INFO main MoveToRepository - processClutserDirectory (chukwa,/chukwa/repos//chukwa)
2010-03-02 01:42:30,106 INFO main MoveToRepository - hdfs://logminerdb:9000/chukwa/postProcess/demuxOutputDir_1267494140727/chukwa/SysLog isDir?true
2010-03-02 01:42:30,106 INFO main MoveToRepository - dest directory path: /chukwa/repos/chukwa/SysLog
2010-03-02 01:42:30,106 INFO main MoveToRepository - processClutserDirectory processing Datasource: (SysLog)
2010-03-02 01:42:30,106 INFO main MoveToRepository - processDatasourceDirectory processing RecordFile: (SysLog_20090302.D.evt)
2010-03-02 01:42:30,106 INFO main MoveToRepository - fileName: SysLog_20090302.D.evt
2010-03-02 01:42:30,106 INFO main MoveToRepository - Datasource: SysLog
2010-03-02 01:42:30,158 INFO main MoveToRepository - >>>>>>>>>>>> Before Renamehdfs://logminerdb:9000/chukwa/postProcess/demuxOutputDir_1267494140727/chukwa/SysLog/SysLog_20090302.D.evt -- /chukwa/repos/chukwa/SysLog/20090302/SysLog_20090302.77.evt
2010-03-02 01:42:30,160 INFO main MoveToRepository - >>>>>>>>>>>> after Rename/chukwa/repos/chukwa/SysLog/20090302/SysLog_20090302.77.evt , rename:true
2010-03-02 01:42:30,161 INFO main MoveToRepository - Done with MoveToRepository main()
2010-03-02 01:42:30,161 INFO main PostProcessorManager - movetoMainRepository Duration:68
2010-03-02 01:42:30,162 INFO main PostProcessorManager - PostProcess Stop, directory:/chukwa/postProcess/demuxOutputDir_1267494140727
2010-03-02 01:42:30,162 INFO main PostProcessorManager - processDemuxOutput Duration:205
Re: why mysql db no data
Posted by Eric Yang <ey...@yahoo-inc.com>.
Hi,
/var/message is not stored into mysql. The only data goes into mysql are
the system metrics and hadoop metrics. System metrics is composed of top,
iostat, sar, and df. You need to start up systemMetrics.sh to get system
related data injected into mysql.
Regards,
Eric
On 3/1/10 5:45 PM, "华凯" <hu...@163.com> wrote:
> hi,
> I deployed chukwa+hadoop+hicc, and use chukwa collect /var/message log, not
> hadoop log
>
> execute start-data-process.sh, CHUKWA_HOME/bin/downSampling.sh
> and it looks like process normally
> mysql db has extented tables like chukwaagent_2097_week ..etc
>
> But there is no data in mysql db. Is this mysql db only store the data from
> hadoop log?
> Or i missed something?
>
> In dbAdmin.log
> 2010-03-02 01:34:29,788 INFO main Aggregator - Aggregator started.
> 2010-03-02 01:34:30,872 INFO main DataExpiration - DataExpiration for:
> 2010-03-02 365 finished: (1.217 seconds)
> 2010-03-02 01:34:31,431 INFO main DataExpiration - DataExpiration for:
> 2010-03-02 91 finished: (1.148 seconds)
> 2010-03-02 01:34:31,770 INFO main DataExpiration - DataExpiration for:
> 2010-03-02 3650 finished: (2.443 seconds)
> 2010-03-02 01:34:31,954 INFO main DataExpiration - DataExpiration for:
> 2010-03-02 7 finished: (1.796 seconds)
> 2010-03-02 01:34:32,028 INFO main DataExpiration - DataExpiration for:
> 2010-03-02 30 finished: (1.627 seconds)
> 2010-03-02 01:34:32,556 INFO main Aggregator - Longest running query: replace
> into [cluster_system_metrics] (select [avg(system_metrics)] from
> [system_metrics] where timestamp between '[past_15_minutes]' and
> '[past_5_minutes]' group by timestamp); (0.283 seconds)
> 2010-03-02 01:34:32,556 INFO main Aggregator - Total running time: (2.767
> seconds)
> 2010-03-02 01:34:32,556 INFO main Aggregator - Aggregator finished.
>
> in postProcess.log
>
> 2010-03-02 01:42:29,956 INFO main PostProcessorManager - PostProcess Start,
> directory:/chukwa/postProcess/demuxOutputDir_1267494140727
> 2010-03-02 01:42:29,957 INFO main PostProcessorManager -
> org.apache.hadoop.chukwa.dataloader.MetricDataLoaderPool processing:
> /chukwa/postProcess/demuxOutputDir_1267494140727
> 2010-03-02 01:42:29,959 INFO main MetricDataLoaderPool - Processing:
> hdfs://logminerdb:9000/chukwa/postProcess/demuxOutputDir_1267494140727/chukwa/
> SysLog/SysLog_20090302.D.evt
> 2010-03-02 01:42:29,959 INFO pool-1-thread-2 MetricDataLoader - StreamName:
> SysLog_20090302.D.evt
> 2010-03-02 01:42:30,093 INFO pool-1-thread-2 MetricDataLoader - Batch mode
> inserted=0records.
> 2010-03-02 01:42:30,093 INFO pool-1-thread-2 MetricDataLoader - batchMode
> commit done
> 2010-03-02 01:42:30,093 INFO pool-1-thread-2 MetricDataLoader - Saved
> (syslog,chukwa) 0 sec. numOfRecords: 10
> 2010-03-02 01:42:30,093 INFO main PostProcessorManager - loadData Duration:136
> 2010-03-02 01:42:30,093 INFO main ChukwaConfiguration - chukwaConf is
> /opt/chukwainstall/chukwaconf
> 2010-03-02 01:42:30,103 INFO main MoveToRepository - Start MoveToRepository
> main()
> 2010-03-02 01:42:30,105 INFO main MoveToRepository - main procesing Cluster
> (chukwa)
> 2010-03-02 01:42:30,105 INFO main MoveToRepository - processClutserDirectory
> (chukwa,/chukwa/repos//chukwa)
> 2010-03-02 01:42:30,106 INFO main MoveToRepository -
> hdfs://logminerdb:9000/chukwa/postProcess/demuxOutputDir_1267494140727/chukwa/
> SysLog isDir?true
> 2010-03-02 01:42:30,106 INFO main MoveToRepository - dest directory path:
> /chukwa/repos/chukwa/SysLog
> 2010-03-02 01:42:30,106 INFO main MoveToRepository - processClutserDirectory
> processing Datasource: (SysLog)
> 2010-03-02 01:42:30,106 INFO main MoveToRepository -
> processDatasourceDirectory processing RecordFile: (SysLog_20090302.D.evt)
> 2010-03-02 01:42:30,106 INFO main MoveToRepository - fileName:
> SysLog_20090302.D.evt
> 2010-03-02 01:42:30,106 INFO main MoveToRepository - Datasource: SysLog
> 2010-03-02 01:42:30,158 INFO main MoveToRepository - >>>>>>>>>>>> Before
> Renamehdfs://logminerdb:9000/chukwa/postProcess/demuxOutputDir_1267494140727/c
> hukwa/SysLog/SysLog_20090302.D.evt --
> /chukwa/repos/chukwa/SysLog/20090302/SysLog_20090302.77.evt
> 2010-03-02 01:42:30,160 INFO main MoveToRepository - >>>>>>>>>>>> after
> Rename/chukwa/repos/chukwa/SysLog/20090302/SysLog_20090302.77.evt ,
> rename:true
> 2010-03-02 01:42:30,161 INFO main MoveToRepository - Done with
> MoveToRepository main()
> 2010-03-02 01:42:30,161 INFO main PostProcessorManager - movetoMainRepository
> Duration:68
> 2010-03-02 01:42:30,162 INFO main PostProcessorManager - PostProcess Stop,
> directory:/chukwa/postProcess/demuxOutputDir_1267494140727
> 2010-03-02 01:42:30,162 INFO main PostProcessorManager - processDemuxOutput
> Duration:205
>
>