You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by Ariel Rabkin <as...@gmail.com> on 2009/05/22 07:06:29 UTC
units in MDL and HICC
Howdy all.
So I've noticed something. The default mdl.xml has entries for
memused, and kbkached.
My version of sar outputs kbcached, and *kb*memused. So memused
doesn't display right.
In general though, I've gotten worried about units.
if I stick 1000 * kbmemused in mdl.xml, will that get pasted into a
SQL command and will the right thing happen?
Is there a better way to do unit conversion, other than hacking the Java?
Is there any way to know what the right units are, actually?
--Ari
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
Re: units in MDL and HICC
Posted by Jiaqi Tan <ta...@gmail.com>.
I think it's fine to have the semantics remain in the Demux, in that
case perhaps the Demux processors can look at the sar-generated column
labels to determine what are the units, and to standardize the units
of the output?
Jiaqi
On Fri, May 22, 2009 at 10:56 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
> Many solutions have been suggested in the past year, but there isn't one
> fits all. Most of the promising library is in the GPL camp. Unfortunately,
> we can't use those. The closest thing in the apache camp is the ganglia
> metrics library. There are 2 bugs that they need to fix in the metrics
> library. First, it uses float to store all values, hence the accuracy
> becomes somewhat questionable for large values. Second, one of the metrics
> only include value from first device. I forget it's either network device
> or disk. I dropped integration of ganglia metrics library after discovering
> those bugs. However, we might want to revisit this, if it has been
> improved. For the windows camp, we may need a completely different solution
> for measuring system metrics.
>
> I believe all parsing logic and data schematics should happen in demux
> parser rather than MDL. Personally, I believe MDL should have zero
> configuration. MDL's purpose is to load data into database by knowing the
> RecordType=Table, Key=Column, Value=Value. This will definitely reduce the
> places that we maintain data transformation. The data schematics should
> happen in demux parser, and database_create_table.sql only. What do you
> guys think?
>
> Regards,
> Eric
>
> On 5/21/09 11:01 PM, "Ariel Rabkin" <as...@gmail.com> wrote:
>
>> Howdy.
>>
>> I agree with your diagnosis -- this is the peril of external
>> dependencies. There was discussion, back in the day, about doing
>> something better. Poking at /proc is certainly one option. Another
>> would be finding some apache-licensed library that does this. Sigar
>> would fit the bill, but it's GPLed and so we can't link against it.
>> Though there was discussion under HADOOP-4959 about a license
>> exemption. That might solve our problem.
>>
>> There's a Java standard approach that does some subset of what we want
>> --
>> http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/managem
>> ent/UnixOperatingSystemMXBean.html
>>
>> What's peculiar about this issue is that right now, the actual Demux
>> processors are largely independent of the versions -- those processors
>> make assumptions about the syntax of the input, but almost none about
>> the semantics. If the data comes in columns with headers, they do
>> basically the right thing. However, when it comes time to do the
>> database insert, the column names don't match the ones in mdl.xml, and
>> so things start to fail.
>>
>> It seems a pity to dirty up the currently clean Java code with lots of
>> special cases for canonical-izing data formats. I'm okay doing some
>> sort of parameterization, but I think in a lot of cases we can do
>> something very simpleminded and still be okay. Perhaps as simple as
>> "if you see field x in a SystemMetrics record, output field y as
>> follows."
>>
>> On Thu, May 21, 2009 at 10:27 PM, Jiaqi Tan <ta...@gmail.com> wrote:
>>> Hi Ari,
>>>
>>> I think the real problem here is that sar metrics are being picked up
>>> by an Exec adaptor which calls sar and there's no control over which
>>> sar gets called (or at least not right now), and sar is ultimately an
>>> external dependency which currently is just assumed to be sitting
>>> there.
>>>
>>> Also, sar just directly emits unstructured plain text, so there's no
>>> self-describing data format a la some XML which says what the units
>>> are, so if sar is changing output units and stuff, then the parser in
>>> the Demux needs to take care of that too. Even more generally, even
>>> any change at all to sar's output would require an update of the
>>> Demux.
>>>
>>> I think the fundamental problem is that having an Exec adaptor which
>>> pulls the unstructured output of an external program and having a
>>> Demux processor that makes assumptions about what that output looks
>>> like and what it means, makes the whole workflow dependent on
>>> something not under the control of Chukwa.
>>>
>>> I can imagine one way of working around that would be to not use sar
>>> and write custom parsers for /proc so that Chukwa is itself aware of
>>> what the proc data actually means without having to make assumptions
>>> about the output of an external parser; it's reinventing the wheel
>>> somewhat but it gives an end-to-end cleaner solution.
>>>
>>> The other answer would perhaps be the "web services" answer of having
>>> a whole standardized way of passing data around in a structured way
>>> but then that starts to look like a generalized pub/sub system.
>>>
>>> But in the meantime maybe the sar version on the system being
>>> monitored could be picked up in some way (metadata in the Chunk?) and
>>> the various Demux processors dependent on such external programs e.g.
>>> IoStat, Df, etc. could be parameterized to handle output from
>>> different versions/variants of the source program. Or to be even more
>>> general, the Exec adaptor could send along an MD5 hash of the program
>>> it's calling, and then you'd have a whole bunch of processors for
>>> every possible variant of the program you want to support; that sounds
>>> terribly hackish to me but I think that way at least the identity of
>>> the external dependency can be identified.
>>>
>
>
Re: units in MDL and HICC
Posted by Jiaqi Tan <ta...@gmail.com>.
Hi,
In my mind, part of the peculiarity is also that Java is supposed to
be platform-neutral, but proc is clearly not a platform neutral thing
(Windows?) and even different versions of Linux can implement
different versions of procfs. In terms of licensing, sar is also
GPL'ed so I guess we can't just bundle some particular version of sar
with Chukwa and force everyone to use that. Even then, that wouldn't
work with non-Unix systems. As a sidenote, I think the Demux
processors are "independent" of semantics only because nobody (as in,
it's not meant for human presentation/consumption until past the MDL)
looks at the output from Demux.
In fact, I did have another thought--it seems the Exec adaptor is
currently also picking up column names from sar, and the processor
does see them, maybe we could stick the column names in as some form
of metadata and have the MDL consider the column names to determine if
any unit conversions are necessary?
Jiaqi
On Thu, May 21, 2009 at 11:01 PM, Ariel Rabkin <as...@gmail.com> wrote:
> Howdy.
>
> I agree with your diagnosis -- this is the peril of external
> dependencies. There was discussion, back in the day, about doing
> something better. Poking at /proc is certainly one option. Another
> would be finding some apache-licensed library that does this. Sigar
> would fit the bill, but it's GPLed and so we can't link against it.
> Though there was discussion under HADOOP-4959 about a license
> exemption. That might solve our problem.
>
> There's a Java standard approach that does some subset of what we want
> -- http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html
>
> What's peculiar about this issue is that right now, the actual Demux
> processors are largely independent of the versions -- those processors
> make assumptions about the syntax of the input, but almost none about
> the semantics. If the data comes in columns with headers, they do
> basically the right thing. However, when it comes time to do the
> database insert, the column names don't match the ones in mdl.xml, and
> so things start to fail.
>
> It seems a pity to dirty up the currently clean Java code with lots of
> special cases for canonical-izing data formats. I'm okay doing some
> sort of parameterization, but I think in a lot of cases we can do
> something very simpleminded and still be okay. Perhaps as simple as
> "if you see field x in a SystemMetrics record, output field y as
> follows."
>
> On Thu, May 21, 2009 at 10:27 PM, Jiaqi Tan <ta...@gmail.com> wrote:
>> Hi Ari,
>>
>> I think the real problem here is that sar metrics are being picked up
>> by an Exec adaptor which calls sar and there's no control over which
>> sar gets called (or at least not right now), and sar is ultimately an
>> external dependency which currently is just assumed to be sitting
>> there.
>>
>> Also, sar just directly emits unstructured plain text, so there's no
>> self-describing data format a la some XML which says what the units
>> are, so if sar is changing output units and stuff, then the parser in
>> the Demux needs to take care of that too. Even more generally, even
>> any change at all to sar's output would require an update of the
>> Demux.
>>
>> I think the fundamental problem is that having an Exec adaptor which
>> pulls the unstructured output of an external program and having a
>> Demux processor that makes assumptions about what that output looks
>> like and what it means, makes the whole workflow dependent on
>> something not under the control of Chukwa.
>>
>> I can imagine one way of working around that would be to not use sar
>> and write custom parsers for /proc so that Chukwa is itself aware of
>> what the proc data actually means without having to make assumptions
>> about the output of an external parser; it's reinventing the wheel
>> somewhat but it gives an end-to-end cleaner solution.
>>
>> The other answer would perhaps be the "web services" answer of having
>> a whole standardized way of passing data around in a structured way
>> but then that starts to look like a generalized pub/sub system.
>>
>> But in the meantime maybe the sar version on the system being
>> monitored could be picked up in some way (metadata in the Chunk?) and
>> the various Demux processors dependent on such external programs e.g.
>> IoStat, Df, etc. could be parameterized to handle output from
>> different versions/variants of the source program. Or to be even more
>> general, the Exec adaptor could send along an MD5 hash of the program
>> it's calling, and then you'd have a whole bunch of processors for
>> every possible variant of the program you want to support; that sounds
>> terribly hackish to me but I think that way at least the identity of
>> the external dependency can be identified.
>>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>
Re: units in MDL and HICC
Posted by Eric Yang <ey...@yahoo-inc.com>.
Many solutions have been suggested in the past year, but there isn't one
fits all. Most of the promising library is in the GPL camp. Unfortunately,
we can't use those. The closest thing in the apache camp is the ganglia
metrics library. There are 2 bugs that they need to fix in the metrics
library. First, it uses float to store all values, hence the accuracy
becomes somewhat questionable for large values. Second, one of the metrics
only include value from first device. I forget it's either network device
or disk. I dropped integration of ganglia metrics library after discovering
those bugs. However, we might want to revisit this, if it has been
improved. For the windows camp, we may need a completely different solution
for measuring system metrics.
I believe all parsing logic and data schematics should happen in demux
parser rather than MDL. Personally, I believe MDL should have zero
configuration. MDL's purpose is to load data into database by knowing the
RecordType=Table, Key=Column, Value=Value. This will definitely reduce the
places that we maintain data transformation. The data schematics should
happen in demux parser, and database_create_table.sql only. What do you
guys think?
Regards,
Eric
On 5/21/09 11:01 PM, "Ariel Rabkin" <as...@gmail.com> wrote:
> Howdy.
>
> I agree with your diagnosis -- this is the peril of external
> dependencies. There was discussion, back in the day, about doing
> something better. Poking at /proc is certainly one option. Another
> would be finding some apache-licensed library that does this. Sigar
> would fit the bill, but it's GPLed and so we can't link against it.
> Though there was discussion under HADOOP-4959 about a license
> exemption. That might solve our problem.
>
> There's a Java standard approach that does some subset of what we want
> --
> http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/managem
> ent/UnixOperatingSystemMXBean.html
>
> What's peculiar about this issue is that right now, the actual Demux
> processors are largely independent of the versions -- those processors
> make assumptions about the syntax of the input, but almost none about
> the semantics. If the data comes in columns with headers, they do
> basically the right thing. However, when it comes time to do the
> database insert, the column names don't match the ones in mdl.xml, and
> so things start to fail.
>
> It seems a pity to dirty up the currently clean Java code with lots of
> special cases for canonical-izing data formats. I'm okay doing some
> sort of parameterization, but I think in a lot of cases we can do
> something very simpleminded and still be okay. Perhaps as simple as
> "if you see field x in a SystemMetrics record, output field y as
> follows."
>
> On Thu, May 21, 2009 at 10:27 PM, Jiaqi Tan <ta...@gmail.com> wrote:
>> Hi Ari,
>>
>> I think the real problem here is that sar metrics are being picked up
>> by an Exec adaptor which calls sar and there's no control over which
>> sar gets called (or at least not right now), and sar is ultimately an
>> external dependency which currently is just assumed to be sitting
>> there.
>>
>> Also, sar just directly emits unstructured plain text, so there's no
>> self-describing data format a la some XML which says what the units
>> are, so if sar is changing output units and stuff, then the parser in
>> the Demux needs to take care of that too. Even more generally, even
>> any change at all to sar's output would require an update of the
>> Demux.
>>
>> I think the fundamental problem is that having an Exec adaptor which
>> pulls the unstructured output of an external program and having a
>> Demux processor that makes assumptions about what that output looks
>> like and what it means, makes the whole workflow dependent on
>> something not under the control of Chukwa.
>>
>> I can imagine one way of working around that would be to not use sar
>> and write custom parsers for /proc so that Chukwa is itself aware of
>> what the proc data actually means without having to make assumptions
>> about the output of an external parser; it's reinventing the wheel
>> somewhat but it gives an end-to-end cleaner solution.
>>
>> The other answer would perhaps be the "web services" answer of having
>> a whole standardized way of passing data around in a structured way
>> but then that starts to look like a generalized pub/sub system.
>>
>> But in the meantime maybe the sar version on the system being
>> monitored could be picked up in some way (metadata in the Chunk?) and
>> the various Demux processors dependent on such external programs e.g.
>> IoStat, Df, etc. could be parameterized to handle output from
>> different versions/variants of the source program. Or to be even more
>> general, the Exec adaptor could send along an MD5 hash of the program
>> it's calling, and then you'd have a whole bunch of processors for
>> every possible variant of the program you want to support; that sounds
>> terribly hackish to me but I think that way at least the identity of
>> the external dependency can be identified.
>>
Re: units in MDL and HICC
Posted by Ariel Rabkin <as...@gmail.com>.
Howdy.
I agree with your diagnosis -- this is the peril of external
dependencies. There was discussion, back in the day, about doing
something better. Poking at /proc is certainly one option. Another
would be finding some apache-licensed library that does this. Sigar
would fit the bill, but it's GPLed and so we can't link against it.
Though there was discussion under HADOOP-4959 about a license
exemption. That might solve our problem.
There's a Java standard approach that does some subset of what we want
-- http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html
What's peculiar about this issue is that right now, the actual Demux
processors are largely independent of the versions -- those processors
make assumptions about the syntax of the input, but almost none about
the semantics. If the data comes in columns with headers, they do
basically the right thing. However, when it comes time to do the
database insert, the column names don't match the ones in mdl.xml, and
so things start to fail.
It seems a pity to dirty up the currently clean Java code with lots of
special cases for canonical-izing data formats. I'm okay doing some
sort of parameterization, but I think in a lot of cases we can do
something very simpleminded and still be okay. Perhaps as simple as
"if you see field x in a SystemMetrics record, output field y as
follows."
On Thu, May 21, 2009 at 10:27 PM, Jiaqi Tan <ta...@gmail.com> wrote:
> Hi Ari,
>
> I think the real problem here is that sar metrics are being picked up
> by an Exec adaptor which calls sar and there's no control over which
> sar gets called (or at least not right now), and sar is ultimately an
> external dependency which currently is just assumed to be sitting
> there.
>
> Also, sar just directly emits unstructured plain text, so there's no
> self-describing data format a la some XML which says what the units
> are, so if sar is changing output units and stuff, then the parser in
> the Demux needs to take care of that too. Even more generally, even
> any change at all to sar's output would require an update of the
> Demux.
>
> I think the fundamental problem is that having an Exec adaptor which
> pulls the unstructured output of an external program and having a
> Demux processor that makes assumptions about what that output looks
> like and what it means, makes the whole workflow dependent on
> something not under the control of Chukwa.
>
> I can imagine one way of working around that would be to not use sar
> and write custom parsers for /proc so that Chukwa is itself aware of
> what the proc data actually means without having to make assumptions
> about the output of an external parser; it's reinventing the wheel
> somewhat but it gives an end-to-end cleaner solution.
>
> The other answer would perhaps be the "web services" answer of having
> a whole standardized way of passing data around in a structured way
> but then that starts to look like a generalized pub/sub system.
>
> But in the meantime maybe the sar version on the system being
> monitored could be picked up in some way (metadata in the Chunk?) and
> the various Demux processors dependent on such external programs e.g.
> IoStat, Df, etc. could be parameterized to handle output from
> different versions/variants of the source program. Or to be even more
> general, the Exec adaptor could send along an MD5 hash of the program
> it's calling, and then you'd have a whole bunch of processors for
> every possible variant of the program you want to support; that sounds
> terribly hackish to me but I think that way at least the identity of
> the external dependency can be identified.
>
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
Re: units in MDL and HICC
Posted by Jiaqi Tan <ta...@gmail.com>.
Hi Ari,
I think the real problem here is that sar metrics are being picked up
by an Exec adaptor which calls sar and there's no control over which
sar gets called (or at least not right now), and sar is ultimately an
external dependency which currently is just assumed to be sitting
there.
Also, sar just directly emits unstructured plain text, so there's no
self-describing data format a la some XML which says what the units
are, so if sar is changing output units and stuff, then the parser in
the Demux needs to take care of that too. Even more generally, even
any change at all to sar's output would require an update of the
Demux.
I think the fundamental problem is that having an Exec adaptor which
pulls the unstructured output of an external program and having a
Demux processor that makes assumptions about what that output looks
like and what it means, makes the whole workflow dependent on
something not under the control of Chukwa.
I can imagine one way of working around that would be to not use sar
and write custom parsers for /proc so that Chukwa is itself aware of
what the proc data actually means without having to make assumptions
about the output of an external parser; it's reinventing the wheel
somewhat but it gives an end-to-end cleaner solution.
The other answer would perhaps be the "web services" answer of having
a whole standardized way of passing data around in a structured way
but then that starts to look like a generalized pub/sub system.
But in the meantime maybe the sar version on the system being
monitored could be picked up in some way (metadata in the Chunk?) and
the various Demux processors dependent on such external programs e.g.
IoStat, Df, etc. could be parameterized to handle output from
different versions/variants of the source program. Or to be even more
general, the Exec adaptor could send along an MD5 hash of the program
it's calling, and then you'd have a whole bunch of processors for
every possible variant of the program you want to support; that sounds
terribly hackish to me but I think that way at least the identity of
the external dependency can be identified.
Jiaqi
On Thu, May 21, 2009 at 10:06 PM, Ariel Rabkin <as...@gmail.com> wrote:
> Howdy all.
>
> So I've noticed something. The default mdl.xml has entries for
> memused, and kbkached.
> My version of sar outputs kbcached, and *kb*memused. So memused
> doesn't display right.
>
> In general though, I've gotten worried about units.
> if I stick 1000 * kbmemused in mdl.xml, will that get pasted into a
> SQL command and will the right thing happen?
> Is there a better way to do unit conversion, other than hacking the Java?
>
> Is there any way to know what the right units are, actually?
>
> --Ari
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>
Re: units in MDL and HICC
Posted by Ariel Rabkin <as...@gmail.com>.
On Fri, May 22, 2009 at 11:10 AM, Jiaqi Tan <ta...@gmail.com> wrote:
> On Fri, May 22, 2009 at 11:06 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
>> On 5/22/09 10:35 AM, "Ariel Rabkin" <as...@gmail.com> wrote:
>>
>>> I'm okay with doing this in the parser. Particularly if it's
>>> declaratively driven with something like that conversion syntax,
>>> embedded in demux-conf. But we should probably be clear to ourselves
>>> what constitutes "truth" when it comes to units and such.
>>
>> I like to use base unit of bytes and raw time millis in HDFS, if possible.
>> The current base units are bytes, and per minute. All data are in bytes at
>> yahoo except HDFS capacity in GB from hadoop metrics.
>
> Sure, but I think what Ari was referring to was when the sar-generated
> numbers are not in the base units? Then in that case I think the Demux
> processor should look at the column labels to determine what's the
> source unit and convert that to the base unit?
>
Yes, exactly. I think we're all in agreement about that now, so I've
opened CHUKWA-260 for it.
--Ari
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
Re: units in MDL and HICC
Posted by Jiaqi Tan <ta...@gmail.com>.
On Fri, May 22, 2009 at 11:06 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
> On 5/22/09 10:35 AM, "Ariel Rabkin" <as...@gmail.com> wrote:
>
>> I'm okay with doing this in the parser. Particularly if it's
>> declaratively driven with something like that conversion syntax,
>> embedded in demux-conf. But we should probably be clear to ourselves
>> what constitutes "truth" when it comes to units and such.
>
> I like to use base unit of bytes and raw time millis in HDFS, if possible.
> The current base units are bytes, and per minute. All data are in bytes at
> yahoo except HDFS capacity in GB from hadoop metrics.
Sure, but I think what Ari was referring to was when the sar-generated
numbers are not in the base units? Then in that case I think the Demux
processor should look at the column labels to determine what's the
source unit and convert that to the base unit?
- Jiaqi
>>
>> I found the conversion code you mention. Right now, it lives in MDL.
>> Eric, are you saying that it should move to demux, so that the HDFS
>> copy of data is accurate? If so, I think I agree.
>
> Yes, we have an agreement.
>
>>
>> On Fri, May 22, 2009 at 10:28 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
>>> There is a special keyword called conversion.<record_type>.<key>,
>>> By putting 1000, it will apply column_value*1000 before inserting into
>>> database. I did not test much of this pre chukwa days. I am not sure if it
>>> stil works.
>>>
>>> The right thing to do is to improve the parser to do the right thing hence,
>>> cleansed and unified truth is stored in hdfs.
>>>
>>> Rgards,
>>> Eric
>>>
>>> On 5/21/09 10:06 PM, "Ariel Rabkin" <as...@gmail.com> wrote:
>>>
>>>> Howdy all.
>>>>
>>>> So I've noticed something. The default mdl.xml has entries for
>>>> memused, and kbkached.
>>>> My version of sar outputs kbcached, and *kb*memused. So memused
>>>> doesn't display right.
>>>>
>>>> In general though, I've gotten worried about units.
>>>> if I stick 1000 * kbmemused in mdl.xml, will that get pasted into a
>>>> SQL command and will the right thing happen?
>>>> Is there a better way to do unit conversion, other than hacking the Java?
>>>>
>>>> Is there any way to know what the right units are, actually?
>>>>
>>>> --Ari
>>>
>>>
>>
>>
>
>
Re: units in MDL and HICC
Posted by Eric Yang <ey...@yahoo-inc.com>.
On 5/22/09 10:35 AM, "Ariel Rabkin" <as...@gmail.com> wrote:
> I'm okay with doing this in the parser. Particularly if it's
> declaratively driven with something like that conversion syntax,
> embedded in demux-conf. But we should probably be clear to ourselves
> what constitutes "truth" when it comes to units and such.
I like to use base unit of bytes and raw time millis in HDFS, if possible.
The current base units are bytes, and per minute. All data are in bytes at
yahoo except HDFS capacity in GB from hadoop metrics.
>
> I found the conversion code you mention. Right now, it lives in MDL.
> Eric, are you saying that it should move to demux, so that the HDFS
> copy of data is accurate? If so, I think I agree.
Yes, we have an agreement.
>
> On Fri, May 22, 2009 at 10:28 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
>> There is a special keyword called conversion.<record_type>.<key>,
>> By putting 1000, it will apply column_value*1000 before inserting into
>> database. I did not test much of this pre chukwa days. I am not sure if it
>> stil works.
>>
>> The right thing to do is to improve the parser to do the right thing hence,
>> cleansed and unified truth is stored in hdfs.
>>
>> Rgards,
>> Eric
>>
>> On 5/21/09 10:06 PM, "Ariel Rabkin" <as...@gmail.com> wrote:
>>
>>> Howdy all.
>>>
>>> So I've noticed something. The default mdl.xml has entries for
>>> memused, and kbkached.
>>> My version of sar outputs kbcached, and *kb*memused. So memused
>>> doesn't display right.
>>>
>>> In general though, I've gotten worried about units.
>>> if I stick 1000 * kbmemused in mdl.xml, will that get pasted into a
>>> SQL command and will the right thing happen?
>>> Is there a better way to do unit conversion, other than hacking the Java?
>>>
>>> Is there any way to know what the right units are, actually?
>>>
>>> --Ari
>>
>>
>
>
Re: units in MDL and HICC
Posted by Ariel Rabkin <as...@gmail.com>.
I'm okay with doing this in the parser. Particularly if it's
declaratively driven with something like that conversion syntax,
embedded in demux-conf. But we should probably be clear to ourselves
what constitutes "truth" when it comes to units and such.
I found the conversion code you mention. Right now, it lives in MDL.
Eric, are you saying that it should move to demux, so that the HDFS
copy of data is accurate? If so, I think I agree.
On Fri, May 22, 2009 at 10:28 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
> There is a special keyword called conversion.<record_type>.<key>,
> By putting 1000, it will apply column_value*1000 before inserting into
> database. I did not test much of this pre chukwa days. I am not sure if it
> stil works.
>
> The right thing to do is to improve the parser to do the right thing hence,
> cleansed and unified truth is stored in hdfs.
>
> Rgards,
> Eric
>
> On 5/21/09 10:06 PM, "Ariel Rabkin" <as...@gmail.com> wrote:
>
>> Howdy all.
>>
>> So I've noticed something. The default mdl.xml has entries for
>> memused, and kbkached.
>> My version of sar outputs kbcached, and *kb*memused. So memused
>> doesn't display right.
>>
>> In general though, I've gotten worried about units.
>> if I stick 1000 * kbmemused in mdl.xml, will that get pasted into a
>> SQL command and will the right thing happen?
>> Is there a better way to do unit conversion, other than hacking the Java?
>>
>> Is there any way to know what the right units are, actually?
>>
>> --Ari
>
>
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
Re: units in MDL and HICC
Posted by Eric Yang <ey...@yahoo-inc.com>.
There is a special keyword called conversion.<record_type>.<key>,
By putting 1000, it will apply column_value*1000 before inserting into
database. I did not test much of this pre chukwa days. I am not sure if it
stil works.
The right thing to do is to improve the parser to do the right thing hence,
cleansed and unified truth is stored in hdfs.
Rgards,
Eric
On 5/21/09 10:06 PM, "Ariel Rabkin" <as...@gmail.com> wrote:
> Howdy all.
>
> So I've noticed something. The default mdl.xml has entries for
> memused, and kbkached.
> My version of sar outputs kbcached, and *kb*memused. So memused
> doesn't display right.
>
> In general though, I've gotten worried about units.
> if I stick 1000 * kbmemused in mdl.xml, will that get pasted into a
> SQL command and will the right thing happen?
> Is there a better way to do unit conversion, other than hacking the Java?
>
> Is there any way to know what the right units are, actually?
>
> --Ari