Hadoop ganglia metrics

[skr]

Hi.

Could some kind soul write a short howto on the hadoop wiki on using the
Hadoop -> Ganglia metrics component?

I've tried to set it up as the javadoc suggests but no luck. I'm no
Ganglia expert though, so perhaps I need to change something on that end?

/Johan

[Juan Manuel Caicedo-2]

Hi

I was thinking in something related last week. I wanted to include
some statistics about DFS, like free space, or the cluster itself
(running jobs). Although the monitoring applications provide that
information, it would be interesting to have it in Ganglia too.

I haven't try it yet, but you could use embeddedgmetric
(http://code.google.com/p/embeddedgmetric/) for injecting metrics to
Ganglia.

Regards,

Juan Manuel Caicedo

On Dec 10, 2007 1:39 PM, Johan Oskarsson <johan@...> wrote:

> Hi.
>
> Could some kind soul write a short howto on the hadoop wiki on using the
> Hadoop -> Ganglia metrics component?
>
> I've tried to set it up as the javadoc suggests but no luck. I'm no
> Ganglia expert though, so perhaps I need to change something on that end?
>
> /Johan
>

[Aroop Maliakkal]

Hi Johan,

You need to use package org.apache.hadoop.metrics.ganglia inorder to
measure the hadoop metrics using Ganglia. Since it is built-in with
hadoop code itself, you don;t need any extra jar for ganglia
implementation.

First you need to have a ganglia setup running on your cluster in which
gmond will be collecting all your metrics . The configuration file for
gmond is /etc/gmond.conf . For integrating Ganglia into Hadoop, you need
to make modification to hadoop-metrics.properties file.

By default, hadoop-metrics.properties will be using NullContext . You
need to modify it to use GangliaContext as given below.

# Configuration of the "dfs" context for ganglia
  dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
  dfs.period=10
  dfs.servers=localhost:8649

# Configuration of the "mapred" context for ganglia
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=localhost:8649

Once you started running your hadoop job, hadoop.metrics.ganglia package
will start sending your hadop metrics  (  DFS metrics to
dfs.servers=localhost:8649 && MAPRED metrics to
mapred.servers=localhost:8649 ). If you want to send this metrics to
some other gmond, just change it to dfs.servers=remote_server1.com:8649
& mapred.servers=remote_server2.com:8649 . Your entire hadoop metrics
will be collected on $remote_server1.com and $remote_server2.com and you
can use gmetad to pull this information from this server.


Hope I answered your question. If anything is not clear, let us know .

Thanks,
/Aroop
Johan Oskarsson wrote:

> Hi.
>
> Could some kind soul write a short howto on the hadoop wiki on using the
> Hadoop -> Ganglia metrics component?
>
> I've tried to set it up as the javadoc suggests but no luck. I'm no
> Ganglia expert though, so perhaps I need to change something on that end?
>
> /Johan
>