Graphite is an open source system for graphing data. It manages storing data (in a database format called “whisper”), as well as graphing data (with a project called “graphite-web”).

These instructions are fairly platform-independent, and should work on any Unix operating system - the only SmartOS specific bits will be at the end to create SMF services that ensure the services will start at boot.

Graphite Example

The above graph is the temperature of my garage, as monitored by a Raspberry Pi, over a 1 month period.

Overview

There are 3 projects that will be used to setup the graphite server:

  1. carbon - stats receiving daemon (called carbon-cache) that manages storage
  2. whisper - database file format used by carbon to store data
  3. graphite-web - web interface to compose graphite graphs

Prerequisites and Dependencies

To get started, create a user for the graphite services to run as:

groupadd graphite
useradd graphite

And the directory for all of the graphite related files (source code, storage, etc).

mkdir /opt/graphite

Install the required packages

pkgin in py27-{twisted,django,django-tagging,sqlite3,pytz,pyparsing,cairocffi}

Install Graphite

Clone the latest source code

mkdir /opt/graphite/src
cd /opt/graphite/src
git clone git://github.com/graphite-project/carbon.git
git clone git://github.com/graphite-project/whisper.git
git clone git://github.com/graphite-project/graphite-web.git

Install each repository checked out

cd /opt/graphite/src/carbon && python setup.py install
cd /opt/graphite/src/whisper && python setup.py install
cd /opt/graphite/src/graphite-web && python setup.py install

Set proper permissions

chown -R graphite:graphite /opt/graphite/storage

Graphite will only ever write data to the directory above

Setup carbon-cache

Two config files need to be put into place

  • /opt/graphite/conf/storage-schemas.conf - determines how data is stored
[carbon]
pattern = ^carbon\.
retentions = 60:90d

[default]
pattern = .*
retentions = 1m:28d,15m:1y,1h:5y
  • /opt/graphite/conf/carbon.conf - The main carbon config file
[cache]
ENABLE_LOGROTATION = True
USER =
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 500
MAX_CREATES_PER_MINUTE = 50

LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003

ENABLE_UDP_LISTENER = True
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2003

PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004

USE_INSECURE_UNPICKLER = False

CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002

USE_FLOW_CONTROL = True

LOG_UPDATES = False
LOG_CACHE_HITS = False

CACHE_WRITE_STRATEGY = sorted
WHISPER_AUTOFLUSH = False

And finally create the service with the given SMF manifest

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
  <service name='application/carbon-cache' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <dependency name='dep0' grouping='require_all' restart_on='error' type='service'>
      <service_fmri value='svc:/milestone/multi-user:default'/>
    </dependency>
    <exec_method name='start' type='method' exec='/opt/graphite/bin/carbon-cache.py start &amp;' timeout_seconds='10'>
      <method_context working_directory='/opt/graphite'>
        <method_credential user='graphite' group='graphite'/>
        <method_environment>
          <envvar name='PATH' value='/opt/local/sbin:/opt/local/bin:/opt/custom/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'/>
        </method_environment>
      </method_context>
    </exec_method>
    <exec_method name='stop' type='method' exec=':kill' timeout_seconds='30'/>
    <template>
      <common_name>
        <loctext xml:lang='C'>Graphite Carbon Cache</loctext>
      </common_name>
    </template>
  </service>
</service_bundle>

By running

svccfg import carbon-cache.xml

Setup graphite-web

To get started, a couple of commands need to be run to configure django (as user graphite)

su - graphite
export PYTHONPATH=/opt/graphite/webapp
cd /opt/graphite
django-admin.py syncdb --noinput --settings=graphite.settings
django-admin.py collectstatic --noinput --settings=graphite.settings

And a single config file needs to be put in place

  • /opt/graphite/webapp/graphite/local_settings.py - various website settings
SECRET_KEY = 'yolo'
TIME_ZONE = 'America/New_York'
DEBUG = True

And finally create the service with the given SMF manifest

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
  <service name='application/graphite-web' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <dependency name='dep0' grouping='require_all' restart_on='error' type='service'>
      <service_fmri value='svc:/milestone/multi-user:default'/>
    </dependency>
    <exec_method name='start' type='method' exec='/opt/graphite/bin/run-graphite-devel-server.py --port 80 --interface 0.0.0.0 /opt/graphite &amp;' timeout_seconds='10'>
      <method_context working_directory='/opt/graphite'>
        <method_credential user='graphite' group='graphite' privileges='basic,net_privaddr'/>
        <method_environment>
          <envvar name='PATH' value='/opt/local/sbin:/opt/local/bin:/opt/custom/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'/>
        </method_environment>
      </method_context>
    </exec_method>
    <exec_method name='stop' type='method' exec=':kill' timeout_seconds='30'/>
    <template>
      <common_name>
        <loctext xml:lang='C'>Graphite Web Interface</loctext>
      </common_name>
    </template>
  </service>
</service_bundle>

By running

svccfg import graphite-web.xml

Conclusion

Go to http://1.2.3.4/ (substituting your servers IP address) to see the graphite web interface.

Send stats to UDP or TCP 1.2.3.4:2003