README.rst

============
Ceph formula
============

Ceph provides extraordinary data storage scalability. Thousands of client
hosts or KVMs accessing petabytes to exabytes of data. Each one of your
applications can use the object, block or file system interfaces to the same
RADOS cluster simultaneously, which means your Ceph storage system serves as a
flexible foundation for all of your data storage needs.

Use salt-formula-linux for initial disk partitioning.


Sample pillars
==============

Common metadata for all nodes/roles

.. code-block:: yaml

    ceph:
      common:
        version: kraken
        config:
          global:
            param1: value1
            param2: value1
            param3: value1
          pool_section:
            param1: value2
            param2: value2
            param3: value2
        fsid: a619c5fc-c4ed-4f22-9ed2-66cf2feca23d
        members:
        - name: cmn01
          host: 10.0.0.1
        - name: cmn02
          host: 10.0.0.2
        - name: cmn03
          host: 10.0.0.3
        keyring:
          admin:
            key: AQBHPYhZv5mYDBAAvisaSzCTQkC5gywGUp/voA==
            caps:
              mds: "allow *"
              mgr: "allow *"
              mon: "allow *"
              osd: "allow *"

Optional definition for cluster and public networks. Cluster network is used
for replication. Public network for front-end communication.

.. code-block:: yaml

    ceph:
      common:
        version: kraken
        fsid: a619c5fc-c4ed-4f22-9ed2-66cf2feca23d
        ....
        public_network: 10.0.0.0/24, 10.1.0.0/24
        cluster_network: 10.10.0.0/24, 10.11.0.0/24


Ceph mon (control) roles
------------------------

Monitors: A Ceph Monitor maintains maps of the cluster state, including the
monitor map, the OSD map, the Placement Group (PG) map, and the CRUSH map.
Ceph maintains a history (called an “epoch”) of each state change in the Ceph
Monitors, Ceph OSD Daemons, and PGs.

.. code-block:: yaml

    ceph:
      common:
        config:
          mon:
            key: value
      mon:
        enabled: true
        keyring:
          mon:
            key: AQAnQIhZ6in5KxAAdf467upoRMWFcVg5pbh1yg==
            caps:
              mon: "allow *"
          admin:
            key: AQBHPYhZv5mYDBAAvisaSzCTQkC5gywGUp/voA==
            caps:
              mds: "allow *"
              mgr: "allow *"
              mon: "allow *"
              osd: "allow *"


Ceph OSD (storage) roles
------------------------

.. code-block:: yaml

    ceph:
      common:
        config:
          osd:
            key: value
      osd:
        enabled: true
        host_id: 10
        copy_admin_key: true
        journal_type: raw
        dmcrypt: disable
        osd_scenario: raw_journal_devices
        fs_type: xfs
        disk:
          '00':
            rule: hdd
            dev: /dev/vdb2
            journal: /dev/vdb1
            class: besthdd
            weight: 1.5
          '01':
            rule: hdd
            dev: /dev/vdc2
            journal: /dev/vdc1
            class: besthdd
            weight: 1.5
          '02':
            rule: hdd
            dev: /dev/vdd2
            journal: /dev/vdd1
            class: besthdd
            weight: 1.5


Ceph client roles
-----------------

Simple ceph client service

.. code-block:: yaml

    ceph:
      client:
        config:
          global:
            mon initial members: ceph1,ceph2,ceph3
            mon host: 10.103.255.252:6789,10.103.255.253:6789,10.103.255.254:6789
        keyring:
          monitoring:
            key: 00000000000000000000000000000000000000==

At OpenStack control settings are usually located at cinder-volume or glance-
registry services.

.. code-block:: yaml

    ceph:
      client:
        config:
          global:
            fsid: 00000000-0000-0000-0000-000000000000
            mon initial members: ceph1,ceph2,ceph3
            mon host: 10.103.255.252:6789,10.103.255.253:6789,10.103.255.254:6789
            osd_fs_mkfs_arguments_xfs:
            osd_fs_mount_options_xfs: rw,noatime
            network public: 10.0.0.0/24
            network cluster: 10.0.0.0/24
            osd_fs_type: xfs
          osd:
            osd journal size: 7500
            filestore xattr use omap: true
          mon:
            mon debug dump transactions: false
        keyring:
          cinder:
            key: 00000000000000000000000000000000000000==
          glance:
            key: 00000000000000000000000000000000000000==


Ceph gateway
------------

Rados gateway with keystone v2 auth backend

.. code-block:: yaml

    ceph:
      radosgw:
        enabled: true
        hostname: gw.ceph.lab
        bind:
          address: 10.10.10.1
          port: 8080
        identity:
          engine: keystone
          api_version: 2
          host: 10.10.10.100
          port: 5000
          user: admin
          password: password
          tenant: admin

Rados gateway with keystone v3 auth backend

.. code-block:: yaml

    ceph:
      radosgw:
        enabled: true
        hostname: gw.ceph.lab
        bind:
          address: 10.10.10.1
          port: 8080
        identity:
          engine: keystone
          api_version: 3
          host: 10.10.10.100
          port: 5000
          user: admin
          password: password
          project: admin
          domain: default


Ceph setup role
---------------

Replicated ceph storage pool

.. code-block:: yaml

    ceph:
      setup:
        pool:
          replicated_pool:
            pg_num: 256
            pgp_num: 256
            type: replicated
            crush_ruleset_name: 0

Erasure ceph storage pool

.. code-block:: yaml

    ceph:
      setup:
        pool:
          erasure_pool:
            pg_num: 256
            pgp_num: 256
            type: erasure
            crush_ruleset_name: 0

Generate CRUSH map
+++++++++++++++++++

It is required to define the `type` for crush buckets and these types must start with `root` (top) and end with `host`. OSD daemons will be assigned to hosts according to it's hostname. Weight of the buckets will be calculated according to weight of it's childen.

.. code-block:: yaml

  ceph:
    setup:
      crush:
        enabled: True
        tunables:
          choose_total_tries: 50
        type:
          - root
          - region
          - rack
          - host
        root:
          - name: root1
          - name: root2
        region:
          - name: eu-1
            parent: root1
          - name: eu-2
            parent: root1
          - name: us-1
            parent: root2
        rack:
          - name: rack01
            parent: eu-1
          - name: rack02
            parent: eu-2
          - name: rack03
            parent: us-1
        rule:
          sata:
            ruleset: 0
            type: replicated
            min_size: 1
            max_size: 10
            steps:
              - take crushroot.performanceblock.satahss.1
              - choseleaf firstn 0 type failure_domain
              - emit

Ceph monitoring
---------------

Collect general cluster metrics

.. code-block:: yaml

    ceph:
      monitoring:
        cluster_stats:
          enabled: true
          ceph_user: monitoring

Collect metrics from monitor and OSD services

.. code-block:: yaml

    ceph:
      monitoring:
        node_stats:
          enabled: true


More information
================

* https://github.com/cloud-ee/ceph-salt-formula
* http://ceph.com/ceph-storage/
* http://ceph.com/docs/master/start/intro/


Documentation and bugs
======================

To learn how to install and update salt-formulas, consult the documentation
available online at:

    http://salt-formulas.readthedocs.io/

In the unfortunate event that bugs are discovered, they should be reported to
the appropriate issue tracker. Use Github issue tracker for specific salt
formula:

    https://github.com/salt-formulas/salt-formula-ceph/issues

For feature requests, bug reports or blueprints affecting entire ecosystem,
use Launchpad salt-formulas project:

    https://launchpad.net/salt-formulas

You can also join salt-formulas-users team and subscribe to mailing list:

    https://launchpad.net/~salt-formulas-users

Developers wishing to work on the salt-formulas projects should always base
their work on master branch and submit pull request against specific formula.

    https://github.com/salt-formulas/salt-formula-ceph

Any questions or feedback is always welcome so feel free to join our IRC
channel:

    #salt-formulas @ irc.freenode.net