Deploying Ceph Cluster

HDD-based Ceph Cluster with Open Cache Acceleration Software (Open CAS)

ID	Date	Version	Classification
778514	05/10/2023	1.0	Public

A Kubernetes cluster (v1.19+), with one control-plane node and two worker nodes, should have been set up.

Deploy the Rook operator first:

Details in Rook Ceph Quickstart.

git clone --single-branch --branch v1.10.13 https://github.com/rook/rook.git
cd rook/deploy/examples

kubectl create -f crds.yaml -f common.yaml -f operator.yaml

# wait until ready
kubectl get po -n rook-ceph -w

Then configure the cluster.yaml, here is an example:

Use another SSD on each machine as the metadataDevice.
The node name should match its kubernetes.io/hostname label.
- You can check it with command: kubectl get node -o wide

Details in CephCluster CRD.

storage:
  useAllNodes: false
  useAllDevices: false
  config:
    osdsPerDevice: "1"
  nodes:
  - name: "ubuntu-2f923ef220"
    devices:
    - name: "cas1-1"
    - name: "cas1-2"
    - name: "cas1-3"
    config:
      metadataDevice: "/dev/disk/by-id/wwn-0x55cd2e415379ee47"
  - name: "ubuntu-1e5e82320b"
    devices:
    - name: "cas1-1"
    - name: "cas1-2"
    - name: "cas1-3"
    config:
      metadataDevice: "/dev/disk/by-id/wwn-0x55cd2e4153791b74"
  - name: "ubuntu-ae1fc3e283"
    devices:
    - name: "cas1-1"
    - name: "cas1-2"
    - name: "cas1-3"
    config:
      metadataDevice: "/dev/disk/by-id/wwn-0x55cd2e4153792e14"

Now we can create a Ceph cluster:

# double check before proceeding
git diff
# let's go
kubectl create -f cluster.yaml

# wait until ready
kubectl -n rook-ceph get pod -w

Deploy Rook toolbox and connect to it:

kubectl create -f toolbox.yaml
# wait until ready
kubectl -n rook-ceph rollout status deploy/rook-ceph-tools
# connect to it
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

Run ceph status command to check the status:

The health should be HEALTH_OK.
At least one OSD should be active.

  cluster:
    id:     e403f9a2-dae8-4040-89d2-94901daa91e9
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 10h)
    mgr: a(active, since 10h), standbys: b
    osd: 9 osds: 9 up (since 2m), 9 in (since 6m)

  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 769 KiB
    usage:   5.2 TiB used, 10 TiB / 15 TiB avail
    pgs:     1 active+clean
...