Rook-Ceph

Intro

Quickstart Deployment examples eli5

Rook is an open source cloud-native storage orchestrator, providing the platform, framework, and support for running ceph on kubernetes.

Ceph is a highly scalable distributed storage solution, providing object, block,and file storage.

Requirements

  1. Nodes with amd64/arm64 architecture
  2. Nodes with dedicated local storage
    • Storage can be allocated directly via cluster definition (storage.config.node) or provided via local pvc using openebs-hostpath storageClass or spec.local.path natively in pvc definition
    • If provisioning local disks, the disks must be raw/unformatted

Updating

Rook v1.20 — CSI driver management moved out of Rook. As of v1.20 the operator no longer deploys the Ceph-CSI drivers. The operator chart installs the ceph-csi-operator subchart (controller + CRDs); the actual driver pods are deployed by the separate rook-ceph-csi-drivers HelmRelease (chart ceph-csi-drivers). Per-driver tunables that used to live under the operator chart’s csi: block (tolerations, kubeletDirPath, cephFSKernelMountOptions) and cephClusterSpec.csi.readAffinity now live there instead. Driver names keep the rook-ceph. prefix so existing PVs still resolve.

Reconcile order is enforced via Flux dependsOn: rook-ceph-operator → rook-ceph-csi-drivers → rook-ceph-cluster.

Teardown and Cleanup

Order of operations is critical! See documentation

Run just rook decommission

Troubleshooting

Dashboard not accessible thru ingress

! If this happens, doublecheck the cephClusterSpec.dashboard section of the helm values

To fix manually, run the following commands against the ceph-toolbox pod:

ceph mgr module disable dashboard
ceph config set mgr mgr/dashboard/ssl false
ceph mgr module enable dashboard

Integrate with Prometheus/Alertmanager

run the following commands against the ceph-toolbox pod (replace the placeholder host/namespace with your Prometheus and Alertmanager services)

ceph
dashboard set-alertmanager-api-host 'http://<alertmanager-svc>.<namespace>.svc.cluster.local:9093'
dashboard set-alertmanager-api-ssl-verify False
dashboard set-prometheus-api-host 'http://<prometheus-svc>.<namespace>.svc.cluster.local:9090'
dashboard set-prometheus-api-ssl-verify False

Crash

run the following commands against the ceph-toolbox pod hint just rook toolbox

ceph health detail
# get new crashes
ceph crash ls-new
# get crash info
ceph crash info <crashid>
# archive crash so it doesn't keep triggering warnings or show in 'ls-new'
ceph crash archive-all

docs

View OSD pods

run the following commands against the ceph-toolbox pod

List current OSD pools:

ceph osd tree
ceph osd pool ls

Get current autoscale status (and coincidentally pg_num):

ceph osd pool autoscale-status

List pools

ceph osd lspools

List current placement groups:

ceph pg dump # list
ceph pg stat # status
# Get OSD Pods
# This uses the example/default cluster name "rook"
OSD_PODS=$(kubectl get pods --all-namespaces -l \
  app=rook-ceph-osd,rook_cluster=rook-ceph -o jsonpath='{.items[*].metadata.name}')

# Find node and drive associations from OSD pods
for pod in $(echo ${OSD_PODS})
do
 echo "Pod:  ${pod}"
 echo "Node: $(kubectl -n rook-ceph get pod ${pod} -o jsonpath='{.spec.nodeName}')"
 kubectl -n rook-ceph exec ${pod} -- sh -c '\
  for i in /var/lib/ceph/osd/ceph-*; do
    [ -f ${i}/ready ] || continue
    echo -ne "-$(basename ${i}) "
    echo $(lsblk -n -o NAME,SIZE ${i}/block 2> /dev/null || \
    findmnt -n -v -o SOURCE,SIZE -T ${i}) $(cat ${i}/type)
  done | sort -V
  echo'
done

Too many PGs per OSD

run the following commands against the ceph-toolbox pod

Show current PGs/OSD:

ceph pg dump | awk '
BEGIN { IGNORECASE = 1 }
 /^PG_STAT/ { col=1; while($col!="UP") {col++}; col++ }
 /^[0-9a-f]+\.[0-9a-f]+/ {
   match($0,/^[0-9a-f]+/);
   pool=substr($0, RSTART, RLENGTH);
   poollist[pool]=0;
   up=$col; i=0; RSTART=0; RLENGTH=0; delete osds;
   while (match(up,/[0-9]+/)>0) {
     osds[++i]=substr(up,RSTART,RLENGTH);
     up=substr(up, RSTART+RLENGTH)
   }
   for (i in osds) { array[osds[i],pool]++; osdlist[osds[i]] }
}
END {
  printf("\n");
  printf("pool :\t"); for (i in poollist) printf("%s\t",i); printf("| SUM \n");
  for (i in poollist) printf("--------"); printf("----------------\n");
  for (i in osdlist) {
    printf("osd.%i\t", i); sum=0;
    for (j in poollist) { printf("%i\t", array[i,j]); sum+=array[i,j]; sumpool[j]+=array[i,j] }
    printf("| %i\n",sum)
  }
  for (i in poollist) printf("--------"); printf("----------------\n");
  printf("SUM :\t"); for (i in poollist) printf("%s\t",sumpool[i]); printf("|\n");
}'

The general rules for deciding how many PGs your pool(s) should contain is:

  • Fewer than 5 OSDs set pg_num to 128
  • Between 5 and 10 OSDs set pg_num to 512
  • Between 10 and 50 OSDs set pg_num to 1024
ceph osd pool set <name> pg_num 128

BlueStore slow operations

When you see BLUESTORE_SLOW_OP_ALERT, it is triggered by log_latency_fn messages in OSD logs. A common example is _txc_committed_kv stalls (KV commit latency). A single slow-op within the default 24h lifetime can keep the alert active.

Identify the slow-op line (example from OSD logs):

kubectl -n rook-ceph logs deploy/rook-ceph-osd-0 --since=24h | grep -i 'log_latency.*slow operation'

If the alert is noisy but workloads are healthy, adjust the slow-op threshold in Ceph config. This repo manages it via cephClusterSpec.cephConfig.global in the rook-ceph-cluster HelmRelease. To change live from toolbox instead:

ceph config set global bluestore_slow_ops_warn_threshold 5 # 5 ops
ceph config set global bluestore_slow_ops_warn_lifetime 86400 # in 24h

Diagnose the issue

Check RocksDB and BlueStore metrics for affected OSDs from rook-ceph-tools:

# Check compaction queue and write delays
ceph tell osd.0 perf dump | jq '.rocksdb | {compact_queue_len, compact_running, rocksdb_write_delay_time}'

# Check BlueStore KV latencies
ceph tell osd.0 perf dump | jq '.bluestore | {kv_flush_lat, kv_final_lat, kv_sync_lat}'

Look for:

  • compact_queue_len > 0 (compactions queued up)
  • compact_running > 0 for extended periods
  • High rocksdb_write_delay_time avgtime
  • High kv_flush_lat or kv_sync_lat avgtime (>5ms is concerning)

Force compaction

If RocksDB has a backlog, manually trigger compaction:

ceph tell osd.0 compact
ceph tell osd.1 compact

Watch OSD logs to confirm compaction completes:

kubectl -n rook-ceph logs deploy/rook-ceph-osd-0 --tail=100 | grep -i 'rocksdb\|compact'

The warning should clear automatically once the slow-op counter stops incrementing.

docs - bluestore config docs - perf counters

Ceph Mon Low Space warning

Identify what is taking up all of the space

# check disk space
df -h
# identify large directories
sudo du -hsx /* | sort -rh | head -n 10
# identify what is taking space in directory /var
sudo du -a /var | sort -n -r | head -n 20

Clean up logs

Prefer running the Ansible playbook across all k8s nodes:

ansible-playbook -i ./ansible/inventory/hosts.yaml ./ansible/playbooks/node-cruft-cleanup.yaml --become

Manual cleanup commands:

# if /var/lib/journal is the problem, rotate and vacuum
sudo systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald.service
sudo journalctl --vacuum-size=50M

Clean up unused images

sudo k3s crictl rmi --prune

Restart pods

Restart mon, mds, rgw, and osd pods

Remove orphan rbd images

Use the helper script:

./scripts/rook-rbd-orphan-cleanup.sh

Default behavior is dry-run and does not mutate cluster state.

Just wrapper is available as just rook rbd-orphan-cleanup (see just docs).

Common usage

# dry-run only (default)
just rook rbd-orphan-cleanup

# move safe candidates to RBD trash (recommended first destructive step)
just rook rbd-orphan-cleanup -- --mode trash

# hard delete safe candidates
just rook rbd-orphan-cleanup -- --mode rm

# non-interactive run (for automation)
just rook rbd-orphan-cleanup -- --mode trash --yes

# include non-CSI image names in candidate list
just rook rbd-orphan-cleanup -- --include-non-csi

Show script options:

./scripts/rook-rbd-orphan-cleanup.sh --help

Source: kubernetes/apps/rook-ceph/README.md