Rook-Ceph Cluster Guidance

This document summarizes operational guidance for the Rook-Ceph cluster in this repo. It is based on recent troubleshooting and expansion planning.

Current Topology Assumptions

  • OSDs run on worker nodes only.
  • Failure domain is host (replicas spread across nodes).
  • Replication size is 3.

Nearfull Diagnostics (Read-Only)

Use these commands to identify nearfull OSDs and pools:

ceph osd df
ceph health detail
ceph osd tree
ceph osd dump | egrep 'nearfull|backfillfull|full'

Interpretation tips:

  • nearfull_ratio is the warning threshold.
  • A single nearfull OSD can flag multiple pools as nearfull.
  • With mixed-size OSDs, smaller disks fill first even if total free space exists.

Rebalancing Guidance

If the built-in balancer reports no optimization but one OSD is hotter than others:

  1. Confirm balancer status:
ceph balancer status
  1. If needed, enable balancer (safe, standard):
ceph balancer on
ceph balancer mode upmap
  1. If balancer still shows no plan, use manual reweighting:
ceph osd reweight osd.<id> 0.95

Monitor and step down in small increments (0.02 to 0.05) if necessary:

ceph -s
ceph osd df

Alternative automatic reweight-by-utilization:

ceph osd reweight-by-utilization 110

Rebalancing does not create capacity. If you are nearfull, add capacity or delete data.

Capacity Expansion Strategy

Recommended order (safest to most risky):

  1. Add larger disks to worker nodes, keep old disks temporarily, rebalance, then retire old OSDs.
  2. Replace disks on workers one node at a time (swap-in-place).
  3. Add OSDs to control-plane nodes (not preferred).

Reasons to avoid control-plane OSDs:

  • Control-plane services are latency-sensitive.
  • Lower memory (16 GiB) increases contention risk.
  • Operational complexity and failure impact are higher.

Mixing sizes is supported, but large skews increase nearfull risk on smaller drives.

Mapping OSDs to Nodes/Devices

Use these to identify which OSD is on which node/device:

ceph osd tree
ceph osd find <id>
ceph device ls-by-osd osd.<id>

Swap-In-Place (No Extra Ports)

If you cannot keep the old disk connected while adding a new one, use this workflow. This is riskier because the cluster is degraded during each swap. Do this one node at a time and keep the window short.

  1. Confirm the cluster is stable:
ceph -s
  1. Mark the target OSD out (start data migration while it is still online):
ceph osd out osd.<id>
  1. Wait until recovery/backfill completes:
ceph -s

Look for these signals before proceeding:

  • pgs shows only active+clean (no remapped, recovering, backfilling).
  • objects misplaced is 0.

[!NOTE] If an OSD is still out and you only have 3 OSDs with size=3 replication, recovery cannot finish. In that case, continue the swap and bring the new OSD in so the cluster has a third placement target.

  1. Power down the node, swap the disk, and boot it back up.
sudo systemctl stop k3s
sudo systemctl disable k3s
  1. Clear the disk to prepare it for rook-ceph
# clear disk
export DISK_ID='/dev/disk/by-id/ata-SATA_SSD_...'
sudo rm -rf /dev/ceph-*
sudo rm -rf /dev/mapper/ceph-*
sudo sgdisk --zap-all "${DISK_ID}"
sudo dd if=/dev/zero of="${DISK_ID}" bs=1M count=100 oflag=direct,dsync
sudo blkdiscard "${DISK_ID}"
sudo partprobe "${DISK_ID}"

# start k3s
sudo systemctl enable k3s
sudo systemctl start k3s
  1. Update the Rook-Ceph HelmRelease (and Ansible vars) with the new disk by-id for that node:

kubernetes/apps/rook-ceph/rook-ceph-cluster/app/helmrelease.yaml

  1. Verify the new OSD comes up:
ceph osd tree
ceph osd df
  1. Confirm health returns to OK/WARN only for recovery:
ceph -s

[!CAUTION] Do not remove a second OSD until the first swap is fully recovered! With size=3 and only 3 OSDs, one OSD out reduces redundancy to 2.

  1. Remove old OSD from CRUSH and OSD map.

[!CAUTION] Do not remove the old OSD until the swap is fully recovered!

ceph osd out osd.X
ceph osd purge osd.X --yes-i-really-mean-it

Slow/Stalled recovery

If ceph -s reports PG’s stuck in active+clean+remapped for a prolonged period of time, it may be possible to force the recovery.

# in ceph toolbox
cd /tmp
ceph osd getcrushmap -o crush.map
crushtool -d crush.map -o crush.txt

Edit crush.txt: vi crush.text; set “tunable choose_total_tries 100”

crushtool -c crush.txt -o crush.map2
ceph osd setcrushmap -i crush.map2

Source: kubernetes/apps/rook-ceph/rook-ceph-cluster/README.md