Ceph

Resize ceph rdb images

Ceph volumes  are thin provisioned, so they don’t actually use any physical storage until it begin saving data to them. A maximum capacity of the volume set with the –size option. If we want to increase (or decrease) the maximum size of a Ceph volume, can be done using the ‘rbd –resize –image <vol_name> –size <new_size>.

Here are a few examples:

Step#1 – Created an image

~/swami# rbd import /root/swami/ubuntu_14.04.qcow2 -p test

Importing image: 100% complete…done.

Step#2: List the images

~/swami# rbd -p test ls

swami_test_img

ubuntu_14.04.qcow2

Step#3: Show image info:

~/swami# rbd -p test info ubuntu_14.04.qcow2

rbd image ‘ubuntu_14.04.qcow2’:

        size 245 MB in 62 objects

        order 22 (4096 kB objects)

        block_name_prefix: rb.0.2bae54.238e1f29

        format: 1

Step#4: Resize to 1000MB (from 242 MB)

~/swami# rbd -p test resize ubuntu_14.04.qcow2 –size 1000

Resizing image: 100% complete…done.

~/swami# rbd -p test info ubuntu_14.04.qcow2

rbd image ‘ubuntu_14.04.qcow2’:

        size 1000 MB in 250 objects

        order 22 (4096 kB objects)

        block_name_prefix: rb.0.2bae54.238e1f29

        format: 1

Step#5: Resize to 250 (from 1000) MB

~/swami# rbd -p test resize ubuntu_14.04.qcow2 –size 250 –allow-shrink

Resizing image: 100% complete…done.

~/swami# rbd -p test info ubuntu_14.04.qcow2

rbd image ‘ubuntu_14.04.qcow2’:

        size 250 MB in 63 objects

        order 22 (4096 kB objects)

        block_name_prefix: rb.0.2bae54.238e1f29

        format: 1

Step#6: Resize to 240 (from 250) MB  (NOTE: resize to shrink allowed with data loss, so resize to shrink should be very careful action needed)

~/swami# rbd -p test resize ubuntu_14.04.qcow2 –size 240 –allow-shrink

Resizing image: 100% complete…done.

~/swami# rbd -p test info ubuntu_14.04.qcow2      

            rbd image ‘ubuntu_14.04.qcow2’:

        size 240 MB in 60 objects

        order 22 (4096 kB objects)

        block_name_prefix: rb.0.2bae54.238e1f29

        format: 1

 

Advertisements
Ceph

RBD images real/actual size

There is no direct way to get the real or actual usage or data in Ceph rbd image. But we can get the actual size of an image using the below command:

rbd diff <pool_name>/image_name  | awk '{ SUM += $2} END { print SUM/1024/1024"  MB }'

For ex:

rbd diff rdb/test_image | awk '{ SUM += $2 } END { print SUM/1024/1024 "  MB }'
Ceph

Ceph: RBD volumes/images live resize

The below are the steps to live resize of a mounted ext4 file system can then be done as follows:

Steps to shrink/grow the volume size:

Step#1: Unmount the voulme from the VM:

      > umount /dev/vda

Step#2: Run w2fsck on the unmounted file system.

     >  e2fsck -f /dev/vda

Step#3: Run the resize2fs  – to to resize the file system with the resize2fs /dev/device

> resize2fs /dev/vda  1024G       // here 1024G is new size

Now login into ceph storage node and run the below rbd resize commands:

Step#4: Resize the volume using the command.

> rbd resize --size 1048576  rbd/rbd-6196b6bf-f7e9-4935-b92c-9ab602ce2be2 --allow-shrink

   Note: size option will takes in MB, so pass the new size with in MB format.

Step#5: Check if the rbd volume is resized to new size using the below command:

     > rbd info rbd/rbd-6196b6bf-f7e9-4935-b92c-9ab602ce2be2

Step#5: Now run the resize2fs and 2dfsck commands as below:

     > e2fsck -fy /dev/vda
     > resize2fs /dev/vda

  Step#6: Now mount the volume and list the devices.

          > mount -a

Now you could see the resized volume on your VM.

Ceph

PGs stuck unclean state – help

I have observed the PGs stuck unclean state when I added a new OSDs to the cluster. When I have added new OSDs, cluster started rebalancing, after rebalancing a few PGs go into “stuck unclean” state. Here are a few tips to address the stuck unclean PGs.

Its  normal for placement groups to enter states like “degraded” or “peering” following a failure. Normally these states indicate the normal progression through the failure recovery process. However, if a placement group stays in one of these states for a long time this may be an indication of a larger problem. For this reason, the monitor will warn when placement groups get “stuck” in a non-optimal state. Specifically, we check for:

  • inactive – The placement group has not been active for too long (i.e., it hasn’t been able to service read/write requests).
  • unclean – The placement group has not been clean for too long (i.e., it hasn’t been able to completely recover from a previous failure).
  • stale – The placement group status has not been updated by a ceph-osd, indicating that all nodes storing this placement group may be down.

The monitor warns about PGs that are stuck in the same status for some time. They can be listed with:

ceph pg dump_stuck stale
ceph pg dump_stuck inactive
ceph pg dump_stuck unclean

Case#1 – For stuck stale placement groups, it is normally a matter of getting the right ceph-osd daemons running again.

Case#2 – For stuck inactive placement groups, it is usually a peering problem.

Case#3 – For stuck unclean placement groups, there is usually something preventing  recovery from completing, like unfound objects.

Debugging:

Examination of cluster state and PG statuses:

ceph osd pg dump # to find PG number for any status
ceph pg {pg_id} query #to see verbose information about PG in JSON

While analyzing the query output, special attention should be paid on “info”, “peer_info”, and “recovery_status” sections.

restart ceph-osd id=252
Ceph

How to deactivate scrub and deep-scrub

How to deactivate or disable the scrubbing and deep scrubbing operations?

Scrub and deep-scrub operations were deactivated with the below commands:

# ceph used set noscrub
  set noscrub

After setting noscrub, ceph health becomes WARN state as below:
#ceph health
HEALTH_WARN noscrub flag(s) set
# ceph osd set nodeep-scrub
  set nodeep-scrub

To restore the scrub and deep-scrub operations, use the below commands:

# ceph osd unset noscrub
  unset noscrub

After setting noscrub, ceph health becomes WARN state as below:
#ceph health
HEALTH_WARN nodeep-scrub flag(s) set
# ceph osd unset nodeep-scrub
 unset nodeep-scrub