Ceph: Health Err Repair PG

Ceph  cluster state shows the below  status:

health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors

Debugging the pg inconsistent and scrub errors are tricky and its not straight forward to proceed with the above error state.


Lets analyze:

Check the ceph health details using the below command:

$ sudo ceph health detail
HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
pg 40.27a is active+clean+inconsistent, acting [124,36,78] 1 scrub errors

With above command ouput,  the  PG is “40.27a"  has issue and its acting on  osd.124, osd.36 and  osd.78.

In this case, a quick way to fix this is with help of ceph pg repain command as below:

$ ceph pg repair <pg-id> 
$ ceph pg repair 40.27a 
1. If you see 'active+clean+inconsistent' states, this may happen due to an error during scrubbing.
   We can repair the PG with help of the above command.
2. This command will take a few minutes, wait for a few minutes and check the status again.

If the above repair command fixes the issue, all set, else we need to dig the issue and manually fix the issue.
Debug the issue:

To debug the issue, need to dig into the ceph  osd logs using ‘grep’ command as below:

$ grep ERR /var/log/ceph/ceph-osd.124.log
osdceph..3-12log [ERR]:  40.27a   67adb2b1/rb.0.83213.438f1f29.00000001242d/head//27 digest 0 != known digest 4042795225FIND THE OBJECT

The above output says  that the object digest id should be 4042795225, but is actually it is shown as “0”.


Fix the bad object manually:

Move the object away  with the following:

  • stop the OSD that has the wrong object responsible for that PG
    •  $ service ceph stop osd=<osd_id>
  • Flush the journal to avoid any writes pending.
    • $ ceph-osdi id –flush-journal
  • Move the bad object to tmp location
    • mv <obj-id> /tmp
  • Start the OSD
    •  $ service ceph stop osd=<osd_id>
  • Now, run the pg rapair cmd:
    • $ ceph pg repair 40.27a

RDB: List the volume dependencies

Here I will discuss to list of all dependencies of a ceph rbd volumes.

Below are the list of commands to find the volumes dependencies:

  1. rbd ls -l volumes | grep <volume_id>
  2. rdb info volumes/<volume-id>

Glance images quick download with Ceph

Currently Ceph RBD backed  disks are created by downloading an image file from Glance to a local file, then uploading that file into Ceph RBD. Here, uploading may take a long time, because “rbd import” is synchronous and slow.

If the glance  images are already stored in Ceph RBD, then there is no need download to a local directory and local copies – it can be cloned to a new image for a new disk without copying the data at all.

Glance and Cinder use the Ceph rbd as back-end, use the below steps to do the quick upload and download the image files.


  • Use the ceph rbd as glance storage.
    • defalut_store=rbd
  • Use Cinder back-end as ceph rdb
    • enabled_backends=ceph
  • Use only RAW format of the images.

Now, follow the below configurations for cinder and glance for quicker images download:


volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
glance_api_version = 2

Note: Make sure that, cinder uses the glance API V2 version ‘glance_api_version = 2"


show_image_direct_url = True
show_multiple_locations = Ture


With above settings,  glance images not download to local directory and no copy. It uses the ceph clone functionality and bootable volume creation takes quickly.


  1. https://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/rbd-clone-image-handler.html
  2. http://docs.ceph.com/docs/master/dev/rbd-layering/