Ceph

Ceph: noout flag

Here, will discuss the ceph “noout” flag details. If you do not want CRUSH to automatically rebalance the cluster as you stop an OSDs for maintenance, set the cluster to  “noout” first.  An OSD marked as   “out” means that it might be  running, but it  doesn’t   part of the cluster CRUSH map.  So this “noout” flag prevents OSDs from being marked as “out” from the cluster. With default configurations, the ceph monitors mark the OSDs out if it is not reachable for  5 minutes i.e 300 seconds.   Use the below command to know the default value set in your ceph cluster:

# ceph daemon /var/run/ceph/cephmon.*.asok config show | grep mon_osd_down_out_interval

To set the noout flag:

$ ceph osd set noout
set noout

After setting the noout, ceph health going into WARN state:

$ ceph health
 health HEALTH_WARN noout flag(s) set

The “noout” flag  tells the ceph monitors  not to “out” any OSDs from the crush map and not to start recovery and re-balance activities,  to maintain the replica count.

Please note that:
1. Ceph  cluster should carefully monitor as any additional host/OSD outage may 
   cause placement groups to become unavailable. 
2. Ceph cluster will be in "HEALTH_WARN"  state, as long as you unset the noout.
3. noout flag is temporary in the sense that, once the flag "unset", the action
   it was blocking should occur shortly after. 

To unset the noout flag:

$ ceph osd unset noout
unset noout

 

Advertisements
Cinder, Openstack

Openstack micro version APIs usage with CURL

Openstack supports the micro version for its APIs. Here is quick tips to use the micro versioned API with CURL command.

First get a token from keystone:

swami@ubuntu:~/devstack$ openstack token issue
+------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2016-07-15T12:56:35Z             |
| id         | 49f5673ad7954f68a46702d52dede0df |
| project_id | 4e5f6ba3f2ba4e68bce144f1f9eb843a |
| user_id    | 68d155723d7342d3ad29047ca0ac378c |
+------------+----------------------------------+

Get the cinder CURL command from “cinder –debug list”

curl -g -i -X GET 
http://10.0.2.15:8776/v2/4e5f6ba3f2ba4e68bce144f1f9eb843a/volumes/detail  
-H "Accept: application/json" -H "X-Auth-Token:49f5673ad7954f68a46702d52dede0df"
HTTP/1.1 200 OK
X-Compute-Request-Id: req-ec992a49-b644-4b53-a9fa-5f3b36ea10c1
Content-Type: application/json
Content-Length: 15
X-Openstack-Request-Id: req-ec992a49-b644-4b53-a9fa-5f3b36ea10c1
Date: Fri, 15 Jul 2016 12:01:49 GMT
{"volumes": []}swami@ubuntu:~/devstack$

Now use the micro-versions using the hearders as below:

curl -g -i -X GET 
http://10.0.2.15:8776/v2/4e5f6ba3f2ba4e68bce144f1f9eb843a/volumes/detail  
-H "Accept: application/json" -H "X-Auth-Token:49f5673ad7954f68a46702d52dede0df"
-H "OpenStack-API-version: volume 3.0"
HTTP/1.1 200 OK
X-Compute-Request-Id: req-154b85db-35ed-4d2b-91eb-2751ba74d8d7
Content-Type: application/json
Content-Length: 1035
X-Openstack-Request-Id: req-154b85db-35ed-4d2b-91eb-2751ba74d8d7
Date: Fri, 15 Jul 2016 12:15:40 GMT
{"volumes": [{"migration_status": null, "attachments": [], "links": 
[{"href": "http://10.0.2.15:8776/v2/4e5f6ba3f2ba4e68bce144f1f9eb843a/
volumes/109a2cb0-2ad2-495f-8334-4593ab06a199", "rel": "self"}, {"href": "
http://10.0.2.15:8776/4e5f6ba3f2ba4e68bce144f1f9eb843a/volumes/109a2cb0-2ad2-495f-8334-4593ab06a199", "rel": "bookmark"}], "availability_zone": "nova", "os-vol-host-attr:host": "ubuntu@lvmdriver-1#lvmdriver-1", "encrypted": false, "updated_at": "2016-07-15T12:07:56.000000", "replication_status": "disabled", "snapshot_id": null, "id": "109a2cb0-2ad2-495f-8334-4593ab06a199", "size": 1, "user_id": "68d155723d7342d3ad29047ca0ac378c", "os-vol-tenant-attr:tenant_id": "4e5f6ba3f2ba4e68bce144f1f9eb843a", "os-vol-mig-status-attr:migstat": null, "metadata": {}, "status": "available", "description": null, "multiattach": false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": null, "bootable": "false", "created_at": "2016-07-15T12:07:51.000000", "volume_type": "lvmdriver-1"}]}swami
HTTP/1.1 200 OK
X-Compute-Request-Id: req-ec992a49-b644-4b53-a9fa-5f3b36ea10c1
Content-Type: application/json
Content-Length: 15
X-Openstack-Request-Id: req-ec992a49-b644-4b53-a9fa-5f3b36ea10c1
Date: Fri, 15 Jul 2016 12:01:49 GMT
{"volumes": []}swami@ubuntu:~/devstack$

If you use a micro version API, which is not supported, then cinder will give the min and max micro versions it supported as below . here I used version 3.7, which is not supported:

curl -g -i -X GET 
http://10.0.2.15:8776/v2/4e5f6ba3f2ba4e68bce144f1f9eb843a/volumes/detail  
-H "Accept: application/json" -H "X-Auth-Token:49f5673ad7954f68a46702d52dede0df"
-H "OpenStack-API-version: volume 3.7"
HTTP/1.1 406 Not Acceptable
Openstack-Api-Version: 3.7
Vary: OpenStack-API-Version
Content-Length: 121
Content-Type: application/json; charset=UTF-8
X-Compute-Request-Id: req-2f89acb7-f7d9-44c1-be06-8854e18332df
X-Openstack-Request-Id: req-2f89acb7-f7d9-44c1-be06-8854e18332df
Date: Fri, 15 Jul 2016 12:26:56 GMT

{"computeFault": {"message": "Version 3.7 is not supported by the API. 
Minimum is 3.0 and maximum is 3.6.", "code": 406}}

Here we can use other component’s API micro versioned APIs in the hearders as below:

curl -g -i -X GET 
http://10.0.2.15:8776/v2/4e5f6ba3f2ba4e68bce144f1f9eb843a/volumes/detail  
-H "Accept: application/json" -H "X-Auth-Token:49f5673ad7954f68a46702d52dede0df"
-H "OpenStack-API-version: volume 3.6, compute 2.21 identity 3.0"
HTTP/1.1 200 OK
X-Compute-Request-Id: req-17ad037c-3230-47be-bf54-8e9267ca24e3
Content-Type: application/json
Content-Length: 1035
Openstack-Api-Version: volume 3.6
Vary: OpenStack-API-Version
X-Openstack-Request-Id: req-17ad037c-3230-47be-bf54-8e9267ca24e3
Date: Fri, 15 Jul 2016 12:33:42 GMT

{"volumes": [{"migration_status": null, "attachments": [], "links": [{"href": 
"http://10.0.2.15:8776/v3/4e5f6ba3f2ba4e68bce144f1f9eb843a/volumes/
109a2cb0-2ad2-495f-8334-4593ab06a199", "rel": "self"}, {"href": 
"http://10.0.2.15:8776/4e5f6ba3f2ba4e68bce144f1f9eb843a/volumes/
109a2cb0-2ad2-495f-8334-4593ab06a199", "rel": "bookmark"}], "availability_zone":
"nova", "os-vol-host-attr:host": "ubuntu@lvmdriver-1#lvmdriver-1", "encrypted": 
false, "updated_at": "2016-07-15T12:07:56.000000", "replication_status":
 "disabled", "snapshot_id": null, "id": "109a2cb0-2ad2-495f-8334-4593ab06a199", 
"size": 1, "user_id": "68d155723d7342d3ad29047ca0ac378c", "os-vol-tenant-attr:
tenant_id": "4e5f6ba3f2ba4e68bce144f1f9eb843a", "os-vol-mig-status-attr:migstat":
 null, "metadata": {}, "status": "available", "description": null, "multiattach":
 false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:
name_id": null, "name": null, "bootable": "false", "created_at": 
"2016-07-15T12:07:51.000000", "volume_type": "lvmdriver-1"}]}

 

Ceph, Openstack

RGW – Enable Keystone Auth for S3 APIs

Ceph rados-gateway (RGW) enables rados based automation internally. If user wants to use the ceph rgw with keystone authentication.

Add the below 2 lines in /etc/ceph/ceph.conf  in [client.radosgw.gateway] section:

rgw_keystone_url = locahost:35357
rgw_s3_auth_use_keystone=True

Note: Replace the port number  “rgw_keystone_url” from 5000 to 35357

Now, restart the radosgw services:

/etc/init.d/radosgw restart

 

Ceph

Remove an OSD from cluster gracefully

Here, I will discuss the process of removing an OSD from Ceph cluster gracefully without impacting the customer operations and network bandwidth. Please refer the below process to remove an OSD from the cluster.

Pre-requisites:

To make less impact  the ceph client/user operations performance, set the below configuration options and should be rolled-backed to default after the OSD removal process:

# ceph tell osd.* injectargs '--osd_max_backfills 1'              // Default 10
# ceph tell osd.* injectargs '--osd_recovery_max_active 1'        // Default 15
# ceph tell osd.* injectargs '--osd_recovery_op_priority 1'       // Default 10
# ceph tell osd.* injectargs '--osd_recovery_max_single_start 1'  // Default 5

Note: The above setting will slow down the recovery/backfill process and prolongs the osd removal process.

Step#1:

Reduce the OSD’s weight from 1 to 0 gradually -1 offset. For ex:

1 -> 0.9 -> 0.8 -> 0.7 -> 0.6 -> 0.5 -> 0.4 -> 0.3 -> 0.2 -> 0.1 -> 0.0

$ ceph osd  crush reweight osd.{osd-id}  <new_weight> 
 //  Where new_weight is 0.9 -> 0.0
 // Where id is osd-id like osd.0, isd.1 etc.

For ex: ceph osd crush reqeight osd.1 0.9

Note: With each new_weight, ceph cluster starts to do the re-balance
      starts by moving the data from this OSD to other OSDs.

Step#2:

Now, take out the OSD, which is in ceph cluster using the below command:

$ ceph osd out osd.{osd-id}

For ex: ceph osd out osd.1

Step#3:

Now, stop the OSD process/daemon and

$ /etc/init.d/ceph  stop  osd.{osd-id}

For ex: /etc/init.d/ceph stop osd.1   
Note: Above command requires sudo access

Once you stop the OSD, it is down.

Step#4:

Remove the specific OSD from the cluster’s crushmap.

$ ceph osd crush remove  osd.{osd-id}

For ex: ceph osd crush remove osd.1

Step#5:

Remove the OSD authentication key from the cluster.

$ ceph auth del osd.{osd-id}

For ex: ceph auth del osd.1

Step#6:

Now remove the osd

$ceph osd rm {osd-id}
#for ex: $ ceph osd rm 1

 

Rollback all the configurations set before starting the step#1:

# ceph tell osd.* injectargs '--osd_max_backfills 10'            // Default 10
# ceph tell osd.* injectargs '--osd_recovery_max_active 15'       // Default 15 
# ceph tellosd.*inj  ectargs '--osd_recovery_op_priority 10'      // Default 10
# ceph tellosd.inj * ectargs '--osd_recovery_max_single_start 5'   // Default 5

NOTE: If  an OSD is down due to Hardware error, then you can skip the step#1 in the above process and just use from step#2 to remove that specific OSD.