Ceph

Ceph node – CPU Performance setting

The default value for CPU performance is “ondemand”. It means CPU will increase or decrease CPU frequency based on average load and usage statistic. Most of the time it will be 50% of maximum CPU frequency.

The recommended value of Ceph OSD nodes is “performance“. It means the server will use 100% CPU frequency 100% time.

NOTE: CPU performance setting will increase the power consumption of that server.

Run the below command to set the CPU performance mode:

echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Note that the above command will work till the node reboot only. Once node reboot, need to run the command again.

Use “cpufrequtil” tool to set above setting permanently.

cpufreq-set -r -g performance

After installing the cpufrequtils, up update the “/etc/init.d/cpufrequtils” file with  “GOVERNOR=”performance“”

 

 

 

Advertisements
Ceph, Swift

Swift: SSBENCH – Setup and running

Here, will describe the SSBENCH (Object storage performance suite)  configuration setup and running the same on Ceph Radosgw as object storage.

Step#1 – Install the ssbench tools from the internet using below links:                                      https://github.com/swiftstack/ssbench

Step#2: Get the Auth from the Swift:

2.1  Create ceph radosgw user as “ssbench”and add subuser “operator” as below. And also set the Swift_keys for “ssbench:operator”  and key: “abc123

Please refer the below ceph docs for adding user and sub-user:
http://docs.ceph.com/docs/hammer/radosgw/admin/
$radosgw-admin user info --uid=ssbench
{ "user_id": "ssbench",
"display_name": "ssbench",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [
       { "id": "ssbench:operator",
           "permissions": "full-control"}],
"keys": [
       { "user": "ssbench:operator",
         "access_key": "2BH3N0R3OTO2TFDK0A3W",
         "secret_key": ""},
       { "user": "ssbench",
         "access_key": "4TSCMJWGHC8Q0CTWYXCR",
         "secret_key": "NRDIRCNqdcwdAPdnD5Bw8s5+ou6wepiGiJRu2pA4"}],
"swift_keys": [
       { "user": "ssbench:operator",
         "secret_key": "abc123"}],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": { "enabled": false,
     "max_size_kb": -1,
     "max_objects": -1},
"user_quota": { "enabled": false,
     "max_size_kb": -1,
     "max_objects": -1},
"temp_url_keys": []}

2.2. Use the below CURL command to get the Auth key:

$ curl -v -H "X-Auth-user: ssbench:operator" -H 'X-Auth-key: abc123'  https://localhost:8080/auth/v1 -k

2.3. Now, run the ssbench-master command using the above Auth-key as below:

ssbench-master run-scenario –os-auth-token AUTH_rgwtk11000000636f7362656e63683a6f70657261746f72593b90c6c7c6a08b5f47b45770c15f230212d1f3bbcd8b190ee36af7cd63554ae904d726  –os-storage-url https://localhost:8080/swift/v1    –workers 4   -f /root/swami/test.scenario

Here is the sample scenario file:

root@swami-test:~/swami# cat /root/swami/demo_test.scenario
{
  "name": "Demo test scenario",
  "sizes": [{
    "name": "tiny",
    "size_min": 2048000000,
    "size_max": 2048000000
  }
  ],
  "initial_files": {
    "tiny": 1
  },
  "operation_count": 1,
  "crud_profile": [0, 0, 1, 0],
  "user_count": 1,
  "container_base": "ssbench",
  "container_count": 1,
  "container_concurrency": 10
}

 

Ceph

Ceph: noup and nodown flags

If something (like network issue,etc) is causing OSDs to ‘flap’ (repeatedly getting marked down and then up again), you can force the monitors to stop the flapping with:

ceph osd set noup      # prevent OSDs from getting marked up
ceph osd set nodown    # prevent OSDs from getting marked down

These flags are recorded in the osdmap structure:

ceph osd dump | grep flags
flags no-up,no-down

You can clear the flags with:

ceph osd unset noup
ceph osd unset nodow

Two other flags are supported, noin and noout, which prevent booting OSDs from being marked in (allocated data) or protect OSDs from eventually being marked out (regardless of what the current value for mon osd down out interval is).

Note:
noup, noout, and nodown are temporary in the sense that once the flags are 
cleared, the action they were blocking should occur shortly after. 
The noin flag, on the other hand, prevents OSDs from being marked in on boot, 
and any daemons that started while the flag was set will remain that way.

References:

http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/