Addressing how to configure multiple GlusterFS backend driver to an existing OpenStack setup.
Few assumptions:
1. that you already have GlusterFS nodes installed.2. that you already have an existing Openstack setup.
GlusterFS setup:
create 2 distributed replicate volumes.
creating first volume for cinder
# gluster vol create cinder-vol replica 2 \
xx.yy.zz.183:/rhs/brick1/cinder-vol \
xx.yy.zz.223:/rhs/brick1/cinder-vol \
xx.yy.zz.183:/rhs/brick2/cinder-vol \
xx.yy.zz.223:/rhs/brick2/cinder-vol
# gluster vol set cinder-vol group virt
# gluster vol set cinder-vol storage.owner-uid 165
# gluster vol set cinder-vol storage.owner-gid 165
# gluster vol info cinder-vol
Volume Name: cinder-vol
Type: Distributed-Replicate
Volume ID: 2f4edaef-678b-492a-b972-bd95c1c490a3
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: xx.yy.zz.183:/rhs/brick1/cinder-vol
Brick2: xx.yy.zz.223:/rhs/brick1/cinder-vol
Brick3: xx.yy.zz.183:/rhs/brick2/cinder-vol
Brick4: xx.yy.zz.223:/rhs/brick2/cinder-vol
Options Reconfigured:
storage.owner-gid: 165
storage.owner-uid: 165
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
# gluster vol start cinder-vol
volume start: cinder-vol: success
Creating second volume for cinder
# gluster vol create cinder-vol2 replica 2 \
xx.yy.zz.183:/rhs/brick1/cinder-vol2 \
xx.yy.zz.223:/rhs/brick1/cinder-vol2 \
xx.yy.zz.183:/rhs/brick2/cinder-vol2 \
xx.yy.zz.223:/rhs/brick2/cinder-vol2
# gluster vol set cinder-vol2 group virt
# gluster vol set cinder-vol2 storage.owner-uid 165
# gluster vol set cinder-vol2 storage.owner-gid 165
# gluster vol info cinder-vol2
Volume Name: cinder-vol2
Type: Distributed-Replicate
Volume ID: a3d4831a-d754-4072-9a3a-c6641759ec09
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: xx.yy.zz.183:/rhs/brick1/cinder-vol2
Brick2: xx.yy.zz.223:/rhs/brick1/cinder-vol2
Brick3: xx.yy.zz.183:/rhs/brick2/cinder-vol2
Brick4: xx.yy.zz.223:/rhs/brick2/cinder-vol2
Options Reconfigured:
storage.owner-gid: 165
storage.owner-uid: 165
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
# gluster vol start cinder-vol2
volume start: cinder-vol2: success
From your allinone openstack node verify that the volumes are available to rule out any network issue.
# showmount -e xx.yy.zz.223
Export list for xx.yy.zz.223:
/cinder-vol *
/cinder-vol2 *
Configuring openstack to use multiple GlusterFS drivers
Defining the names of the groups for multiple volume backends
# openstack-config --set /etc/cinder/cinder.conf DEFAULT enabled_backends GLUSTERFS_DRIVER1,GLUSTERFS_DRIVER2
Specifying cinder.volume.drivers.glusterfs options in /etc/cinder/cinder.conf for both the drivers
GLUSTERFS_DRIVER1:
# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER1 volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver
# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER1 glusterfs_shares_config /etc/cinder/cinder-volume1.conf
# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER1 glusterfs_mount_point_base /var/lib/cinder/cinder-volume1
# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER1 volume_backend_name GLUSTERFS_cinder_volume1
GLUSTERFS_DRIVER2:
# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER2 volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver
# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER2 glusterfs_shares_config /etc/cinder/cinder-volume2.conf
# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER2 glusterfs_mount_point_base /var/lib/cinder/cinder-volume2
# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER2 volume_backend_name GLUSTERFS_cinder_volume2
Adding GlusterFS volumes in shares config
# cat > /etc/cinder/cinder-volume1.conf << EOF
> xx.yy.zz.183:cinder-vol
> EOF
# cat > /etc/cinder/cinder-volume2.conf << EOF
> xx.yy.zz.223:cinder-vol
> EOF
Just restart cinder-volume service
# service openstack-cinder-volume restart
# mount | grep cinder
xx.yy.zz.183:cinder-vol on /var/lib/cinder/cinder-volume1/586c24173ac3ab5d1d43aed1f113d9f6 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
xx.yy.zz.223:cinder-vol2 on /var/lib/cinder/cinder-volume2/2277a5d6ce46f3cd52771e1314f74ce5 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
All set!!!
# cinder-manage host list
host zone
2013-08-01 18:02:29 DEBUG [cinder.utils] backend <module 'cinder.db.sqlalchemy.api' from '/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.pyc'>
dhcpzz-146.shanks.com nova
dhcpzz-146.shanks.com@GLUSTERFS_DRIVER1 nova
dhcpzz-146.shanks.com@GLUSTERFS_DRIVER2 nova
NOTE: The volume_type must be defined in the volume_type table in the database. When creating a volume for a particular backend (or group of > 1 backends) the volume_type is passed in. There can be > 1 backend per volume_type, and the capacity scheduler kicks in and keeps the backends of a particular volume_type even. In order to group a volume_type to a backend, you must define extra specs for the type. src: https://wiki.openstack.org/wiki/Cinder-multi-backend#Volume_Type
# cinder type-create GLUSTERFS_cinder_volume1
# cinder type-create GLUSTERFS_cinder_volume2
(Thanks to Shilpa M Jagannath for point out an error above which is now fixed.)
# cinder type-key glusterfs_cinder_volume1 set volume_backend_name=glusterfs_cinder_volume1
# cinder type-key glusterfs_cinder_volume2 set volume_backend_name=glusterfs_cinder_volume2
# cinder extra-specs-list
+--------------------------------------+--------------------------+-------------------------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+--------------------------+-------------------------------------------------------+
| 50b7f795-29c0-4fe8-8ca3-cf66e2f34886 | glusterfs_cinder_volume1 | {u'volume_backend_name': u'glusterfs_cinder_volume1'} |
| cea39f0e-cb83-4c13-b712-ae29551da19f | glusterfs_cinder_volume2 | {u'volume_backend_name': u'glusterfs_cinder_volume2'} |
+--------------------------------------+--------------------------+-------------------------------------------------------+
Usage and tests
(keystone_admin)]# cinder create --display-name shanks-vol1 --volume-type glusterfs_cinder_volume1 3+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-08-01T13:22:16.810225 |
| display_description | None |
| display_name | shanks-vol1 |
| id | 312d61b8-9abc-4c7c-915a-7e9dfff0b098 |
| metadata | {} |
| size | 3 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | glusterfs_cinder_volume1 |
+---------------------+--------------------------------------+
# ll -h /var/lib/cinder/cinder-volume1/586c24173ac3ab5d1d43aed1f113d9f6/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098
-rw-rw-rw-. 1 root root 3.0G Aug 1 18:52 /var/lib/cinder/cinder-volume1/586c24173ac3ab5d1d43aed1f113d9f6/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098
# getfattr -d -etext -m. -n trusted.glusterfs.pathinfo /var/lib/cinder/cinder-volume1/586c24173ac3ab5d1d43aed1f113d9f6/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098
getfattr: Removing leading '/' from absolute path names
# file: var/lib/cinder/cinder-volume1/586c24173ac3ab5d1d43aed1f113d9f6/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098
trusted.glusterfs.pathinfo="(<DISTRIBUTE:cinder-vol-dht> (<REPLICATE:cinder-vol-replicate-1> <POSIX(/rhs/brick2/cinder-vol):dhcpzz-183.shanks.com:/rhs/brick2/cinder-vol/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098> <POSIX(/rhs/brick2/cinder-vol):dhcpzz-223.shanks.com:/rhs/brick2/cinder-vol/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098>))"
(keystone_admin)]# cinder create --display-name shanks-vol2 --volume-type glusterfs_cinder_volume2 4
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-08-01T13:23:47.948160 |
| display_description | None |
| display_name | shanks-vol2 |
| id | 152fcb35-31ca-4e66-a87e-4b921e3171aa |
| metadata | {} |
| size | 4 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | glusterfs_cinder_volume2 |
+---------------------+--------------------------------------+
# ls -lh /var/lib/cinder/cinder-volume2/2277a5d6ce46f3cd52771e1314f74ce5/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa
-rw-rw-rw-. 1 root root 4.0G Aug 1 18:53 /var/lib/cinder/cinder-volume2/2277a5d6ce46f3cd52771e1314f74ce5/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa
# getfattr -d -etext -m. -n trusted.glusterfs.pathinfo /var/lib/cinder/cinder-volume2/2277a5d6ce46f3cd52771e1314f74ce5/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa
getfattr: Removing leading '/' from absolute path names
# file: var/lib/cinder/cinder-volume2/2277a5d6ce46f3cd52771e1314f74ce5/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa
trusted.glusterfs.pathinfo="(<DISTRIBUTE:cinder-vol2-dht> (<REPLICATE:cinder-vol2-replicate-1> <POSIX(/rhs/brick2/cinder-vol2):dhcpzz-223.shanks.com:/rhs/brick2/cinder-vol2/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa> <POSIX(/rhs/brick2/cinder-vol2):dhcpzz-183.shanks.com:/rhs/brick2/cinder-vol2/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa>))"