CBSD jail on GlusterFS HOWTO

This howto demonstrates CBSD jail setup on cluster filesystem GluserFS

Please note that replication causes additional overhead for system resources. You will need about +1Gb RAM for glusterfs processes. Also, it is desirable to have fast hard drives (SSD and 1Gbit/s give you aprox. 250 Gbit/s on gluster(?)) In addition, avoid such installations for containers with a high write load

---

Prepare:

1) Install sysutils/cbsd

2) Init CBSD workdir: env workdir=/usr/jails /usr/local/cbsd/sudoexec/initenv

Glusterfs setup:

1) kldload fuse

2) sysrc -qf /boot/loader.conf fuse_load="YES"

3) zfs create -o mountpoint=/glusterfs zroot/glusterfs

4) pkg install -y net/glusterfs

5) sysrc glusterd_enable="YES"

6) service glusterd start

7) vi /etc/hosts: - add IP/name for each node, e.g:

192.168.0.30 glu1
192.168.0.31 glu2
	

8) mkdir /glusterfs/brick1 /glusterfs/brick2 /glusterfs/brick3

glu1% gluster peer probe glu2

glu2% gluster peer status

glu1% (optional): gluster peer probe glu1

glu1% gluster pool list

glu1% gluster volume create jails-data replica 2 glu1:/glusterfs/brick1 glu2:/glusterfs/brick1

glu1% gluster volume create jails-system replica 2 glu1:/glusterfs/brick2 glu2:/glusterfs/brick2

glu1% gluster volume create jails-fstab replica 2 glu1:/glusterfs/brick3 glu2:/glusterfs/brick3

glu1% gluster volume start jails-data

glu1% gluster volume start jails-system

glu1% gluster volume start jails-fstab

glu1% gluster volume bitrot jails-data enable

glu1% gluster volume heal jails-data enable

glu1% gluster volume bitrot jails-system enable

glu1% gluster volume heal jails-system enable

glu1% gluster volume bitrot jails-fstab enable

glu1% gluster volume heal jails-fstab enable

Check state:

gluster volume info jails-data

gluster volume info jails-fstab

gluster volume info jails-system

Automount for flusterfs volumes:

Create script in /usr/local/etc/rc.d/gluster_mount:

#!/bin/sh
# PROVIDE: gluster_mount
# REQUIRE: LOGIN glusterd
# KEYWORD: shutdown

. /etc/rc.subr

name="gluster_mount" 
rcvar="glusterd_enable" 
start_cmd="gluster_mount" 
stop_cmd="gluster_umount" 

gluster_mount()
{
        [ ! -d ${cbsd_workdir}/jails-data/.trashcan ] && /usr/local/sbin/mount_glusterfs localhost:/jails-data ${cbsd_workdir}/jails-data
        [ ! -d ${cbsd_workdir}/jails-system/.trashcan ] && /usr/local/sbin/mount_glusterfs localhost:/jails-system ${cbsd_workdir}/jails-system
        [ ! -d ${cbsd_workdir}/jails-fstab/.trashcan ] && /usr/local/sbin/mount_glusterfs localhost:/jails-system ${cbsd_workdir}/jails-fstab
}

gluster_umount()
{
        [ -d ${cbsd_workdir}/jails-data/.trashcan ] && /sbin/umount ${cbsd_workdir}/jails-data
        [ -d ${cbsd_workdir}/jails-system/.trashcan ] && /sbin/umount ${cbsd_workdir}/jails-system
        [ -d ${cbsd_workdir}/jails-fstab/.trashcan ] && /sbin/umount ${cbsd_workdir}/jails-fstab
}

load_rc_config $name
run_rc_command "$1" 

chmod +x /usr/local/etc/rc.d/gluster_mount

service gluster_mount start

Re-run CBSD initenv to invert zfsfeat (automatically):

cbsd initenv

Profit:

Create jail and start it as usual