FreeBSD virtual environment management and repository

2020-10 upd: we reached the first fundraising goal and rented a server in Hetzner for development! Thank you for donating !

Attention! I apologize for the automatic translation of this text. You can improve it by sending me a more correct version of the text or fix html pages via GITHUB repository.

Creating CBSD images with Puppet

Current HowTo is designed for those who are already familiar with the CBSD virtual environment management system and Puppet configuration manager.

At some point in time, you might want to create your own 'jail registry' (by analogy with Docker registry) to create and upload your own images to the image repository

With these scripts you can create CBSD images from Puppet code. This can be useful, for example, in such cases:

  • - You have an existing Puppet code base and want to split some of your services into container. A joint using code between containers and a non-containerized environment will allow you to minimize code dublication and save time on infrastructure maintenance;
  • - You create a large number of images with customized services and certain parameters. In this case, Puppet allows you to use a declarative language by describing your images and using script you can create and update your images quite simply and quickly.

Setup

Our implementation will consist of two parts. The first is a file structure that contains regular Puppet resources - the standard Puppet-manifest, which describes your environment and Puppetfile. In the Puppetfile you must describe the modules that you need for the created manifest.

The second part is the script that creates the container, launches r10k on Puppetfile and execute puppet apply.

This HowTo addresses to server-less architecture. However, you can easily use your Puppet server instead of local manifests.

So, create a working directory /root/buid-jail. In it, we will store per-instance directories with all the necessary files. For example, for the service 'redis' we will use the directory /root/build-jail/redis:

% mkdir -p /root/build-jail/redis

Inside this directory, create a directory 'manifest' in which we create the Puppet manifest. In it, you should describe everything that you write in the usual manifest Puppet server:

% mkdir /root/build-jail/redis/manifest

Since our example is aimed at demonstrating the launch of redis, we attach the 'redis' class:

% echo "class { 'redis': }" > /root/build-jail/redis/manifest/init.pp

To expand the example, and demonstrate the possibility of using Hiera, write some parameters in the yaml backend. To do this, create the appropriate directory:

% mkdir /root/build-jail/redis/hieradata

And we'll write some parameters in yaml format as you do in regular yaml Puppet server files:

cat > /root/build-jail/redis/hieradata/common.yaml << EOF
redis::manage_repo: false
redis::bind: 0.0.0.0
redis::port: 6379
redis::log_level: warning
EOF

Next, we need to write Puppetfile. We will need the redis module itself, as well as its dependencies:

cat > /root/build-jail/redis/Cbsdfile <<EOF
forge 'https://forgeapi.puppetlabs.com'

mod 'arioch-redis', '1.2.4'

mod 'puppetlabs/apt'
mod 'puppetlabs/stdlib'
mod 'stahnma/epel'
EOF

Now, we need to write a script that will create a jail from the template and apply Puppet inside.

The template itself, where we replaced our working directory with the construction #workdir# and jname - with #jname#, would look like this:

relative_path="1";
jname="#jname#";
path="#workdir#/jails/#jname#";
host_hostname="#jname#.my.domain";
ip4_addr="DHCP";
mount_devfs="1";
allow_mount="1";
allow_devfs="1";
allow_nullfs="1";
mount_fstab="#workdir#/jails-fstab/fstab.#jname#";
arch="native";
mkhostsfile="1";
devfs_ruleset="4";
ver="native";
basename="";
baserw="0";
mount_src="0";
mount_obj="0";
mount_kernel="0";
mount_ports="1";
astart="1";
data="#workdir#/jails-data/#jname#-data";
vnet="0";
nic_hwaddr="0";
applytpl="1";
mdsize="0";
rcconf="#workdir#/jails-rcconf/rc.conf_#jname#";
floatresolv="1";
zfs_snapsrc="";

exec_poststart="0";
exec_poststop="";
exec_prestart="0";
exec_prestop="0";

exec_master_poststart="0";
exec_master_poststop="0";
exec_master_prestart="0";
exec_master_prestop="0";
pkg_bootstrap="1";
with_img_helpers="";
runasap="0";
interface="auto";
jailskeldir="#workdir#/share/FreeBSD-jail-skel";
jail_profile="default";
# root password
user_pw_root='cbsd';
exec_start="/bin/sh /etc/rc"
exec_stop="/bin/sh /etc/rc.shutdown"
emulator="jail"

Save this template as /root/build-jail/tpl.jconf

You can get this template yourself by running 'cbsd jconstruct-tui', entering the required parameters and answering 'no' to the question 'Do you want to create jail immediately?'.

At the output, you get the same template in which you need to replace your working directory with #workdir# and the name of the jail on the #jname#.

The script itself can have something like the following (call it Cbsdfile):

#!/bin/sh
JAILNAME="jredis"

##
MYDIR="$( /usr/bin/dirname $0 )"
MYPATH="$( /bin/realpath ${MYDIR} )"

. /etc/rc.conf

cbsd jstatus jname=${JAILNAME} > /dev/null 2>&1

ret=$?

if [ ${ret} -ne 0 ]; then
        echo "Warning, jail exist. Please remove them first"
        # cbsd jremove ${JAILNAME}
        exit 0
fi

tmpjconf=$( mktemp )

trap "/bin/rm -f ${tmpjconf}" HUP INT QUIT ABRT KILL ALRM TERM BUS EXIT

sed -e "s:#jname#:${JAILNAME}:g" \
-e "s:#workdir#:${cbsd_workdir}:g" ${MYPATH}/tpl.jconf > ${tmpjconf}

cbsd jcreate jconf=${tmpjconf}
cbsd jstart inter=0 ${JAILNAME}

cbsd jexec jname=${JAILNAME} pkg update
cbsd jexec jname=${JAILNAME} pkg install -y sysutils/puppet4 ca_root_nss rubygem-r10k

cbsd jailscp ${MYPATH}/Puppetfile ${JAILNAME}:/tmp/Puppetfile

cbsd jexec jname=${JAILNAME} "cd /tmp && r10k puppetfile install --moduledir /tmp/module"

[ -f ${MYPATH}/hiera.yaml ] && cbsd jailscp ${MYPATH}/hiera.yaml ${JAILNAME}:/tmp/hiera.yaml
[ -d ${MYPATH}/hieradata ] && cbsd jailscp ${MYPATH}/hieradata/ ${JAILNAME}:/tmp/hieradata
cbsd jailscp ${MYPATH}/manifests/ ${JAILNAME}:/tmp/manifests

cbsd jexec jname=${JAILNAME} env FACTER_hostname=${JAILNAME} puppet apply /tmp/manifests/init.pp --detailed-exitcodes --verbose --show_diff --summarize --hiera_config=/tmp/hiera.yaml --app_management --modulepath=/tmp/module

cbsd jexec jname=${JAILNAME} pkg clean -y

# And push to registry:
#   cbsd jstop jname=${JAILNAME}
#   cbsd jexport jname=${JAILNAME}
#   scp ~cbsd/exports/${JAILNAME} builder@repo.my.domain:/path/.../
#   ..

exit 0
 

Note that the name of the created container is specified in this script in the JAILNAME parameter.

You can put this parameter outside the script in the configuration file, or take it as an argument.

Also, you might want to not check for the existence of the container and delete it automatically at startup every time. To do this, uncomment the line with 'cbsd jremove ${JAILNAME}'

At the output of this script, we get a working jail named jredis with a configured and started redis service inside.

Then you can do anything with this, for example, clone or export and push your image to repository.

You can turn a similar script into a container creation service within a company where your employees (for example, developers) configure the container settings themselves and give them 'order' to build images with custom settings

For example, you can create a Jenkins Job for the above case by creating a Job that has been parametized, in which the user can change the parameters you specify (this is how the author of the article used to build the Docker images came in)

See an example of how it might look like: https://youtu.be/AQ7FbCHhIBk

Note that most of the Puppet modules do not have FreeBSD support (like any other configuration management system). In this regard, the author collects (and adds support for FreeBSD by virtue of its capabilities) modules that work with FreeBSD. For this purpose, a group PuppetBSD was created on the GitHub resource, where you want collect such modules in one place. If you use Puppet and FreeBSD, please add such modules to this group. This will help many FreeBSD users and can benefit from it.