Introduction
============
Ceph's vstart.sh utility is very useful for deploying and testing a
mock Ceph cluster directly from the source repository. It can:
- Generate a cluster configuration file and authentication keys
- Provision and deploy a number of OSDs
  + Backed by local disk, or memory using the --memstore parameter
- Deploy an arbitrary number of monitor, MDS or rados gateway nodes

All services are deployed as the running user. I.e. root access is not
needed.

Once deployed, the mock cluster can be used with any of the existing
Ceph client utilities, or exercised with the unit tests in the Ceph
src/test directory.

When developing or testing Linux kernel changes for CephFS or RBD, it's
useful to also be able to use these clients against a vstart.sh deployed
Ceph cluster. This can be done using the procedure outlined below, which
was performed on openSUSE Leap 42.1, but is intended to be portable.


Network Setup
=============
br_setup.sh creates a bridge network device, assigns a network address
to it, and connects two new TAP interfaces to the bridge.
- Ceph OSDs, mons and mdses will listen on the bridge network address
- Kernel client VMs will be connected to the TAP interfaces directly

The bridge network is isolated, and isn't connected to any physical
adapters. All parameters (device names, IP addresses, etc.) are
configured in rapido.conf.
br_setup.sh should be run as root.

For more information on the bridge setup, see:
http://blog.elastocloud.org/2015/07/qemukvm-bridged-network-with-tap.html


Ceph Cluster Deployment
=======================
Once the bridge network has been configured, the Ceph cluster can be
deployed. Invoke vstart.sh via:

> cd $ceph_source_dir
<build ceph>
> cd src
> OSD=3 MON=1 RGW=0 MDS=1 ./vstart.sh -i 192.168.155.1 -n --memstore

Be sure to use the same IP address as was assigned to the bridge
interface (BR_ADDR in rapido.conf).


Ceph Cluster Initialisation
===========================
vstart.sh deploys the Ceph cluster with an RBD image pool and CephFS
filesystem already provisioned:
> cd $ceph_source_dir/src
> ./rados lspools
rbd
cephfs_data_a
cephfs_metadata_a

> ./ceph fs ls
name: cephfs_a, metadata pool: cephfs_metadata_a, data pools: [cephfs_data_a ]

For RBD/LIO testing (via cut_lio_rbd.sh), an RBD image must be created,
matching the rapido.conf CEPH_RBD_POOL/CEPH_RBD_IMAGE values, e.g.:
> ./rbd create --size 1024 iscsi_test
> ./rbd ls -l
NAME           SIZE PARENT FMT PROT LOCK
iscsi_test    1024M          2


Kernel Build
============
Checkout the kernel source, and specify the path in rapido.conf.
Enter the source directory:
> cd $kernel_source_dir

Generate a suitable kernel build config. vanilla_config is provided as a
minimal sample config with all Ceph client components enabled:
> cp $rapido_source_dir/vanilla_config .config && make oldconfig
or
> make menuconfig
	- set CONFIG_BLK_DEV_RBD=y, CONFIG_CEPH_FS=y, CONFIG_CEPH_LIB=y,
	  CONFIG_E1000=y and CONFIG_IP_PNP=y

Compile the kernel, and install the modules into a mods subdirectory:
> make
> INSTALL_MOD_PATH=./mods make modules_install
	- this installs modules into ./mods

Create a link to the modules directory, so that Dracut can find them:
> sudo ln -s $PWD/mods/lib/modules/$(make kernelrelease) \
		/lib/modules/$(make kernelrelease)


Kernel Client VM Generation
===========================
Rapido ships with two scripts for kernel client VM generation:
- cut_lio_rbd.sh: generates an RBD + LIO client VM
  + lio_rbd_autorun.sh is executed by the VM at boot-time
- cut_fstests_cephfs.sh: generates a CephFS + xfstests client VM
  + fstests_cephfs_autorun.sh is executed by the VM at boot-time
  + FSTESTS_DIR must be configured in rapido.conf

The cut_* scripts use Dracut to generate a VM image with all
dependencies included for the corresponding Ceph kernel client. The
images are very lightweight (20M-40M).

The VM images will need to be regenerated if any of the following
components/files are changed:
- rapido.conf
- Ceph vstart.sh cluster configuration/keyring
- Ceph binaries
- Kernel modules

For more information on kernel/Dracut setup, see:
http://blog.elastocloud.org/2015/06/rapid-linux-kernel-devtest-with-qemu.html


Kernel VM Deployment
====================
Once a VM has been generated, it can be booted directly via vm.sh .
The same image can be booted twice, to allow for multiple CephFS/RBD
clients or iSCSI gateways. Network parameters for both VMs are defined
in rapido.conf.

The VMs run the corresponding autorun script (e.g. lio_rbd_autorun.sh or
fstests_cephfs_autorun.sh) during boot, and then present an interactive
Dracut shell. The VMs can be shutdown via the shutdown command.


Kernel RBD/LIO Usage (lio_rbd_autorun.sh)
=====================================
If the client VM was generated using cut_lio_rbd.sh, then
lio_rbd_autorun.sh will be executed on boot. lio_rbd_autorun.sh performs
the following:
- Initialises udev and then mounts configfs and debugfs
- Loads the LIO kernel modules
- Maps the rapido.conf configured RBD image locally
  + As /dev/rbd/${CEPH_RBD_POOL}/${CEPH_RBD_IMAGE}
- Exposes the RBD image via an LIO iSCSI target
  + iSCSI target parameters, such as portal addresses, IQN, etc., are
    all configured in rapido.conf

Once deployed, an iSCSI initiator can connect to the configured iSCSI
portal address(es), and perform I/O against RBD image backed LUN0.

When finished, the RBD image can be unmapped via:
dracut:/# echo -n 0 > /sys/bus/rbd/remove


Kernel CephFS Usage (fstests_cephfs_autorun.sh)
===============================================
If the client VM was generated using cut_fstests_cephfs.sh, then
fstests_cephfs_autorun.sh will be executed on boot.
fstests_cephfs_autorun.sh performs the following:
- mounts configfs and debugfs
- mounts the vstart.sh provisioned CephFS filesystem under /mnt/test
- generates an xfstests configuration under /fstests/xfstests/configs

Once booted, the CephFS filesystem mount can be used for I/O. xfstests
can be invoked via:
dracut:/fstests/xfstests# ./check -ceph generic/001

When finished, the filesystem can be unmounted via:
dracut:/# umount /mnt/test


Conclusion
==========
A mock Ceph cluster can be deployed from source in a matter of seconds
using the vstart.sh utility.
Likewise, a kernel can be booted directly from source alongside a
throwaway VM and connected to the mock Ceph cluster in a couple of
minutes with QEMU/KVM.

This environment is ideal for rapid development and integration testing
of Ceph user-space and kernel components, including RBD and CephFS.