minutes. Once you have a base image you can then recreate a cluster
or create new virtual clusters in minutes.
-The current implementation creates virtual clusters of RHEL5 nodes.
+Autocluster has recently been tested to create virtual clusters of
+RHEL 6/7 nodes. Older versions were tested with RHEL 5 and some
+versions of CentOS.
CONTENTS
Before you start, make sure you have the latest version of
autocluster. To download autocluster do this:
- git clone git://git.samba.org/tridge/autocluster.git autocluster
+ git clone git://git.samba.org/autocluster.git
Or to update it, run "git pull" in the autocluster directory
clusters generated by autocluster.
- 1) Install kvm, libvirt, qemu-nbd, nbd-client and expect.
+ 1) Install and configure required software.
+
+ a) Install kvm, libvirt and expect.
Autocluster creates virtual machines that use libvirt to run under
KVM. This means that you will need to install both KVM and
- libvirt on your host machine. You will also need the qemu-nbd and
- nbd-client programs, which autocluster uses to loopback-nbd-mount
- the disk images when configuring each node. Expect is used by the
- "waitfor" script and should be available for installation form
- your distribution.
+ libvirt on your host machine. Expect is used by the waitfor()
+ function and should be available for installation from your
+ distribution.
For various distros:
* RHEL/CentOS
+ Autocluster should work with the standard RHEL qemu-kvm and
+ libvirt packages. It will try to find the qemu-kvm binary. If
+ you've done something unusual then you'll need to set the KVM
+ configuration variable.
+
For RHEL5/CentOS5, useful packages for both kvm and libvirt used
to be found here:
However, since recent versions of RHEL5 ship with KVM, 3rd party
KVM RPMs for RHEL5 are now scarce.
- RHEL5.4 ships with KVM but it doesn't have the SCSI disk
- emulation that autocluster uses by default. There are also
- problems when autocluster uses virtio on RHEL5.4's KVM. See the
- sections below on "iSCSI shared disks" and "Raw IDE system
- disks". Also, to use the RHEL5 version of KVM you will need to
- set
-
- KVM=/usr/libexec/qemu-kvm
-
- in your configuration file.
+ RHEL5.4's KVM also has problems when autocluster uses virtio
+ shared disks, since multipath doesn't notice virtio disks. This
+ is fixed in RHEL5.6 and in a recent RHEL5.5 update - you should
+ be able to use the settings recommended above for RHEL6.
- Unless you can find an RPM for nbd-client then you need to
- download source from:
-
- http://sourceforge.net/projects/nbd/
-
- and build it.
+ If you're still running RHEL5.4, you have lots of time, you have
+ lots of disk space, and you like complexity, then see the
+ sections below on "iSCSI shared disks" and "Raw IDE system
+ disks". :-)
- * Fedora Core
+ * Fedora
Useful packages ship with Fedora Core 10 (Cambridge) and later.
-
- qemu-nbd is in the kvm package.
-
- nbd-client is in the nbd package.
+ Some of the above notes on RHEL might apply to Fedora's KVM.
* Ubuntu
Useful packages ship with Ubuntu 8.10 (Intrepid Ibex) and later.
-
- qemu-nbd is in the kvm package but is called kvm-nbd, so you
- need to set the QEMU_NBD configuration variable.
-
- nbd-client is in the nbd-client package.
+ In recent Ubuntu versions (e.g. 10.10 Maverick Meerkat) the KVM
+ package is called "qemu-kvm". Older versions have a package
+ called "kvm".
For other distributions you'll have to backport distro sources or
compile from upstream source as described below.
http://libvirt.org/
- * As mentioned about, nbd can be found at:
+ b) Install guestfish or qemu-nbd and nbd-client.
+
+ Autocluster needs a method of updating files in the disk image for
+ each node.
+
+ Recent Linux distributions, including RHEL since 6.0, contain
+ guestfish. Guestfish (see http://libguestfs.org/ - there are
+ binary packages for several distros here) is a CLI for
+ manipulating KVM/QEMU disk images. Autocluster supports
+ guestfish, so if guestfish is available then you should use it.
+ It should be more reliable than NBD.
+ Autocluster attempts to use the best available method (guestmount
+ -> guestfish -> loopback) for accessing disk image. If it chooses
+ a suboptimal method (e.g. nodes created with guestmount sometimes
+ won't boot), you can force the method:
+
+ SYSTEM_DISK_ACCESS_METHOD=guestfish
+
+ If you can't use guestfish then you'll have to use NBD. For this
+ you will need the qemu-nbd and nbd-client programs, which
+ autocluster uses to loopback-nbd-mount the disk images when
+ configuring each node.
+
+ NBD for various distros:
+
+ * RHEL/CentOS
+
+ qemu-nbd is only available in the old packages from lfarkas.org.
+ Recompiling the RHEL5 kvm package to support NBD is quite
+ straightforward. RHEL6 doesn't have an NBD kernel module, so is
+ harder to retrofit for NBD support - use guestfish instead.
+
+ Unless you can find an RPM for nbd-client then you need to
+ download source from:
+
http://sourceforge.net/projects/nbd/
- You will need to add the autocluster directory to your PATH.
+ and build it.
+
+ * Fedora Core
+
+ qemu-nbd is in the qemu-kvm or kvm package.
+
+ nbd-client is in the nbd package.
+
+ * Ubuntu
+
+ qemu-nbd is in the qemu-kvm or kvm package. In older releases
+ it is called kvm-nbd, so you need to set the QEMU_NBD
+ configuration variable.
+
+ nbd-client is in the nbd-client package.
+
+ * As mentioned above, nbd can be found at:
- You will need to configure the right kvm networking setup. The
- files in host_setup/etc/libvirt/qemu/networks/ should help. This
- command will install the right networks for kvm:
+ http://sourceforge.net/projects/nbd/
- rsync -av --delete host_setup/etc/libvirt/qemu/networks/ /etc/libvirt/qemu/networks/
+ c) Environment and libvirt virtual networks
- Note that you'll need to edit the installed files to reflect any
- changes to IPBASE, IPNET0, IPNET1, IPNET2 away from the defaults.
- This is also true for named.conf.local and squid.conf (see below).
+ You will need to add the autocluster directory to your PATH.
- After this you might need to reload libvirt:
+ You will need to configure the right libvirt networking setup. To
+ do this, run:
- /etc/init.d/libvirt reload
+ host_setup/setup_networks.sh [ <myconfig> ]
- or similar.
+ If you're using a network setup different to the default then pass
+ your autocluster configuration filename, which should set the
+ NETWORKS variable. If you're using a variety of networks for
+ different clusters then you can probably run this script multiple
+ times.
You might also need to set:
in your environment so that virsh does KVM/QEMU things by default.
- 2) You need a caching web proxy on your local network. If you don't
- have one, then install a squid proxy on your host. See
- host_setup/etc/squid/squid.conf for a sample config suitable for a
- virtual cluster. Make sure it caches large objects and has plenty
- of space. This will be needed to make downloading all the RPMs to
- each client sane
+ 2) Configure a local web/install server to provide required YUM
+ repositories
+
+ If your install server is far away then you may need a caching web
+ proxy on your local network.
+
+ If you don't have one, then you can install a squid proxy on your
+ host amd set:
+
+ WEBPROXY="http://10.0.0.1:3128/"
+
+ See host_setup/etc/squid/squid.conf for a sample config suitable
+ for a virtual cluster. Make sure it caches large objects and has
+ plenty of space. This will be needed to make downloading all the
+ RPMs to each client sane
To test your squid setup, run a command like this:
3) Setup a DNS server on your host. See host_setup/etc/bind/ for a
sample config that is suitable. It needs to redirect DNS queries
- for your virtual domain to your windows domain controller
+ for your virtual domain to your windows domain controller.
- 4) Download a RHEL install ISO.
+ 4) Download a RHEL (or CentOS) install ISO.
CREATING A CLUSTER
base disk image - without them the disk image for each cluster node
would need to contain the entire RHEL install.
-The cluster creation process can be broken down into 2 mains steps:
+The cluster creation process can be broken down into several main
+steps:
+
+ 1) Create a base disk image.
+
+ 2) Create per-node disk images and corresponding XML files.
+
+ 3) Update /etc/hosts to include cluster nodes.
- 1) Creating the base disk image.
+ 4) Boot virtual machines for the nodes.
- 2) Create the per-node disk images and corresponding XML files.
+ 5) Post-boot configuration.
However, before you do this you will need to create a configuration
file. See the "CONFIGURATION" section below for more details.
1) Create the base disk image using:
- ./autocluster create base
+ ./autocluster base create
The first thing this step does is to check that it can connect to
the YUM server. If this fails make sure that there are no
The install will take about 10 to 15 minutes and you will see the
packages installing in your terminal
- The installation process uses kickstart. If your configuration
- uses a SoFS release then the last stage of the kickstart
- configuration will be a postinstall script that installs and
- configures packages related to SoFS. The choice of postinstall
- script is set using the POSTINSTALL_TEMPLATE variable, allowing you
- to adapt the installation process for different types of clusters.
-
- It makes sense to install packages that will be common to all
+ The installation process uses kickstart. The choice of
+ postinstall script is set using the POSTINSTALL_TEMPLATE variable.
+ This can be used to install packages that will be common to all
nodes into the base image. This save time later when you're
- setting up the cluster nodes. However, you don't have to do this
- - you can set POSTINSTALL_TEMPLATE to "" instead - but then you
- will lose the quick cluster creation/setup that is a major feature
- of autocluster.
+ setting up the cluster nodes. However, current usage (given that
+ we test many versions of CTDB) is to default POSTINSTALL_TEMPLATE
+ to "" and install packages post-boot. This seems to be a
+ reasonable compromise between flexibility (the base image can be,
+ for example, a pristine RHEL7.0-base.qcow2, CTDB/Samba packages
+ are selected post-base creation) and speed of cluster creation.
When that has finished you should mark that base image immutable
like this:
image will be used as a basis file for the per-node images, and if
it changes your cluster will become corrupt
- 2) Now run "autocluster create cluster" specifying a cluster
- name. For example:
+ 2-5)
+ Now run "autocluster cluster build", specifying a configuration
+ file. For example:
- autocluster create cluster c1
+ autocluster -c m1.autocluster cluster build
This will create and install the XML node descriptions and the
disk images for your cluster nodes, and any other nodes you have
configured. Each disk image is initially created as an "empty"
copy-on-write image, which is linked to the base image. Those
- images are then loopback-nbd-mounted and populated with system
- configuration files and other potentially useful things (such as
- scripts).
+ images are then attached to using guestfish or
+ loopback-nbd-mounted, and populated with system configuration
+ files and other potentially useful things (such as scripts).
+ /etc/hosts is updated, the cluster is booted and post-boot
+ setup is done.
+ Instead of doing all of the steps 2-5 using 1 command you call do:
-BOOTING A CLUSTER
-=================
+ 2) autocluster -c m1.autocluster cluster create
+
+ 3) autocluster -c m1.autocluster cluster update_hosts
+
+ 4) autocluster -c m1.autocluster cluster boot
+
+ 5) autocluster -c m1.autocluster cluster setup
+
+BOOTING/DESTROY A CLUSTER
+=========================
-At this point the cluster has been created but isn't yet running.
Autocluster provides a command called "vircmd", which is a thin
wrapper around libvirt's virsh command. vircmd takes a cluster name
instead of a node/domain name and runs the requested command on all
nodes in the cluster.
- 1) Now boot your cluster nodes like this:
-
- vircmd start c1
-
The most useful vircmd commands are:
- start : boot a node
- shutdown : graceful shutdown of a node
- destroy : power off a node immediately
+ start : boot a cluster
+ shutdown : graceful shutdown of a cluster
+ destroy : power off a cluster immediately
- 2) You can watch boot progress like this:
+ You can watch boot progress like this:
tail -f /var/log/kvm/serial.c1*
kernel panic messages and watch the nodes via ssh
-POST-CREATION SETUP
-===================
-
-Now you have a cluster of nodes, which might have a variety of
-packages installed and configured in a common way. Now that the
-cluster is up and running you might need to configure specialised
-subsystems like GPFS or Samba. You can do this by hand or use the
-sample scripts/configurations that are provided
-
- 1) Now you can ssh into your nodes. You may like to look at the
- small set of scripts in /root/scripts on the nodes for
- some scripts. In particular:
+POST-BOOT SETUP
+===============
- mknsd.sh : sets up the local shared disks as GPFS NSDs
- setup_gpfs.sh : sets up GPFS, creates a filesystem etc
- setup_samba.sh : sets up Samba and many other system compoents
- setup_tsm_server.sh: run this on the TSM node to setup the TSM server
- setup_tsm_client.sh: run this on the GPFS nodes to setup HSM
+Autocluster copies some scripts to cluster nodes to enable post-boot
+configuration. These are used to configure specialised subsystems
+like GPFS or Samba, and are installed in /root/scripts/ on each node.
+The main entry point is cluster_setup.sh, which invokes specialised
+scripts depending on the cluster filesystem type or the node type.
+cluster_setup.sh is invoked by the cluster_setup() function in
+autocluster.
- To setup a SoFS system you will normally need to run
- setup_gpfs.sh and setup_samba.sh.
-
- 2) If using the SoFS GUI, then you may want to lower the memory it
- uses so that it fits easily on the first node. Just edit this
- file on the first node:
-
- /opt/IBM/sofs/conf/overrides/sofs.javaopt
-
- 3) For automating the SoFS GUI, you may wish to install the iMacros
- extension to firefox, and look at some sample macros I have put
- in the imacros/ directory of autocluster. They will need editing
- for your environment, but they should give you some hints on how
- to automate the final GUI stage of the installation of a SoFS
- cluster.
+See cluster_setup() if you want to do things manually or if you want
+to add support for other node types and/or cluster filesystems.
+There are also some older scripts that haven't been used for a while
+and have probably bit-rotted, such as setup_tsm_client.sh and
+setup_tsm_server.sh. However, they are still provided as examples.
CONFIGURATION
=============
and move on from there.
-* Use the --with-release option on the command-line or the
- with_release function in a configuration file to get default values
- for building virtual clusters for releases of particular "products".
- Currently there are only release definitions for SoFS.
-
- For example, you can setup default values for SoFS-1.5.3 by running:
-
- autocluster --with-release=SoFS-1.5.3 ...
-
- Equivalently you can use the following syntax in a configuration
- file:
-
- with_release "SoFS-1.5.3"
-
- So the smallest possible config file would have something like this
- as the first line and would then set FIRSTIP:
-
- with_release "SoFS-1.5.3"
-
- FIRSTIP=<whatever>
-
- Add other options as you need them.
-
- The release definitions are stored in releases/*.release. The
- available releases are listed in the output of "autocluster --help".
-
- NOTE: Occasionally you will need to consider the position of
- with_release in your configuration. If you want to override options
- handled by a release definition then you will obviously need to set
- them later in your configuration. This will be the case for most
- options you will want to set. However, some options will need to
- appear before with_release so that they can be used within a release
- definition - the most obvious one is the (rarely used) RHEL_ARCH
- option, which is used in the default ISO setting for each release.
- If things don't work as expected use --dump to confirm that
- configuration variables have the values that you expect.
-
* The NODES configuration variable controls the types of nodes that
are created. At the time of writing, the default value is:
- NODES="rhel_base:0-3"
-
- This means that you get 4 nodes, at IP offsets 0, 1, 2, & 3 from
- FIRSTIP, all part of the CTDB cluster. That is, with standard
- settings and FIRSTIP=35, 4 nodes will be created in the IP range
- 10.0.0.35 to 10.0.0.38.
+ NODES="nas:0-3 rhel_base:4"
- The SoFS releases use a default of:
+ This means that you get 4 clustered NAS nodes, at IP offsets 0, 1,
+ 2, & 3 from FIRSTIP, all part of the CTDB cluster. You also get an
+ additional utility node at IP offset 4 that can be used, for
+ example, as a test client. The base node will not be part of the
+ CTDB cluster. It is just extra node that can be used as a test
+ client or similar.
- NODES="tsm_server:0 sofs_gui:1 sofs_front:2-4"
-
- which should produce a set of nodes the same as the old SoFS
- default. You can add extra rhel_base nodes if you need them for
- test clients or some other purpose:
-
- NODES="$NODES rhel_base:7,8"
+Corrupt system disks
+====================
- This produces an additional 2 base RHEL nodes at IP offsets 7 & 8
- from FIRSTIP. Since sofs_* nodes are present, these base nodes will
- not be part of the CTDB cluster - they're just extra.
+Recent versions of KVM seem to have fixed problems where the
+combination of qcow2 file format, virtio block devices and writeback
+caching would cause result in corrupt. This means the default system
+disk bus type (a.k.a. SYSTEM_DISK_TYPE) is now virtio.
- For many standard use cases the nodes specified by NODES can be
- modified by setting NUMNODES, WITH_SOFS_GUI and WITH_TSM_NODE.
- However, these options can't be used to create nodes without
- specifying IP offsets - except WITH_TSM_NODE, which checks to see if
- IP offset 0 is vacant. Therefore, for many uses you can ignore the
- NODES variable.
+If using an older version of KVM or if you experience corruption of
+the system disk, try using IDE system disks:
- However, NODES is the recommended mechanism for specifying the nodes
- that you want in your cluster. It is powerful, easy to read and
- centralises the information in a single line of your configuration
- file.
+ SYSTEM_DISK_TYPE=ide
iSCSI shared disks
==================
The RHEL5 version of KVM does not support the SCSI block device
emulation. Therefore, you can use either virtio or iSCSI shared
-disks. Unfortunately, at the time of writing, virtio block devices
-are not supported by the version of multipath in RHEL5. So this
-leaves iSCSI as the only choice.
+disks. Unfortunately, in RHEL5.4 and early versions of RHEL5.5,
+virtio block devices are not supported by the version of multipath in
+RHEL5. So this leaves iSCSI as the only choice.
The main configuration options you need for iSCSI disks are:
Raw IDE system disks
====================
-The RHEL5 version of KVM does not support the SCSI block device
-emulation. Therefore, you can use virtio or ide system disks.
-However, writeback caching, qcow2 and virtio are incompatible and
-result in I/O corruption. So, you can use either virtio system disks
-without any caching, accepting reduced performance, or you can use IDE
-system disks with writeback caching, with nice performance.
+Older RHEL versions of KVM did not support the SCSI block device
+emulation, and produced corruption when virtio disks were used with
+qcow2 disk images and writeback caching. In this case, you can use
+either virtio system disks without any caching, accepting reduced
+performance, or you can use IDE system disks with writeback caching,
+with nice performance.
For IDE disks, here are the required settings:
One good use of this option is to test template substitution using the
function substitute_vars(). For example:
- ./autocluster --with-release=SoFS-1.5.3 -e 'CLUSTER=foo; DISK=foo.qcow2; UUID=abcdef; NAME=foon1; set_macaddrs; substitute_vars templates/node.xml'
+ ./autocluster -c example.autocluster -e 'CLUSTER=foo; DISK=foo.qcow2; UUID=abcdef; NAME=foon1; set_macaddrs; substitute_vars templates/node.xml'
This prints templates/node.xml with all appropriate substitutions
done. Some internal variables (e.g. CLUSTER, DISK, UUID, NAME) are
git.samba.org / autocluster.git / blobdiff