minutes. Once you have a base image you can then recreate a cluster
or create new virtual clusters in minutes.
-The current implementation creates virtual clusters of RHEL5/6 nodes.
+Autocluster has recently been tested to create virtual clusters of
+RHEL 6/7 nodes. Older versions were tested with RHEL 5 and some
+versions of CentOS.
CONTENTS
Before you start, make sure you have the latest version of
autocluster. To download autocluster do this:
- git clone git://git.samba.org/tridge/autocluster.git autocluster
+ git clone git://git.samba.org/autocluster.git
Or to update it, run "git pull" in the autocluster directory
Autocluster creates virtual machines that use libvirt to run under
KVM. This means that you will need to install both KVM and
- libvirt on your host machine. Expect is used by the "waitfor"
- script and should be available for installation form your
+ libvirt on your host machine. Expect is used by the waitfor()
+ function and should be available for installation from your
distribution.
For various distros:
* RHEL/CentOS
- Autocluster should work with the standard RHEL6 qemu-kvm and
- libvirt packages. However, you'll need to tell autocluster
- where the KVM executable is:
-
- KVM=/usr/libexec/qemu-kvm
+ Autocluster should work with the standard RHEL qemu-kvm and
+ libvirt packages. It will try to find the qemu-kvm binary. If
+ you've done something unusual then you'll need to set the KVM
+ configuration variable.
For RHEL5/CentOS5, useful packages for both kvm and libvirt used
to be found here:
be able to use the settings recommended above for RHEL6.
If you're still running RHEL5.4, you have lots of time, you have
- lots of disk space and you like complexity then see the sections
- below on "iSCSI shared disks" and "Raw IDE system disks".
+ lots of disk space, and you like complexity, then see the
+ sections below on "iSCSI shared disks" and "Raw IDE system
+ disks". :-)
* Fedora
b) Install guestfish or qemu-nbd and nbd-client.
+ Autocluster needs a method of updating files in the disk image for
+ each node.
+
Recent Linux distributions, including RHEL since 6.0, contain
guestfish. Guestfish (see http://libguestfs.org/ - there are
binary packages for several distros here) is a CLI for
Autocluster attempts to use the best available method (guestmount
-> guestfish -> loopback) for accessing disk image. If it chooses
- a suboptimal method, you can force the method:
+ a suboptimal method (e.g. nodes created with guestmount sometimes
+ won't boot), you can force the method:
SYSTEM_DISK_ACCESS_METHOD=guestfish
You will need to add the autocluster directory to your PATH.
- You will need to configure the right kvm networking setup. The
- files in host_setup/etc/libvirt/qemu/networks/ should help. This
- command will install the right networks for kvm:
-
- rsync -av --delete host_setup/etc/libvirt/qemu/networks/ /etc/libvirt/qemu/networks/
+ You will need to configure the right libvirt networking setup. To
+ do this, run:
- Note that you'll need to edit the installed files to reflect any
- changes to IPBASE, IPNET0, IPNET1, IPNET2 away from the defaults.
- This is also true for named.conf.local and squid.conf (see below).
+ host_setup/setup_networks.sh [ <myconfig> ]
- After this you might need to reload libvirt:
-
- /etc/init.d/libvirt reload
-
- or similar.
+ If you're using a network setup different to the default then pass
+ your autocluster configuration filename, which should set the
+ NETWORKS variable. If you're using a variety of networks for
+ different clusters then you can probably run this script multiple
+ times.
You might also need to set:
in your environment so that virsh does KVM/QEMU things by default.
- 2) You need a caching web proxy on your local network. If you don't
- have one, then install a squid proxy on your host. See
- host_setup/etc/squid/squid.conf for a sample config suitable for a
- virtual cluster. Make sure it caches large objects and has plenty
- of space. This will be needed to make downloading all the RPMs to
- each client sane
+ 2) Configure a local web/install server to provide required YUM
+ repositories
+
+ If your install server is far away then you may need a caching web
+ proxy on your local network.
+
+ If you don't have one, then you can install a squid proxy on your
+ host amd set:
+
+ WEBPROXY="http://10.0.0.1:3128/"
+
+ See host_setup/etc/squid/squid.conf for a sample config suitable
+ for a virtual cluster. Make sure it caches large objects and has
+ plenty of space. This will be needed to make downloading all the
+ RPMs to each client sane
To test your squid setup, run a command like this:
3) Setup a DNS server on your host. See host_setup/etc/bind/ for a
sample config that is suitable. It needs to redirect DNS queries
- for your virtual domain to your windows domain controller
+ for your virtual domain to your windows domain controller.
- 4) Download a RHEL install ISO.
+ 4) Download a RHEL (or CentOS) install ISO.
CREATING A CLUSTER
base disk image - without them the disk image for each cluster node
would need to contain the entire RHEL install.
-The cluster creation process can be broken down into 2 mains steps:
+The cluster creation process can be broken down into several main
+steps:
+
+ 1) Create a base disk image.
- 1) Creating the base disk image.
+ 2) Create per-node disk images and corresponding XML files.
- 2) Create the per-node disk images and corresponding XML files.
+ 3) Update /etc/hosts to include cluster nodes.
+
+ 4) Boot virtual machines for the nodes.
+
+ 5) Post-boot configuration.
However, before you do this you will need to create a configuration
file. See the "CONFIGURATION" section below for more details.
1) Create the base disk image using:
- ./autocluster create base
+ ./autocluster base create
The first thing this step does is to check that it can connect to
the YUM server. If this fails make sure that there are no
The installation process uses kickstart. The choice of
postinstall script is set using the POSTINSTALL_TEMPLATE variable.
- An example is provided in
- base/all/root/scripts/gpfs-nas-postinstall.sh.
-
- It makes sense to install packages that will be common to all
+ This can be used to install packages that will be common to all
nodes into the base image. This save time later when you're
- setting up the cluster nodes. However, you don't have to do this
- - you can set POSTINSTALL_TEMPLATE to "" instead - but then you
- will lose the quick cluster creation/setup that is a major feature
- of autocluster.
+ setting up the cluster nodes. However, current usage (given that
+ we test many versions of CTDB) is to default POSTINSTALL_TEMPLATE
+ to "" and install packages post-boot. This seems to be a
+ reasonable compromise between flexibility (the base image can be,
+ for example, a pristine RHEL7.0-base.qcow2, CTDB/Samba packages
+ are selected post-base creation) and speed of cluster creation.
When that has finished you should mark that base image immutable
like this:
image will be used as a basis file for the per-node images, and if
it changes your cluster will become corrupt
- 2) Now run "autocluster create cluster" specifying a cluster
- name. For example:
+ 2-5)
+ Now run "autocluster cluster build", specifying a configuration
+ file. For example:
- autocluster create cluster c1
+ autocluster -c m1.autocluster cluster build
This will create and install the XML node descriptions and the
disk images for your cluster nodes, and any other nodes you have
images are then attached to using guestfish or
loopback-nbd-mounted, and populated with system configuration
files and other potentially useful things (such as scripts).
+ /etc/hosts is updated, the cluster is booted and post-boot
+ setup is done.
+ Instead of doing all of the steps 2-5 using 1 command you call do:
-BOOTING A CLUSTER
-=================
+ 2) autocluster -c m1.autocluster cluster create
+
+ 3) autocluster -c m1.autocluster cluster update_hosts
+
+ 4) autocluster -c m1.autocluster cluster boot
+
+ 5) autocluster -c m1.autocluster cluster configure
+
+BOOTING/DESTROY A CLUSTER
+=========================
-At this point the cluster has been created but isn't yet running.
Autocluster provides a command called "vircmd", which is a thin
wrapper around libvirt's virsh command. vircmd takes a cluster name
instead of a node/domain name and runs the requested command on all
nodes in the cluster.
- 1) Now boot your cluster nodes like this:
-
- vircmd start c1
-
The most useful vircmd commands are:
start : boot a node
shutdown : graceful shutdown of a node
destroy : power off a node immediately
- 2) You can watch boot progress like this:
+ You can watch boot progress like this:
tail -f /var/log/kvm/serial.c1*
kernel panic messages and watch the nodes via ssh
-POST-CREATION SETUP
-===================
-
-Now you have a cluster of nodes, which might have a variety of
-packages installed and configured in a common way. Now that the
-cluster is up and running you might need to configure specialised
-subsystems like GPFS or Samba. You can do this by hand or use the
-sample scripts/configurations that are provided.
-
-Now you can ssh into your nodes. You may like to look at the small set
-of scripts in /root/scripts on the nodes for some scripts. In
-particular:
-
- mknsd.sh : sets up the local shared disks as GPFS NSDs
- setup_gpfs.sh : sets up GPFS, creates a filesystem etc
- setup_cluster.sh : sets up clustered Samba and other NAS services
- setup_tsm_server.sh: run this on the TSM node to setup the TSM server
- setup_tsm_client.sh: run this on the GPFS nodes to setup HSM
- setup_ad_server.sh : run this on a node to setup a Samba4 AD
+POST-BOOT SETUP
+===============
+Autocluster copies some scripts to cluster nodes to enable post-boot
+configuration. These are used to configure specialised subsystems
+like GPFS or Samba and are installed in /root/scripts/ on each node.
+The main 2 entry points are install_packages.sh and setup_cluster.sh.
To setup a clustered NAS system you will normally need to run
-setup_gpfs.sh and setup_cluster.sh on one of the nodes.
-
-
-AUTOMATED CLUSTER CREATION
-==========================
-
-The last 2 steps can be automated. An example script for doing this
-can be found in examples/create_cluster.sh.
+setup_gpfs.sh and setup_cluster.sh on one of the nodes. If you want
+to run these manually, see autocluster's cluster_configure() function
+for example usage.
+There are also some older scripts that haven't been used for a while
+and have probably bit-rotted, such as setup_tsm_client.sh and
+setup_tsm_server.sh. However, they are still provided as examples.
CONFIGURATION
=============
* The NODES configuration variable controls the types of nodes that
are created. At the time of writing, the default value is:
- NODES="sofs_front:0-3 rhel_base:4"
+ NODES="nas:0-3 rhel_base:4"
This means that you get 4 clustered NAS nodes, at IP offsets 0, 1,
2, & 3 from FIRSTIP, all part of the CTDB cluster. You also get an
additional utility node at IP offset 4 that can be used, for
- example, as a test client. Since sofs_* nodes are present, the base
- node will not be part of the CTDB cluster - it is just extra.
+ example, as a test client. The base node will not be part of the
+ CTDB cluster. It is just extra node that can be used as a test
+ client or similar.
For many standard use cases the nodes specified by NODES can be
- modified by setting NUMNODES, WITH_SOFS_GUI and WITH_TSM_NODE.
+ modified by setting NUMNODES and WITH_TSM_NODE.
However, these options can't be used to create nodes without
specifying IP offsets - except WITH_TSM_NODE, which checks to see if
IP offset 0 is vacant. Therefore, for many uses you can ignore the
git.samba.org / autocluster.git / blobdiff