Multiserver NFS configuration (building shared file storage) (WebMail Pro 6 PHP)

From AfterLogic Wiki

Contents 1 Introduction 2 Security Notes 3 Configuration 3.1 Directory service 3.2 Data service 3.3 Client servers 4 Startup 5 Shutdown 6 Distributed Replicated Storage 7 Automatic File Replication (Mirror) across Two Storage Servers 7.1 Server Volfile 7.2 Client Volfile 8 High-Availability Storage Cluster example With GlusterFS On Ubuntu

Introduction

Currently shared file storage is implemented utilizing Gluster FS.

Security Notes

Gluster Documentation: What's New - Known Issues

Specifically: Replicate will only self-heal if the files exist on the first subvolume. Server A-> B works, Server A <-B does not work.

Configuration

Consists from 3 main parts:

Directory service

in our environment there are 2 directory servers:
a) directory server for data nodes. Provides data nodes with configuration files
b) directory server for client nodes. Provides client nodes with configuration files.

In general, directory service in not mandatory part of the system, but it is significantly easier to maintain and distribute configuration files from one centralized point (i.e. directory service) than to maintain configuration on each client and data nodes. Especially in case of any changes in environments with numerous data and client nodes.

Data service

Consists of a set of nodes that provide disk storage for sharing. Each node of this set is running data server. Configuration parameters in our environment are fetched from Directory service. The only configurable parameter for Data Service is address of Directory service where to fetch configuration from.

Client servers

Consists of a set of nodes that provide access to shared disk storage. Each client node provides mount point through which unified disk storage is accessible seamlessly by client applications.

Startup

You may need to modify start scripts according to IP addresses and TCP ports you are really using

1. Start directory service

a) directory data server

start-dir-server-data.sh

b) directory client server

start-dir-server-client.sh

2. Start data servers on all data nodes

start-server.sh

3. Start client servers on all client nodes

start-client.sh

Shutdown

1. Stop directory service

a) directory data server

stop-dir-server-data.sh

b) directory client server

stop-dir-server-client.sh

2. Stop data servers on all data nodes

stop-server.sh

3. Stop client servers on all client nodes

stop-client.sh

Distributed Replicated Storage

# yum install rpm-build flex gcc bison yacc libibverbs-devel make

Answer 'y' when requested, - yum should install all dependeces automatically.

Then we load the fuse kernel module...

# modprobe fuse
# vi /etc/rc.d/rc.local

Add the following string to the end of file:

modprobe fuse

Then download the GlusterFS's latests sources form the official file repository.

As binaries of GlusterFS are distributed for 64-bit CentOS only, it is neccessary to download src.rpm to build missing 32-bit or correct 64-bit GlusterFS binary distribution (here and below we'll describe 32-bit installation only):

# wget -c http://download.gluster.com/pub/gluster/glusterfs/3.1/3.1.1/CentOS/glusterfs-3.1.1-1.src.rpm

Install downloaded package with rpm:

# rpm -Uvh glusterfs-3.1.1-1.src.rpm

and build it:

# cd /usr/src/redhat/SPECS/
# rpmbuild -bb glusterfs.spec

Once rpmbuild is completed, time to install GlusterFS into the system:

# cd /usr/src/redhat/RPMS/i386
# rpm -Uvh glusterfs-core-3.1.1-1.i386.rpm glusterfs-fuse-3.1.1-1.i386.rpm glusterfs-rdma-3.1.1-1.i386.rpm

Next, reconfigure and restart glusterd/glusterfs:

# mkdir -p /mnt/glusterfs
# mkdir -p /etc/glusterfs

Additional script: glusterfs_mkdirs.sh; for example - we've two nodes and wish to provide RAID1-like NAS shared between these node hosts.

Additional config files: glusterd.vol glusterfs.vol

Copy the following config-files to the system:

# cp -r gluster*.vol /etc/glusterfs/

NOTE: Make sure you use the correct server hostnames or IP addresses in the option remote-host lines in config files!

And finally, restart gluster daemon:

# /etc/init.d/glusterd restart

Now we can mount the GlusterFS filesystem to /mnt/glusterfs with one of the following command:

# glusterfs -f /etc/glusterfs/glusterfs.vol /mnt/glusterfs

or using another method:

# mount -t glusterfs /etc/glusterfs/glusterfs.vol /mnt/glusterfs

That's all. Now we can check functionality of file storage clustering:

# mount
...
glusterfs#/etc/glusterfs/glusterfs.vol on /mnt/glusterfs type fuse (rw,allow_other,default_permissions,max_read=131072)

# df -h
Filesystem            Size  Used Avail Use% Mounted on
...
glusterfs#/etc/glusterfs/glusterfs.vol
                      6.7G  1.1G  5.3G  17% /mnt/glusterfs

If there is no problems, we may configure fstab to automount glusterfs during system boot:

# vi /etc/fstab

and add the following line to the end of file:

/etc/glusterfs/glusterfs.vol  /mnt/glusterfs  glusterfs  defaults  0  0

Now system may be rebooted and must work as cluster node.

Automatic File Replication (Mirror) across Two Storage Servers

Server Volfile

# file: /etc/glusterfs/glusterfsd.vol

volume posix
  type storage/posix
  option directory /data/export
end-volume

volume locks
  type features/locks
  subvolumes posix
end-volume

volume brick
  type performance/io-threads
  option thread-count 8
  subvolumes locks
end-volume

volume server
  type protocol/server
  option transport-type tcp
  option auth.addr.brick.allow *
  subvolumes brick
end-volume

Client Volfile

# file: /etc/glusterfs/glusterfs.vol

volume remote1
  type protocol/client
  option transport-type tcp
  option remote-host storage1.example.com
  option remote-subvolume brick
end-volume

volume remote2
  type protocol/client
  option transport-type tcp
  option remote-host storage2.example.com
  option remote-subvolume brick
end-volume

volume replicate
  type cluster/replicate
  subvolumes remote1 remote2
end-volume

volume writebehind
  type performance/write-behind
  option window-size 1MB
  subvolumes replicate
end-volume

volume cache
  type performance/io-cache
  option cache-size 512MB
  subvolumes writebehind
end-volume

High-Availability Storage Cluster example With GlusterFS On Ubuntu

Page 1
Page 2

Last edit: 2011/1/28

WebMail Pro PHP documentation
Installation System Requirements Installation Instructions for Windows Installation Instructions for Linux Upgrading instructions Configuring calendar support Install on cPanel Install on Plesk Panel Install on DirectAdmin	Configuring WebMail Getting started Configuring default domain settings Managing domains and users Managing subadmins Configuring WebMail via settings.xml Advanced configuration via config.php file Configuring web domain names Use POPPASSD to let users change their passwords Configuring voice messages support Configuring reminders Use Webmail without a database AdminPanel Guide	Customization Translating to another language How to place banner ads Adding global signature for outgoing mails Modifying or removing copyright info Creating your own skin Adding toolbar buttons Adding menu item Adding language constant Modifying columns in message list How to add new tab in account settings area	Integration Creating domains, logging accounts in, deleting domains and accounts via API Embedding WebMail into your web application via IFRAME Enable LDAP authentication Use address book stored in LDAP server like Active Directory or OpenLDAP Use Sieve for filtering API reference Logging in programmatically Synchronization Outlook Sync Add-in SyncML support	Clustering Overview Multiserver NFS configuration (building shared file storage) MySQL NDB (building shared database storage) Setting MySQL for high availability Clustering PHP Using Round robin DNS (building load balancer)	See Also Changelog Importing contacts Keyboard Shortcuts License Agreement Support Credits