Hp StorageWorks Scalable File Share User Manual

HP Scalable File Share User GuideG3.2-0HP Part Number: SFSUGG32-EPublished: May 2010Edition: 5

10

About This DocumentThis document provides installation and configuration information for HP Scalable File Share(SFS) G3.2-0. Overviews of installing a

WARNING A warning calls attention to important information that if notunderstood or followed will result in personal injury ornonrecoverable system pr

For documentation of previous versions of HP SFS, see:• HP StorageWorks Scalable File Share Client Installation and User Guide Version 2.2 at:http://d

14

1 What's In This Version1.1 About This ProductHP SFS G3.2-0 uses the Lustre File System on MSA hardware to provide a storage system forstandalone

1 CentOS 5.3 is available for download from the HP Software Depot at:http://www.hp.com/go/softwaredepot1.3.1 Hardware ConfigurationA typical HP SFS sy

Figure 1-1 Platform Overview1.3 Supported Configurations 17

Figure 1-2 Server PairsFigure 1-2 shows typical wiring for server pairs.IMPORTANT: If you are using MSA2000fc G1 (MSA2212fc), see (page 66) for import

minimum memory for OSS servers according to the following guidelines, based on the numberof OSTs connected to the OSS server pair, at a rate of 2 GB p

© Copyright 2010 Hewlett-Packard Development Company, L.P.Confidential computer software. Valid license from HP required for possession, use or copyin

standard methods available for CentOS. CentOS security updates can be monitored by subscribingto the CentOS Announce mailing list.1.5 Release Notes1.5

2 Installing and Configuring MSA ArraysThis chapter summarizes the installation and configuration steps for MSA2000fc arrays use inHP SFS G3.2-0 syste

IMPORTANT: The size of a Lustre MDT or OST is limited to 8 TB. Therefore, any volume createdon the MSA2000 must be less than or equal to 8796 GB. If a

# forallmsas show disks ; doneThe CLI syntax for specifying disks in enclosures differs based on the controller type usedin the array. The following v

correct assignment of multipath priorities. HP recommends mapping all ports to each volumeto facilitate proper hardware failover.a. Create vdisks in t

1. Enable FTP on the MSA with the CLI command:# set protocols ftp enable2. Use FTP from a Linux host to upload log files:# ftp MSAIPaddress3. Log in w

1. Install the sendmail-cf RPM from your operating system distribution media, if it is notalready installed.2. If you are running with a firewall, the

3 Installing and Configuring HP SFS Software on ServerNodesThis chapter provides information about installing and configuring HP SFS G3.2-0 software o

3.1 Supported FirmwareFollow the instructions in the documentation which was included with each hardware componentto ensure that you are running the l

3.2 Installation RequirementsA set of HP SFS G3.2-0 file system server nodes should be installed and connected by HP inaccordance with the HP SFS G3.2

Table of ContentsAbout This Document...11Intended

The following optional, but recommended, line sets up an Ethernet network interface. More thanone Ethernet interface may be set up using additional ne

During the Kickstart post-installation phase, you are prompted to install the HP SFS G3.2-0 DVDinto the DVD drive:Please insert the HP SFS G3.2-0 DVD

NOTE: USB drives are not scanned before the installer reads the Kickstart file, so you areprompted with a message indicating that the Kickstart file c

NOTE: The output from Installation Phase 1 is contained in /var/log/postinstall.log.Proceed to “Installation Phase 2”.3.4 Installation Phase 2After th

IMPORTANT: This step must be performed for 10 GigE systems only. Do not use this processon InfiniBand systems.If your system uses Mellanox ConnectX HC

3.5.2 Creating the /etc/hosts fileCreate an /etc/hosts file with the names and IP addresses of all the Ethernet interfaces oneach system in the file s

enabling direct user login access to the file system server nodes. In particular, the shadowpassword information should not be provided through NIS or

This import command should be performed by root on each system that installs signed RPMpackages.3.6 Upgrade InstallationIn some situations you may upg

1. For the first member of the failover pair, stop the Heartbeat service to migrate the Lustrefile system components from this node to its failover pa

6. For the upgrade from SFS G3.0-0 to G3.1-0 or SFS G3.2-0, you must re-create the Heartbeatconfiguration files to account for licensing. For the deta

3.5.1 Configuring Ethernet and InfiniBand or 10 GigE Interfaces...343.5.2 Creating the /etc/hosts file.

40

4 Installing and Configuring HP SFS Software on ClientNodesThis chapter provides information about installing and configuring HP SFS G3.2-0 software o

If the client is using the HP recommended 10 GigE ConnectX cards from Mellanox, the ConnectXEN drivers must be installed. These drivers can be downloa

NOTE: The network addresses shown above are the InfiniBand IPoIB ib0 interfaces for theHP SFS G3.2-0 Management Server (MGS) node, and the MGS failove

4. Run the following command:# make rpms 2>&1 | tee make.log5. When successfully completed, the newly built RPMs are available in/usr/src/redha

5 Using HP SFS SoftwareThis chapter provides information about creating, configuring, and using the file system.5.1 Creating a Lustre File SystemThe f

To see the multipath configuration, use the following command. Output will be similar to theexample shown below:# multipath -llmpath7 (3600c0ff000d547

node3,options lnet networks=o2ib0,/dev/mapper/mpath6,/mnt/ost4,ost,testfs,icnode1@o2ib0:icnode2@o2ib0 ,,,,"_netdev,noauto",icnode4@o2ib0

as needed on the file system server nodes. The lustre_config command can take hoursto complete depending on the size of the disks.2. Start the file sy

heartbeat-2.1.3-1.01hp2. Obtain the failover pair information from the overall Lustre configuration.3. Heartbeat uses one or more of the network inter

7 Known Issues and Workarounds...637.1 Server Reboot...

The haresources files for both members of a failover pair (Heartbeat cluster) must be identical.The ha.cf files should be identical.You can generate t

haresources2cib.py reads the ha.cf and haresources files from /etc/ha.d and writesthe output to /var/lib/heartbeat/crm/cib.xml.The haresources2cib.py

NOTE: Passwordless ssh must be set up on the HP SFS servers before using this -c option.5.2.5 Things to Double-CheckEnsure that the following conditio

mcast eth0 239.0.1.1 694 1 0NOTE: Changing the authentication string in /etc/ha.d/authkeys causes Heartbeat toreport numerous warnings instead of erro

1. Stop the Heartbeat service on all the OSS nodes:# pdsh -w oss[1-n] service heartbeat stop2. Stop the Heartbeat service on the MDS and MGS nodes:# p

If you cannot start a resource on a node, check that node for values of -INFINITY in/var/lib/heartbeat/crm/cib.xml. There should be none. For more det

10 UP osc hpcsfsc-OST000b-osc hpcsfsc-mdtlov_UUID 5 11 UP osc hpcsfsc-OST000a-osc hpcsfsc-mdtlov_UUID 5 12 UP osc hpcsfsc-OST0005-osc hpcsfsc-mdtlov_U

To see if the problem can be fixed with writeconf, run the following test:1. On the MGS node run:[root@adm ~]# debugfs -c -R 'dump CONFIGS/testfs

10. From one client node, mount the Lustre file system. The mount initiates a file system recovery.If the file system has a large amount of data, the

The following commands show the file system component connections and the network interfacesthat serve them.# ls /proc/fs/lustre/*/*/*conn_uuid/proc/f

6

60

6 LicensingA valid license is required for normal operation of HP SFS G3.2-0. HP SFS G3.2-0 systems arepreconfigured with the correct license file at

[root@atlas1] grep "SFS License" /var/log/messagesFeb 9 17:04:08 atlas1 SfsLicenseAgent: Error: No SFS License file found. Check /var/flexlm

7 Known Issues and WorkaroundsThe following items are known issues and workarounds.7.1 Server RebootAfter the server reboots, it checks the file syste

NOTE: Use the appropriate device in place of /dev/mapper/mpath?b. For example, if the --dryrun command returned:Parameters: mgsnode=172.31.80.1@o2ib m

7.9 Misconfigured Lustre target config logs due to incorrect CSV file usedduring lustre_configThis problem has been identified with HP SFS G3.0 and sy

to determine how to correct the configuration log information. If assistance is needed, contactHP support.7.10 MSA2000fc G1 incorrect MSA cabling betw

A HP SFS G3 PerformanceA.1 Benchmark PlatformHP SFS G3, based on Lustre File System Software, is designed to provide the performance andscalability ne

Figure A-2 shows more detail about the storage configuration. The storage comprised a numberof HP MSA2212fc arrays. Each array had a redundant pair of

Figure A-3 Single Stream ThroughputFor a file written on a single OST (a single RAID volume), throughput is in the neighborhood of200 MB/s. As the str

List of Figures1-1 Platform Overview...

filled with the new data. At the point (14:10:14 in the graph) where the amount of data reachedthe cache limit imposed by Lustre (12 GB), throughput d

Figure A-6 Multi-Client Throughput ScalingIn general, Lustre scales quite well with additional OSS servers if the workload is evenlydistributed over t

A.4 One Shared FileFrequently in HPC clusters, a number of clients share one file either for read or for write. Forexample, each of N clients could wr

Another way to measure throughput is to only average over the time while all the clients areactive. This is represented by the taller, narrower box in

For workloads that require a lot of disk head movement relative to the amount of data moved,SAS disk drives provide a significant performance benefit.

Each disk shelf in the platform used for deep shelf testing was configured in the same manneras described in “Benchmark Platform” (page 67). The arran

Figure A-12 Client Count Versus Total Throughput (MB/s)A.7.3 Throughput ScalingA single file accessed by eight clients benefits from increased stripin

The OSTs were populated with 146 GB SAS drives. Stripe placement was controlled by defaultoperation of the HP SFS file system software. Specific contr

network buffering parameters were set as described in the documentation for the configurednetwork controller.A.8.2 Single Stream ThroughputThroughput

Figure A-16 Client Count Versus Total Throughput (MB/s)A.8.3 Throughput ScalingAs in “Throughput Scaling” (page 70), a set of 16 clients wrote or read

8

80

IndexSymbols/etc/hosts fileconfiguring, 3510 GigEclients, 41configuring, 34installation, 33performance, 76Bbenchmark platform, 67Ccache limit, 70cib.x

release notes, 20rolling upgrades, 37Sscaling, 70server security policy, 19shared files, 72stonewalling, 72stonith, 48support, 12Tthroughput scaling,

83

List of Tables1-1 Supported Configurations ...........................