Saturday, July 7, 2018

ZFS - Create Disk Pools

Once you've installed support for ZFS, you need to create a zpool on which we can store data. This tutorial will cover how to create pools with different RAID levels.
  • singular/basic (no RAID)
  • raidz
  • raidz2
  • raid 0
  • raid 10
For this tutorial, I will be using Virtualbox with fixed size disk images of to emulate physical drives attached to my server.

Update 21st October 2017

After performing some tests, I realize now that creating the pools by specifying disks like /dev/sda and /dev/sdb is somewhat dangerous. Your pool could become degraded or disappear if you plug in another drive and all the letters change. To remedy this, use paths with /dev/disk/by-id/... instead or immediately convert your pool after creation by running:
sudo zpool export [my pool name]
sudo zpool import -d [my pool name]

Also, if your pool does become degraded because of the situation described above, running those export/import commands will fix the situation, even if you have already written data since the pool became degraded.

Single Disk (No RAID)

Substitute sdb with whatever your drive is identified as.
sudo zpool create -f [new pool name] /dev/sdb

The -f prevents the error message from preventing the creation. Be careful when using this as you could overwrite existing pools/partitions.

Multiple Disk (RAID 0)

This will create a pool of storage where data is striped across all of the devices specified. Loss of any of the drives will result in losing all of your data.
sudo zpool create -f [new pool name] /dev/sdb /dev/sdc

Adding Drives

You can add drives to a pool to increase its capacity. Any new data will be dynamically striped across the pool, but existing data will not be moved in order to "balance" the pool.
sudo zpool add [existing pool name] /dev/sdd

Raid 1

To create a RAID1 pool (or mirror), simply add the command mirror when creating or adding drives. For example:
sudo zpool create  -f [new pool name] mirror /dev/sdb /dev/sdc

sudo zpool add [existing pool name] mirror /dev/sdd /dev/sde

RAID 10

Creating a RAID1 pool of two drives, and then adding another pair of mirroring drives as shown above would actually create a RAID 10 pool whereby data is striped over two mirrors. This results in better performance without sacrificing redundancy.
To create a RAID10 array in a single command, simply run:
sudo zpool create [pool name] \
mirror disk1 disk2 \
mirror disk3 disk4

The output of sudo zpool status would show:
  pool: myPool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        myPool      ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sde     ONLINE       0     0     0

RAID Z (RAID 5)

RAIDz is very similar to RAID 5 in which parity is generated and stored in order for the RAID array to be able to gracefully handle a drive failing. This means that only one drive's worth of capacity is "lost" in order to provide redundancy for the entire pool of drives. This means that you lose a lot less capacity than with using mirrors which result in you losing 50% of your storage capacity. However, parity cannot be used for scrubbing like RAID 10 can, and calculating parity for every tile transaction can slow down your IO.
To create a RAIDz pool simply run:
sudo zpool create -f [pool name] raidz /dev/sdb /dev/sdc /dev/sdd

You can't add to a RAIDz pool. If you were to run the command below:
sudo zpool add -f [pool name] /dev/sde

Then you would be creating a RAID 0 array over the raidz pool and the drive you just added. This is best shown by the ouput of zpool status:
  pool: poolz
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        poolz       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
          sde       ONLINE       0     0     0

RAIDz2

This is much like RAIDz except that two drives will store parity instead of just one. This means that your array can lose any two drives without loss of data. This means it is actually safer than RAID10 where you would lose all of your data if two drives in the same mirror died. Using RAIDz2 over RAIDz makes a lot of sense because the likelihood of losing a drive when rebuilding a degraded RAIDz array, is greatly increased. Thus losing a drive during the rebuild shouldn't be a problem.
To create a RAIDz2 pool simply run:
sudo zpool create -f [pool name] raidz2 /dev/sdb /dev/sdc /dev/sdd 
You need a minimum of 3 drives

RAIDz3

Exactly the same as RAIDz3 except a third drive holds parity and the minimum number of drives is 4. Your array can lose 3 drives without loss of data.
To create a RAIDz3 pool simply run:
sudo zpool create -f [pool name] raidz3 /dev/sdb /dev/sdc /dev/sdd /dev/sde

Mixing and Matching

You may have realized by now that you can mix and match pools together. The pools are always "striped" together creating an effective RAID 0 (although you may need to rebalance/resilver if you added to an existing pool). This means you can combine any number of mirrors and RAIDz pools to create any kind of crazy configuration of your choice. For example, you could run RAID 0 over the top of a RAIDz and mirror arrays.

Pool Balancing

Since ZFS does not have an in-built tool to re-stripe existing data when a drive has been added, I wrote a tool to re-write every file in the pool in order to achieve this. Restriping in this manner results in drives containing an equal share of data, and improves IO. Best of all, this is all done internally, so you do not need to export all of your data and then write it back again like others have suggested.
To use the tool, simply clone the repository and make sure you have PHP installed before running the tool with a path to where your files are kept.
sudo apt-get install php5-cli git -y
git clone https://github.com/programster/zfs-balancer.git
cd zfs-balancer
php main.php /path/to/zfs/pool

Destroying Pools

sudo zpool destroy [pool name]

Conclusion

Now you've setup your storage pools, you probably want to learn the various commands for interfacing with your pool and possibly take snapshots.

Wednesday, July 4, 2018

Linux Servers security recommendation

Securing Linux server is very important to protect your data, intellectual property from the hands of crackers (hackers). The system administrator is responsible for security of the Linux box. In this blog we will go through important tips for hardening a Linux server.
Note: In this blog we are targeting specifically CentOS & RHEL Linux Operating system but same concept can be applied to other Linux/Unix flavors as well.

1) Keep System updated

Always keep system updated with latest releases patches, security fixes and kernel when it’s available. Applying security patches is an important part of maintaining Linux server and Linux provides all necessary tools to keep your system updated.
# yum updates
# yum check-update

2) Physical System Security

For best  practices you must protect Linux servers physical console access. Configure the BIOS to disable booting from CD/DVD, External Devices, Floppy Drive in BIOS. Next, enable BIOSpassword & also protect GRUB with password to restrict physical access of your system.

3) Avoid Using FTP, Telnet, And Rlogin / Rsh Services

Use secure medium to transfer files like scp, sftp etc.. and delete other services like ftp, telnet etc..
# yum erase xinetd ypserv tftp-server telnet-server rsh-server

4) Delete unnecessary packages Minimize Software to Minimize Vulnerability

It is important to delete unnecessary package to minimize vulnerability.
# yum list installed
# yum list packageName
# yum remove packageName

5) Disable Unwanted Services from server

Disable all unnecessary services and daemons (services that runs in the background).
# chkconfig –list | grep ’3:on’
To disable service, enter:
# service serviceName stop
# chkconfig serviceName off

6) Check Listening Network Ports

With the help of ‘netstat‘ networking command you can view all open ports and associated programs. As I said above use ‘chkconfig‘ command to disable all unwanted network services from the system.
# netstat -tulpn
OR use the ss command as follows:
$ ss -tulpn
OR
nmap -sT -O localhost
nmap -sT -O server.example.com

7) Turn on SELinux

Security-Enhanced Linux (SELinux) is a compulsory access control security mechanism provided in the kernel. Disabling SELinux means removing security mechanism from the system.
You can view current status of SELinux mode from the command line using ‘system-config-selinux‘, ‘getenforce‘ or ‘sestatus‘ commands.
# sestatus
If it is disabled, enable SELinux using the following command.
# setenforce enforcing
It also can be managed from ‘/etc/selinux/config‘ file, where you can enable or disable it.

8) Turn Off IPv6

If you’re not using a IPv6 protocol, then you should disable it
# vi /etc/sysconfig/network
NETWORKING_IPV6=no
IPV6INIT=no


9) Enable Iptables (Firewall)

For best practices it is recommended to enable and configure server firewall to allow only specific ports that are required and block all the remaining ports.
# service iptables start

10) Keep /boot as read-only

Linux kernel and its related files are in /boot directory which is by default as read-write. Changing it to read-only reduces the risk of unauthorized modification of critical boot files. To do this, open “/etc/fstab” file.
# vi /etc/fstab
Add the following line at the bottom, save and close it.
LABEL=/boot     /boot     ext2     defaults,ro     1 2
Please note that you need to reset the change to read-write if you need to upgrade the kernel in future.

11) Ignore ICMP or Broadcast Request

Add following line in “/etc/sysctl.conf” file to ignore ping or broadcast request.
Ignore ICMP request:
net.ipv4.icmp_echo_ignore_all = 1
Ignore Broadcast request:
net.ipv4.icmp_echo_ignore_broadcasts = 1
Load new settings or changes, by running following command
#sysctl -p

12) Important file Backup

In a production system, it is necessary to take important files backup and keep them in safety vault, remote site or offsite for Disasters recovery.

13) Checking Accounts for Empty Passwords

You must make sure all accounts have strong passwords and no one has any authorized access. Empty password accounts are security risks and that can be easily hackable.
# cat /etc/shadow | awk -F: ‘($2==””){print $1}’

14) Monitor User Activities psacct or acct

If you are dealing with lots of users, then it’s important to collect the information of each user activities and processes consumed by them and analyse them at a later time or in case if any kind of performance, security issues
Monitor User Activity with psacct or acct Commands

15) Disable Ctrl+Alt+Delete in Inittab

In most Linux distributions, pressing ‘CTRL-ALT-DELETE’ will takes your system to reboot process. So, it’s not a good idea to have this option enabled at least on production servers, if someone by mistakenly does this.
# Trap CTRL-ALT-DELETE
#ca::ctrlaltdel:/sbin/shutdown -t3 -r now

16) Enforcing Stronger Passwords

A number of users use soft or weak passwords and their password might be hacked with a dictionary based or brute-force attacks. The ‘pam_cracklib‘ module is available in PAM (Pluggable Authentication Modules) module stack which will force user to set strong passwords. Open the following file with an editor.
# vi /etc/pam.d/system-auth
And add line using credit parameters as (lcredit, ucredit, dcredit and/or ocredit respectively lower-case, upper-case, digit and other)
/lib/security/$ISA/pam_cracklib.so retry=3 minlen=8 lcredit=-1 ucredit=-2 dcredit=-2 ocredit=-1

17) Remove KDE/GNOME Desktops

X Windows on server is not required. There is no reason to run X Windows on your dedicated server. You can remove or disable them to increase security of server and performance.
yum groupremove “X Window System”
On CentOS 7/RHEL 7 server use the following commands:
# yum group remove “GNOME Desktop”
# yum group remove “KDE Plasma Workspaces”
# yum group remove “Server with GUI”
# yum group remove “MATE Desktop”

18) World-Writable Files

Anyone can modify world-writable file resulting into a security issue. Use the following command to find all world writable and sticky bits set files:
find /dir -xdev -type d \( -perm -0002 -a ! -perm -1000 \) -print
You need to investigate each reported file and either set correct user and group permission or remove it.

19 No-Owner Files

Files not owned by any user or group can pose a security problem. Just find them with the following command which do not belong to a valid user and a valid group
find /dir -xdev \( -nouser -o -nogroup \) -print
You need to investigate each reported file and either assign it to an appropriate user and group or remove it.