Saturday, February 6, 2016

LVM Migration using mirroring and pvmove method

What is LVM Migration?

LVM migration is one of the excellent feature, where we can migrate the logical volumes to a new disk without the data-loss and downtime. The purpose of this feature is it to move our data from old disk to a new disk. Usually, we do migrations from one disk to other disk storage, only when an error occur in some disks.

Features of Migration

Moving logical volumes from one disk to other disk.
We can use any type of disk like SATA, SSD, SAS, SAN storage iSCSI or FC.
Migrate disks without data loss and downtime.

In LVM Migration, we will swap every volumes, file-system and it’s data in the existing storage. For example, if we have a single Logical volume, which has been mapped to one of the physical volume, that physical volume is a physical hard-drive.

Now if we need to upgrade our server with SSD Hard-drive, what we used to think at first? reformat of disk? No! we don’t have to reformat the server. The LVM has the option to migrate those old SATA Drives with new SSD Drives. The Live migration will support any kind of disks, be it local drive, SAN or Fiber channel too.

Requirements

There are two ways to migrate LVM partitions (Storages), one is using Mirroring method and other usingpvmove command. For demonstration purpose, here I’m using Centos6.5, but same instructions can also be supported for RHEL, Fedora, Oracle Linux and Scientific Linux.

My Server Setup

Operating System : CentOS 6.5 Final
IP Address  : 192.168.0.224
System Hostname  : lvmmig.tecmintlocal.com

Step 1: Check for Present Drives

1. Assume we are already having one virtual drive named “vdb“, which mapped to one of the logical volume “tecmint_lv“. Now we want to migrate this “vdb” logical volume drive to some other new storage. Before moving further, first verify that the virtual drive and logical volume names with the help of fdisk and lvs commands as shown.

# fdisk -l | grep vd
# lvs

Check Logical Volume Disk

Step 2: Check for Newly added Drive

2. Once we confirm our existing drives, now it’s time to attach our new SSD drive to system and verify newly added drive with the help of fdisk command.

# fdisk -l | grep dev

Check New Added Drive

Note: Did you see in the above screen, that the new drive has been added successfully with name “/dev/sda“.

Step 3: Check Present Logical and Physical Volume

3. Now move forward to create physical volume, volume group and logical volume for migration. Before creating volumes, make sure to check the present logical volume data under /mnt/lvm mount point. Use the following commands to list the mounts and check the data.

# df -h
# cd /mnt/lvm
# cat tecmint.txt

Check Logical Volume Data

Note: For demonstration purpose, we’ve created two files under /mnt/lvm mount point, and we migrate these data to a new drive without any downtime.

4. Before migrating, make sure to confirm the names of logical volume and volume group for which physical volume is related to and also confirm which physical volume used to hold this volume group and logical volume.

# lvs
# vgs -o+devices | grep tecmint_vg

Confirm Logical Volume Names

Note: Did you see in the above screen, that “vdb” holds the volume group tecmint_vg.

Step 4: Create New Physical Volume

5. Before creating Physical Volume in our new added SSD Drive, we need to define the partition using fdisk. Don’t forget to change the Type to LVM(8e), while creating partitions.

# pvcreate /dev/sda1 -v
# pvs

Create Physical Volume

6. Next, add the newly created physical volume to existing volume group tecmint_vg using ‘vgextend command’

# vgextend tecmint_vg /dev/sda1
# vgs

Add Physical Volume

7. To get the full list of information about volume group use ‘vgdisplay‘ command.

# vgdisplay tecmint_vg -v

List Volume Group Info

Note: In the above screen, we can see at the end of result as our PV has added to the volume group.

8. If in-case, we need to know more information about which devices are mapped, use the ‘dmsetup‘ dependency command.

# lvs -o+devices
# dmsetup deps /dev/tecmint_vg/tecmint_lv

In the above results, there is 1 dependencies (PV) or (Drives) and here 17 were listed. If you want to confirm look into the devices, which has major and minor number of drives that are attached.

# ls -l /dev | grep vd

List Device Information

Note: In the above command, we can see that major number with 252 and minor number 17 is related to vdb1. Hope you understood from above command output.

Step 5: LVM Mirroring Method

9. Now it’s time to do migration using Mirroring method, use ‘lvconvert‘ command to migrate data from old logical volume to new drive.

# lvconvert -m 1 /dev/tecmint_vg/tecmint_lv /dev/sda1

-m = mirror
1 = adding a single mirror

Mirroring Method Migration

Note: The above migration process will take long time according to our volume size.

10. Once migration process completed, verify the converted mirror.

# lvs -o+devices

Verify Converted Mirror

11. Once you sure that the converted mirror is perfect, you can remove the old virtual disk vdb1. The option -mwill remove the mirror, earlier we’ve used 1 for adding the mirror.

# lvconvert -m 0 /dev/tecmint_vg/tecmint_lv /dev/vdb1

Remove Virtual Disk

12. Once old virtual disk is removed, you can re-check the devices for logical volumes using following command.

# lvs -o+devices
# dmsetup deps /dev/tecmint_vg/tecmint_lv
# ls -l /dev | grep sd

Check New Mirrored Device

In the above picture, did you see that our logical volume now depends on 8,1 and has sda1. This indicates that our migration process is done.

13. Now verify the files that we’ve migrated from old to new drive. If same data is present at the new drive, that means we have done every steps perfectly.

# cd /mnt/lvm/
# cat tecmin.txt

Check Mirrored Data

14. After everything perfectly created, now it’s time to delete the vdb1 from volume group and later confirm, which devices are depends on our volume group.

# vgreduce /dev/tecmint_vg /dev/vdb1
# vgs -o+devices

15. After removing vdb1 from volume group tecmint_vg, still our logical volume is present there because we have migrated it to sda1 from vdb1.

# lvs

Delete Virtual Disk

Step 6: LVM pvmove Mirroring Method

16. Instead using ‘lvconvert’ mirroring command, we use here ‘pvmove‘ command with option ‘-n‘ (logical volume name) method to mirror data between two devices.

# pvmove -n /dev/tecmint_vg/tecmint_lv /dev/vdb1 /dev/sda1

The command is one of the simplest way to mirror the data between two devices, but in real environmentMirroring is used more often than pvmove.

Conclusion

In this article, we have seen how to migrate the logical volumes from one drive to other. Hope you have learnt new tricks in logical volume management. For such setup one should must know about the basic of logical volume management.

Sunday, January 24, 2016

Packet Capturing with TCPDUMP command in linux

Its no tool other than TCPDUMP. Tcpdump is a very powerful tool because of its strength in capturing packets based on different parameters given. It operates on network layer, so will be able to capture all the packets in and out of the machine. You can use tcpdump to capture and save the packets to a file to analyse it later.

TCPDUMP uses Libpcap(a c/c++ library that's used for packet capturing.)

There are other tools out there which does the same job of packet capture/analyzing like wireshark, but tcpdump keeps all the captures raw. Which means its shows us the raw data it captures as it is.

Things to understand before we go ahead.

tcpdump works in network layer
a network packet header consists of sender,destination,state information and other flag informations.
TCPDUMP only captures the first 96bytes of data from the packet by default.

TCPDUMP can be downloaded from here

Most of the linux distributions these days comes preloaded with tcpdump tool. But you need to be root or sudo permissions to run the tool.

Checking if TCPDUMP is already installed on the machine.

[root@myvm ~]# rpm -qa | grep tcpdump
tcpdump-3.9.4-15.el5

the above command searches the rpm database and greps for tcpdump package.

The advantage of using TCPDUMP over other packet analyzers is that you will need to understand a certain protocol in TCP in its detailed form. Otherwise deciphering the raw data captured by tcpdump is quite difficult without the understanding of TCP protocols.

Hence using TCPDUMP in a way will keep yourself updated about how a certain protocol communicates over the wire.

Lets have a look at some of the basic options available in TCPDUMP, and then will go into further options.

-i option in tcpdump

this option is used to specify the interface. Using this option we can tell tcpdump to capture packets that's coming towards a particular interface. For example

1

2

3

[root@myvm ~]# tcpdump -i lo

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes

Its clear from the above command that tcpdump is only listening on loopback interface for packets. And as mentioned before, the output clearly says that its capturing only 96bytes of the packet.

1

2

3

[root@myvm ~]# tcpdump -i eth0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

the above command will dump all the packets thats destined towards eth0 interface. TCPDUMP output will be very fast, and will fill the screen if you got lot of connections.

-n option in tcpdump

if you do not use tcpdump with -n option, all the sender and destination host address will be in "name" format, which means all ip's will be displayed with hostnames.

Using -n option with tcpdump will disable name lookup. This will display all the output in sender and reciever's IP address format.

-c option in tcpdump

by using -c option you can specify the number of packets that needs to be captured. For example if you only want to capture 2 packets you will do something as shown below.

[root@myvm ~]# tcpdump -n -c 2 -i eth0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

01:55:55.393805 IP 172.16.140.41.5910 > 172.16.134.85.51907: . 31288437:31292817(4380) ack 3681524545 win 71

01:55:55.394626 IP 172.16.134.85.51907 > 172.16.140.41.5910: . ack 2920 win 1053

2 packets captured

4 packets received by filter

0 packets dropped by kernel

as shown in the above command and its result you can clearly see that we told tcpdump to only capture 2 packets from eth0 interface using -c option.

-s option in tcpdump

as mentioned earlier by default tcpdump only captures the firs 96bytes of a packet. But suppose you need to capture packets in its full size then you need to pass the size option -s with its argument.

You can either use -s0 option to capture the whole packet or use number of bytes with -s argument.

[root@myvm ~]# tcpdump -s0 -n -c 2 -i eth0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

01:58:31.011304 IP 172.16.134.85.51907 > 172.16.140.41.5910: P 3681527491:3681527497(6) ack 32290881 win 776

01:58:31.011312 IP 172.16.140.41.5910 > 172.16.134.85.51907: . ack 6 win 71

2 packets captured

2 packets received by filter

0 packets dropped by kernel

as you can see from the above output, its clearly mentioned that capture size is 65535 bytes instead of 96 bytes(the capture size is made bold in the output of the above command)

-e option in tcpdump

from all the above output we till now saw, the output only showed us information about the sender and receivers ip address. Suppose you want the mac address of the sender and reciever then you can include -e option.

See our example output below.

[root@myvm ~]# tcpdump -s0 -e -n -c 2 -i eth0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

02:00:10.545520 1a:e5:ad:00:b5:20 > Broadcast, ethertype IPv4 (0x0800), length 92: 172.16.140.20.netbios-ns > 172.16.140.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST

02:00:10.546598 00:15:17:8d:0c:9c > 46:0a:98:3d:41:b1, ethertype IPv4 (0x0800), length 210: 172.16.140.41.56153 > 172.16.140.33.49155: P 1489870205:1489870349(144) ack 2064846002 win 96 <nop,nop,timestamp 1157179149 382990921>

2 packets captured

4 packets received by filter

0 packets dropped by kernel

from the above output shown you can see the MAC address in the output(Mac addressess are made bold in the output)

-vvv option for more verbose output in tcpdump

If you want your tcpdumpt output to show you more verbose information like, show all the flags, and headers in tcp we can use verbose options.

-v for little more packet information,-vv for further more, and -vvv option for even more information. An example output is shown below

1

2

3

4

5

6

7

[root@myvm ~]# tcpdump -s0 -vvv -e -n -c 2 -i eth0

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

02:21.671828 00:15:17:83:0c:9c > 00:23:47:4c:97:00, ethertype IPv4 (0x0800), length 4434: (tos 0x0, ttl  64, id 20588, offset 0, flags [DF], proto: TCP (6), length: 4420) 172.16.140.41.5910 > 172.16.134.85.51907: ., cksum 0x7bd6 (incorrect (-> 0x2515), 34181155:34185535(4380) ack 3681533777 win 71

02:21.672181 00:23:47:4c:97:00 > 00:15:17:83:0c:9c, ethertype IPv4 (0x0800), length 338: (tos 0x10, ttl  59, id 34117, offset 0, flags [DF], proto: TCP (6), length: 324) 172.17.4.111.ssh > 172.16.140.41.58728: P, cksum 0x4294 (correct), 4107752623:4107752895(272) ack 2150369028 win 33148 <nop,nop,timestamp 2658690172 1157308260>

packets captured

packets received by filter

packets dropped by kernel

-S option in tcpdump

this option in tcpdump can be used for showing absolute sequence numbers. Now what is sequence number?

Sequence number is used in TCP, to identify the number of packets send or recieved. Whenever a machine initiates a TCP connection it informs the other side about its sequence number during thethree way handshake.

With the help of the sequence number's the receiver and the sender comes to know how much data has been transferred.

TCPDUMP even show these sequence numbers. Using -S option will shown the abosolute tcp sequence numbers rather than relative with previous packets.

-w option used in tcpdump

using this -w option we can capture the output and save all the output to a specified file. This file can be later analyzed with the help of tools like editcap.

using .pcap extention to the filename is advisable as this makes it readable by other packet analyzers.

1

2

3

4

5

[root@myvm ~]#  tcpdump -w sampletcpdump.pcap -s0 -vvv -e -n -c 2 -i eth0

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

2 packets captured

5 packets received by filter

0 packets dropped by kernel

Dont read the file by opening it thorugh cat or vim...because you will not be able to read it.

-r option used in tcpdump

in order to read the file we just captured we need to use -r option with tcpdump command and passing filename as the argument to the command.

1

2

3

4

[root@myvm ~]# tcpdump -r sampletcpdump.pcap

reading from file sampletcpdump.pcap, link-type EN10MB (Ethernet)

02:04:04.712709 IP 172.16.134.150.50438 > m1-sv-xbox2.5919: P 1245960226:1245960232(6) ack 671383339 win 64701

02:04:04.712724 IP myvm.5919 > 172.16.134.150.50438: . ack 6 win 46

Display packets for a particular port using TCPDUMP

Till now in all above shown example we got all the packets towards all ports and were from random protocols, whatever the tool got during the capture, it showed those things.

Now in case if you want to capture the packets thats coming towards port 22 of one server.

1

2

3

4

5

6

7

[root@myvm ~]# tcpdump -s0 -vvv -e -n -c 2 -i eth0 port 22

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

09:33.185445 00:23:47:4c:97:00 > 00:15:17:83:0c:9c, ethertype IPv4 (0x0800), length 338: (tos 0x10, ttl  59, id 5386, offset 0, flags [DF], proto: TCP (6), length: 324) 172.16.0.111.ssh > 172.16.140.41.58728: P, cksum 0xacee (correct), 4107814047:4107814319(272) ack 2150369076 win 33148 <nop,nop,timestamp 2659121792 1157739776>

09:33.185453 00:15:17:83:0c:9c > 00:23:47:4c:97:00, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl  64, id 46661, offset 0, flags [DF], proto: TCP (6), length: 52) 172.16.140.41.58728 > 172.16.0.111.ssh: ., cksum 0x49c9 (correct), 1:1(0) ack 272 win 501 <nop,nop,timestamp 1157741788 2659121792>

packets captured

packets received by filter

packets dropped by kernel

you can clearly see from the above output that all the packets captured with the port 22 option are for ssh.

Ignoring Packets with TCPDUMP

If you want to ignore the packets coming towards port 80 and show all rest of the packets then you can do that by using the same port option but in a different way.

Lets look at an example to do that with tcpdump

[root@myvm ~]# tcpdump -i eth0 -n -c 5 'port !80'

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

11:30.822544 arp who-has 192.168.0.152 (Broadcast) tell 192.168.0.152

11:30.875907 IP 172.16.140.41.5910 > 172.16.134.85.51907: . 44976382:44982222(5840) ack 3681551939 win 71

11:30.876707 IP 172.16.134.85.51907 > 172.16.140.41.5910: . ack 2920 win 1791

11:30.876724 IP 172.16.140.41.5910 > 172.16.134.85.51907: . 5840:8760(2920) ack 1 win 71

11:30.877604 IP 172.16.134.85.51907 > 172.16.140.41.5910: . ack 5840 win 1780

packets captured

packets received by filter

packets dropped by kernel

By doing the above thing your will screen will be dumped with all the traffic other than the traffic towards port 80.

show packets towards a particular host

Suppose you are trouble shooting something and only interested in knowing the traffic towards or from a particular host. In that case you can ask tcpdump to only show packets for that host, by the following command.

[root@slashroot ~]# tcpdump -i eth0 -c 5 host 192.168.159.128

host option can be used to do that. Always using -c option for specifying no of packets to capture is a good idea, other wise your screen will be dumped with all packets captured.

Show packets from source with tcpdump

Now you can even go further by only asking to show packets with a particular source address. This can be done by the following command.

[root@slashroot ~]# tcpdump -i eth0 -c 5 src host 192.168.159.128

So you just need to put "src" option along with the host option for doing that as shown above.

Similarly you can do for destination as shown below.

[root@slashroot ~]# tcpdump -i eth0 -c 5 dst host 192.168.159.128

Filtering protocols using tcpdump command

You can easily get information about packets of a certain protocol with the help of tcpdump. Without filtering tcpdump output with relevant options and arguments, the packets of interest can get lost in the huge amount of output dumped by tcpdump.

Lets see how can we look at the packets with certain protocols in it. Doing that is quite simple, you need to just pass the protocol name as argument after the command.

[root@slashroot ~]# tcpdump -i eth0 icmp

[root@slashroot ~]# tcpdump -i eth0 tcp

[root@slashroot ~]# tcpdump -i eth0 udp

[root@slashroot ~]# tcpdump -i eth0 arp