第五周总结

软RAID

mdadm：为软RAID提供管理界面

为空余磁盘添加冗余

结合内核中的md(multi devices)

RAID设备可命名为/dev/md0、/dev/md1、/dev/md2、/dev/md3等

如果硬盘只分出了一个分区，可以用

dd if=/dev/sdb of=/dev/sdc bs=1 count=66 skip=446 seek=446

dd if=/dev/sdb of=/dev/sdd bs=1 count=66 skip=446 seek=446

dd if=/dev/sdb of=/dev/sde bs=1 count=66 skip=446 seek=446

raid5的实现

实验：raid5

1 分区（fd），磁盘大小一致

2 mdadm -C /dev/md0 -a yes -l5 -n 3 -x1 -c32 /dev/sd{b,c,d,e}1

3 mkfs.ext4 /dev/md0

4 vim /etc/fstab

5 mdadm -Ds /dev/md0 > /etc/mdadm.conf

6 mdadm -S /dev/md0 停止(umount)

7 mdadm -A /dev/md0 激活

8 mdadm /dev/md0 -f /dev/sde1 损坏

9 mdadm /dev/md0 -r /dev/sde1 删除

10 mdadm /dev/md0 -a /dev/sde1 加

dd if=/dev/zero of=/dev/sdb1 清空文件系统

[root@centos ~]#mdadm -C /dev/md0 -a yes（自动初始化） -l5（raid5） -c（chunk大小） 32 -n（几块硬盘） 3 -x（空闲硬盘） 1 /dev/sdb4 /dev/sdc2 /dev/sdd1 /dev/sde1

创建md0时不能超过现在机器的物理磁盘的容量

[root@centos ~]#mdadm -D /dev/md0 查看raid 等同于 mdadm –detail /dev/md0

/dev/md0:

Version : 1.2

Creation Time : Wed Apr 25 18:42:03 2018

Raid Level : raid5

Array Size : 4204864 (4.01 GiB 4.31 GB)

Used Dev Size : 2102432 (2.01 GiB 2.15 GB)

Raid Devices : 3

Total Devices : 4

Persistence : Superblock is persistent

Update Time : Wed Apr 25 18:42:14 2018

State : clean

Active Devices : 3

Working Devices : 4

Failed Devices : 0

Spare Devices : 1

Layout : left-symmetric

Chunk Size : 32K

Name : centos.localdomain:0 (local to host centos.localdomain)

UUID : 334cca96:877322f5:7cd0110c:afdf35bf

Events : 18

Number Major Minor RaidDevice State

0 8 20 0 active sync /dev/sdb4

1 8 34 1 active sync /dev/sdc2

4 8 49 2 active sync /dev/sdd1

3 8 65 – spare /dev/sde1

[root@centos ~]#blkid

/dev/sda1: UUID=”414ab9ee-af40-44f1-b300-d29fc674eb8e” TYPE=”ext4″

/dev/sda2: UUID=”408b65d7-7551-480b-8916-5eff4cca1b00″ TYPE=”ext4″

/dev/sda3: UUID=”37d5cf25-ead4-427b-9655-40fc1602b5df” TYPE=”ext4″

/dev/sda5: UUID=”9fc26932-7094-4948-b395-24ab05d8ea73″ TYPE=”swap”

/dev/sdb2: UUID=”e4c92849-b5b6-4736-a027-25c99f9a9da1″ TYPE=”ext4″

/dev/sdb1: UUID=”309c1b15-1d6d-475a-a077-0caf7634442a” TYPE=”ext4″

/dev/sdc1: LABEL=”swap_sdc1″ UUID=”ca1c9c9d-7ed6-45b3-89a4-e8ff36548d0b” TYPE=”swap”

/dev/sdb3: LABEL=”/data/sdb3″ UUID=”e7742e5e-f923-4c7e-b43e-ee0952a0045e” TYPE=”ext4″

/dev/sdb4: UUID=”334cca96-8773-22f5-7cd0-110cafdf35bf” UUID_SUB=”396e1715-7f32-a631-46ab-6819b9a95f42″ LABEL=”centos.localdomain:0″ TYPE=”linux_raid_member”

/dev/sdc2: UUID=”334cca96-8773-22f5-7cd0-110cafdf35bf” UUID_SUB=”8534ca6d-ff60-618d-d973-13566069ebd4″ LABEL=”centos.localdomain:0″ TYPE=”linux_raid_member”

/dev/sdd1: UUID=”334cca96-8773-22f5-7cd0-110cafdf35bf” UUID_SUB=”755ca4f0-1a5d-ca6a-086e-866fc3164673″ LABEL=”centos.localdomain:0″ TYPE=”linux_raid_member”

/dev/sde1: UUID=”334cca96-8773-22f5-7cd0-110cafdf35bf” UUID_SUB=”b84b7b36-3b55-6a14-ae42-3e6f1dbf3af5″ LABEL=”centos.localdomain:0″ TYPE=”linux_raid_member”

[root@centos ~]#mkfs.ext4 /dev/md0 -L “raid5” 为raid5创建文件系统

mke2fs 1.41.12 (17-May-2010)

Filesystem label=raid5

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=8 blocks, Stripe width=16 blocks

262944 inodes, 1051216 blocks

52560 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=1077936128

33 block groups

32768 blocks per group, 32768 fragments per group

7968 inodes per group

Superblock backups stored on blocks:

32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 23 mounts or

180 days, whichever comes first. Use tune2fs -c or -i to override.

[root@centos ~]#blkid /dev/md0

/dev/md0: LABEL=”raid5″ UUID=”d9b419d6-ae30-4c1a-b447-deeb1b5a0111″ TYPE=”ext4″

[root@centos ~]#mkdir /data/raid

[root@centos ~]#vim /etc/fstab

UUID=d9b419d6-ae30-4c1a-b447-deeb1b5a0111 /data/raid ext4 defaults 0 0

[root@centos ~]#mount -a

[root@centos ~]#df

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/sda2 50264772 5721952 41982820 12% /

tmpfs 953128 76 953052 1% /dev/shm

/dev/sda1 999320 34944 911948 4% /boot

/dev/sda3 30106576 99988 28470588 1% /data

/dev/sr0 6049912 6049912 0 100% /media/CDROM

/dev/md0 4007728 8216 3789272 1% /data/raid

[root@centos ~]#mdadm -Ds /dev/md0 显示raid0的必要信息

ARRAY /dev/md0 metadata=1.2 spares=1 name=centos.localdomain:0 UUID=334cca96:877322f5:7cd0110c:afdf35bf

[root@centos ~]#mdadm -Ds /dev/md0 > /etc/mdadm.conf 将信息写入配置文件中，否则会在raid禁用启用时找不到文件

[root@centos ~]#ll /etc/mdadm.conf

-rw-r–r–. 1 root root 104 Apr 25 18:59 /etc/mdadm.conf

[root@centos ~]#dd if=/dev/zero of=/data/raid/f1 bs=1M count=1024 conv=fdatasync 测试一下写入速度

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 51.6999 s, 20.8 MB/s

停用raid

1.卸载挂载

[root@centos ~]#umount /data/raid

[root@centos ~]#mdadm -S /dev/md0

mdadm: stopped /dev/md0

[root@centos ~]#ll /dev/md0

ls: cannot access /dev/md0: No such file or directory

启用

[root@centos ~]#mdadm -A /dev/md0

mdadm: /dev/md0 has been started with 3 drives and 1 spare.

[root@centos ~]#ll /dev/md0

brw-rw—-. 1 root disk 9, 0 Apr 25 19:30 /dev/md0

模拟损坏硬盘

[root@centos ~]#mdadm -f /dev/md0 /dev/sdd1 损坏的是/dev/sdd1

mdadm: set /dev/sdd1 faulty in /dev/md0

[root@centos ~]#mdadm -D /dev/md0

Number Major Minor RaidDevice State

0 8 20 0 active sync /dev/sdb4

1 8 34 1 active sync /dev/sdc2

3 8 65 2 spare rebuilding /dev/sde1

4 8 49 – faulty /dev/sdd1

移除损坏的硬盘

[root@centos ~]#mdadm -r /dev/md0 /dev/sdd1

mdadm: hot removed /dev/sdd1 from /dev/md0

[root@centos ~]#mdadm -D /dev/md0

/dev/md0:

Version : 1.2

Creation Time : Wed Apr 25 18:42:03 2018

Raid Level : raid5

Array Size : 4204864 (4.01 GiB 4.31 GB)

Used Dev Size : 2102432 (2.01 GiB 2.15 GB)

Raid Devices : 3

Total Devices : 3

Persistence : Superblock is persistent

Update Time : Wed Apr 25 19:39:01 2018

State : clean

Active Devices : 3

Working Devices : 3

Failed Devices : 0

Spare Devices : 0

Layout : left-symmetric

Chunk Size : 32K

Name : centos.localdomain:0 (local to host centos.localdomain)

UUID : 334cca96:877322f5:7cd0110c:afdf35bf

Events : 38

Number Major Minor RaidDevice State

0 8 20 0 active sync /dev/sdb4

1 8 34 1 active sync /dev/sdc2

3 8 65 2 active sync /dev/sde1

[root@centos ~]#blkid

EL=”centos.localdomain:0″ TYPE=”linux_raid_member”

/dev/sdc2: UUID=”334cca96-8773-22f5-7cd0-110cafdf35bf” UUID_SUB=”8534ca6d-ff60-618d-d973-13566069ebd4″ LABEL=”centos.localdomain:0″ TYPE=”linux_raid_member”

/dev/sdd1: UUID=”334cca96-8773-22f5-7cd0-110cafdf35bf” UUID_SUB=”755ca4f0-1a5d-ca6a-086e-866fc3164673″ LABEL=”centos.localdomain:0″ TYPE=”linux_raid_member”

/dev/sde1: UUID=”334cca96-8773-22f5-7cd0-110cafdf35bf” UUID_SUB=”b84b7b36-3b55-6a14-ae42-3e6f1dbf3af5″ LABEL=”centos.localdomain:0″ TYPE=”linux_raid_member”

/dev/md0: LABEL=”raid5″ UUID=”d9b419d6-ae30-4c1a-b447-deeb1b5a0111″ TYPE=”ext4″

移除后再加回去，加回去后变成备用硬盘

[root@centos ~]#mdadm -a /dev/md0 /dev/sdd1

mdadm: added /dev/sdd1

[root@centos ~]#mdadm -D /dev/md0

/dev/md0:

Version : 1.2

Creation Time : Wed Apr 25 18:42:03 2018

Raid Level : raid5

Array Size : 4204864 (4.01 GiB 4.31 GB)

Used Dev Size : 2102432 (2.01 GiB 2.15 GB)

Raid Devices : 3

Total Devices : 4

Persistence : Superblock is persistent

Update Time : Wed Apr 25 19:41:42 2018

State : clean

Active Devices : 3

Working Devices : 4

Failed Devices : 0

Spare Devices : 1

Layout : left-symmetric

Chunk Size : 32K

Name : centos.localdomain:0 (local to host centos.localdomain)

UUID : 334cca96:877322f5:7cd0110c:afdf35bf

Events : 39

Number Major Minor RaidDevice State

0 8 20 0 active sync /dev/sdb4

1 8 34 1 active sync /dev/sdc2

3 8 65 2 active sync /dev/sde1

4 8 49 – spare /dev/sdd1

关机移除备用硬盘和raid中任意一块硬盘，启动后还是可以使用但是速度变慢

[root@centos ~]#df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 48G 5.5G 41G 12% /

tmpfs 931M 76K 931M 1% /dev/shm

/dev/sda1 976M 35M 891M 4% /boot

/dev/sda3 29G 98M 28G 1% /data

/dev/md0 3.9G 9.9M 3.7G 1% /data/raid

/dev/sr0 5.8G 5.8G 0 100% /media/CDROM

[root@centos ~]#cat /data/raid/passwd

root:x:0:0:root:/root:/bin/bash

bin:x:1:1:bin:/bin:/sbin/nologin

daemon:x:2:2:daemon:/sbin:/sbin/nologin

adm:x:3:4:adm:/var/adm:/sbin/nologin

[root@centos ~]#mdadm -D /dev/md0

/dev/md0:

Version : 1.2

Creation Time : Wed Apr 25 18:42:03 2018

Raid Level : raid5

Array Size : 4204864 (4.01 GiB 4.31 GB)

Used Dev Size : 2102432 (2.01 GiB 2.15 GB)

Raid Devices : 3

Total Devices : 2

Persistence : Superblock is persistent

Update Time : Wed Apr 25 20:27:58 2018

State : clean, degraded 降级使用是特别危险的，需要及时更换硬盘

Active Devices : 2

Working Devices : 2

Failed Devices : 0

Spare Devices : 0

Layout : left-symmetric

Chunk Size : 32K

Name : centos.localdomain:0 (local to host centos.localdomain)

UUID : 334cca96:877322f5:7cd0110c:afdf35bf

Events : 41

Number Major Minor RaidDevice State

0 8 20 0 active sync /dev/sdb4

1 8 34 1 active sync /dev/sdc2

4 0 0 4 removed

更换硬盘

先分出一个分区（类型是fd）2G大小

[root@centos ~]#fdisk /dev/sdc

WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to

switch off the mode (command ‘c’) and change display units to

sectors (command ‘u’).

Command (m for help): n

Command action

e extended

p primary partition (1-4)

Invalid partition number for type `3′

Command action

e extended

p primary partition (1-4)

Partition number (1-4): 3

First cylinder (525-10443, default 525):

Using default value 525

Last cylinder, +cylinders or +size{K,M,G} (525-10443, default 10443): +2G

Command (m for help): t

Partition number (1-4): 3

Hex code (type L to list codes): fd

Changed system type of partition 3 to fd (Linux raid autodetect)

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.

The kernel still uses the old table. The new table will be used at

the next reboot or after you run partprobe(8) or kpartx(8)

Syncing disks.

[root@centos ~]#mdadm -a /dev/md0 /dev/sdc3

mdadm: Cannot find /dev/sdc3: No such file or directory

[root@centos ~]#partx -a /dev/sdc

BLKPG: Device or resource busy

error adding partition 1

BLKPG: Device or resource busy

error adding partition 2

[root@centos ~]#mdadm -a /dev/md0 /dev/sdc3

mdadm: added /dev/sdc3

实现raid10

mdadm -C /dev/md0 -a yes -l1 -n2 /dev/sd{b,c}

mdadm -C /dev/md1 -a yes -l1 -n2 /dev/sd{d,e}

mdadm -C /dev/md2 -a yes -l0 -n2 /dev/md{0,1}

或者

mdadm -C /dev/md0 -a yes -l10 -n4 /dev/sd{b,c,d,e}

如果不够用了增加一个磁盘

[root@centos ~]#fdisk /dev/sdc

WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to

switch off the mode (command ‘c’) and change display units to

sectors (command ‘u’).

Command (m for help): n

Command action

e extended

p primary partition (1-4)

Selected partition 4

First cylinder (787-10443, default 787):

Using default value 787

Last cylinder, +cylinders or +size{K,M,G} (787-10443, default 10443): +2G

Command (m for help): t

Partition number (1-4): 4

Hex code (type L to list codes): fd

Changed system type of partition 4 to fd (Linux raid autodetect)

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.

The kernel still uses the old table. The new table will be used at

the next reboot or after you run partprobe(8) or kpartx(8)

Syncing disks.

[root@centos ~]#mdadm /dev/md0 -G -n4 -a /dev/sdc4

mdadm: added /dev/sdc4

[root@centos ~]#mdadm -D /dev/md0

/dev/md0:

Version : 1.2

Creation Time : Wed Apr 25 18:42:03 2018

Raid Level : raid5

Array Size : 4204864 (4.01 GiB 4.31 GB)

Used Dev Size : 2102432 (2.01 GiB 2.15 GB)

Raid Devices : 4

Total Devices : 4

Persistence : Superblock is persistent

Update Time : Wed Apr 25 20:53:55 2018

State : clean, reshaping

Active Devices : 4

Working Devices : 4

Failed Devices : 0

Spare Devices : 0

Layout : left-symmetric

Chunk Size : 32K

Reshape Status : 5% complete

Delta Devices : 1, (3->4)

Name : centos.localdomain:0 (local to host centos.localdomain)

UUID : 334cca96:877322f5:7cd0110c:afdf35bf

Events : 84

Number Major Minor RaidDevice State

0 8 20 0 active sync /dev/sdb4

1 8 34 1 active sync /dev/sdc2

3 8 35 2 active sync /dev/sdc3

4 8 36 3 active sync /dev/sdc4

[root@centos ~]#df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 48G 5.5G 41G 12% /

tmpfs 931M 76K 931M 1% /dev/shm

/dev/sda1 976M 35M 891M 4% /boot

/dev/sda3 29G 98M 28G 1% /data

/dev/md0 3.9G 9.9M 3.7G 1% /data/raid 增加了一块硬盘却没有增大，是因为没有文件系统。

/dev/sr0 5.8G 5.8G 0 100% /media/CDROM

[root@centos ~]#resize2fs /dev/md0 同步文件系统，但是这个命令只适用于ext4

resize2fs 1.41.12 (17-May-2010)

Filesystem at /dev/md0 is mounted on /data/raid; on-line resizing required

old desc_blocks = 1, new_desc_blocks = 1

Performing an on-line resize of /dev/md0 to 1576824 (4k) blocks.

The filesystem on /dev/md0 is now 1576824 blocks long.

[root@centos ~]#df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 48G 5.5G 41G 12% /

tmpfs 931M 76K 931M 1% /dev/shm

/dev/sda1 976M 35M 891M 4% /boot

/dev/sda3 29G 98M 28G 1% /data

/dev/md0 5.8G 9.9M 5.5G 1% /data/raid

/dev/sr0 5.8G 5.8G 0 100% /media/CDROM

centos7也就是xfs同步命令是 xfs_growfs /data/raid 不能写设备，只能写挂载点。

[root@centos ~]#cat /proc/mdstat 查看raid信息有备用硬盘会显示S

Personalities : [raid6] [raid5] [raid4]

md0 : active raid5 sdc4[4] sdc3[3] sdb4[0] sdc2[1]

6307296 blocks super 1.2 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

删除raid

1.卸载挂载

删除 /etc/fstab中相应的行

[root@centos ~]#ll /dev/md0

brw-rw—-. 1 root disk 9, 0 Apr 25 21:02 /dev/md0

[root@centos ~]#rm -f /dec/mdamd.conf

[root@centos ~]#mdadm -S /dev/md0

mdadm: stopped /dev/md0

[root@centos ~]#ll /dev/md0 有点延迟，正常不显示

brw-rw—-. 1 root disk 9, 0 Apr 25 21:17 /dev/md0

[root@centos ~]#mdadm -D /dev/md0

/dev/md0:

Version :

Raid Level : raid0

Total Devices : 0

State : inactive

Number Major Minor RaidDevice

[root@centos ~]#ll /dev/md0

ls: cannot access /dev/md0: No such file or directory

[root@centos ~]#blkid 设备还在