MogileFS是一个开源的分布式文件存储系统，由LiveJournal旗下的Danga Interactive公司开发。Danga团队开发了包括 Memcached、MogileFS、Perlbal 等多个知名的开源项目。目前使用MogileFS 的公司非常多，如日本排名先前的几个互联公司及国内的yupoo(又拍)、digg、豆瓣、1号店、大众点评、搜狗和安居客等，分别为所在的组织或公司管理着海量的图片。
MogileFS由3个部分组成：
(1)server：主要包括mogilefsd和mogstored两个应用程序。mogilefsd实现的是tracker，它通过数据库来保存元数据信息，包括站点domain、class、host等；mogstored是存储节点(store node)，它其实是个WebDAV服务，默认监听在7500端口，接受客户端的文件存储请求。在MogileFS安装完后，要运行mogadm工具将所有的store node注册到mogilefsd的数据库里，mogilefsd会对这些节点进行管理和监控。
(2) utils（工具集）：主要是MogileFS的一些管理工具，例如mogadm等。
(3)客户端API：MogileFS的客户端API很多，例如Perl、PHP、Java、Python等，用这个模块可以编写客户端程序，实现文件的备份管理功能等。
现在我们在做一个mogilefs集群分布式存储,节点设计如下:
192.168.42.150 node1 [Nginx,Tracker, Storage,DataBase]
192.168.42.151 node2 [Tracker, Storage]
192.168.42.152 node3 [Tracker, Storage]
1.准备这三台机器,初始化工作,时间同步,分别在这三台机器上安装mogilefs,我们这里用rpm安装
cd /usr/local/src #下载RPM安装包 wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Server-2.46-2.el6.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Utils-2.19-1.el6.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/perl-MogileFS-Client-1.14-1.el6.noarch.rpm wget http://192.168.42.26/install_package/down/mogilefs/perl-Perlbal-1.78-1.el6.noarch.rpm #安装依赖 yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO -y #安装rpm yum install *.rpm -y
2.在node1节点上部署mariadb
yum install mariadb-server -y vim /etc/my.cnf.d/server.cnf [mysqld] skip_name_resolve=1 innodb_file_per_table = 1 #启动mysql,做以下操作 systemctl start mariadb mysql MariaDB [(none)]> GRANT ALL ON *.* TO 'root'@'192.168.42.%' IDENTIFIED BY 'mageedu'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'moguser'@'192.168.42.%' IDENTIFIED BY 'mogpass' WITH GRANT OPTION; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> CREATE DATABASE mogilefs; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> exit Bye #设定数据库,生成数据表 mogdbsetup --dbhost=192.168.42.150 --dbrootpass='mageedu' --dbuser='moguser' --dbpass='mogpass' --dbname=mogilefs MariaDB [(none)]> use mogilefs; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MariaDB [mogilefs]> show tables; +----------------------+ | Tables_in_mogilefs | +----------------------+ | checksum | | class | | device | | domain | | file | | file_on | | file_on_corrupt | | file_to_delete | | file_to_delete2 | | file_to_delete_later | | file_to_queue | | file_to_replicate | | fsck_log | | host | | server_settings | | tempfile | | unreachable_fids | +----------------------+ 17 rows in set (0.00 sec)
3.配置mogilefsd
vim /etc/mogilefs/mogilefsd.conf db_dsn = DBI:mysql:mogilefs:host=192.168.42.150 db_user = moguser db_pass = mogpass listen = 0.0.0.0:7001 conf_port = 7001
4.启动mogilefsd
service mogilefsd start [root@node1 my.cnf.d]# service mogilefsd start Starting mogilefsd (via systemctl): [ OK ] [root@node1 my.cnf.d]# ss -tnl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:22 *:* LISTEN 0 1024 *:7001 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 50 *:3306 *:* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::*
5.配置storage(可以是所有节点)
mkdir -pv /data/mogilefs/dev1 #创建设备目录
chown -R mogilefs.mogilefs /data/mogilefs/dev1 #权限
vim /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /data/mogilefs/
修改脚本
lockfile下添加
pidfile=/var/run/mogilefsd/mogstored.pid
start()修改
[ $RETVAL = 0 ] && success && touch ${lockfile} && echo $(pidof mogstored) > ${pidfile} || failure
stop()修改
[ $RETVAL = 0 ] && success && rm -f ${lockfile} ${pidfile} || failure
#启动mogstored
[root@node1 init.d]# service mogstored start
Starting mogstored (via systemctl): [ OK ]
[root@node1 init.d]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 1024 *:7500 *:*
LISTEN 0 1024 *:7501 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 1024 *:7001 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 50 *:3306 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
6.node2,node3上分别创建设备目录和分发配置文件,脚本文件
node2上操作: mkdir -pv /data/mogilefs/dev2 chown -R mogilefs.mogilefs /data/mogilefs/dev2 node3上操作: mkdir -pv /data/mogilefs/dev3 chown -R mogilefs.mogilefs /data/mogilefs/dev3 #分发脚本和配置文件 scp /etc/rc.d/init.d/mogstored root@192.168.42.151:/etc/rc.d/init.d/ scp /etc/mogilefs/mogstored.conf root@192.168.42.151:/etc/mogilefs/ scp /etc/rc.d/init.d/mogstored root@192.168.42.152:/etc/rc.d/init.d/ scp /etc/mogilefs/mogstored.conf root@192.168.42.152:/etc/mogilefs/ 在node2,和node3启动mogstored service mogstored start
7.node1节点上需要添加
#查看帮助 mogadm -help [root@node1 init.d]# mogadm check Checking trackers... 127.0.0.1:7001 ... OK Checking hosts... No devices found on tracker(s). 我们还可以使用mogadm --trackers=192.168.42.150:7001 check 得到同样的结果 #添加主机 mogadm --trackers=192.168.42.150:7001 host add 192.168.42.150 --ip=192.168.42.150 --status=alive mogadm --trackers=192.168.42.150:7001 host add 192.168.42.151 --ip=192.168.42.151 --status=alive mogadm --trackers=192.168.42.150:7001 host add 192.168.42.152 --ip=192.168.42.152 --status=alive [root@node1 init.d]# mogadm host list 192.168.42.150 [1]: alive IP: 192.168.42.150:7500 192.168.42.151 [2]: alive IP: 192.168.42.151:7500 192.168.42.152 [3]: alive IP: 192.168.42.152:7500
添加设备
(1).没有添加设备之前查看,是没有结果的
[root@node1 init.d]# mogadm device list
192.168.42.150 [1]: alive
used(G) free(G) total(G) weight(%)
192.168.42.151 [2]: alive
used(G) free(G) total(G) weight(%)
192.168.42.152 [3]: alive
used(G) free(G) total(G) weight(%)
(2)添加设备的操作命令
mogadm device add 192.168.42.150 1 mogadm device add 192.168.42.151 2 mogadm device add 192.168.42.152 3
(3)此时查看有结果了
[root@node1 init.d]# mogadm device list
192.168.42.150 [1]: alive
used(G) free(G) total(G) weight(%)
dev1: alive 1.688 15.298 16.986 100
192.168.42.151 [2]: alive
used(G) free(G) total(G) weight(%)
dev2: alive 1.557 15.430 16.986 100
192.168.42.152 [3]: alive
used(G) free(G) total(G) weight(%)
dev3: alive 1.557 15.430 16.986 100
(4)此时查看mogadm check
[root@node1 init.d]# mogadm check
Checking trackers...
127.0.0.1:7001 ... OK
Checking hosts...
[ 1] 192.168.42.150 ... OK
[ 2] 192.168.42.151 ... OK
[ 3] 192.168.42.152 ... OK
Checking devices...
host device size(G) used(G) free(G) use% ob state I/O%
---- ------------ ---------- ---------- ---------- ------ ---------- -----
[ 1] dev1 16.986 1.689 15.297 9.95% writeable 0.0
[ 2] dev2 16.986 1.557 15.429 9.17% writeable 0.0
[ 3] dev3 16.986 1.557 15.429 9.17% writeable 0.0
---- ------------ ---------- ---------- ---------- ------
total: 50.959 4.803 46.156 9.43%
添加域
(1)查看没有添加域的状态 [root@node1 init.d]# mogadm domain list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- (2)添加域操作 mogadm domain add files mogadm domain add images (3)再次查看列出 [root@node1 init.d]# mogadm domain list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- files default 2 MultipleHosts() NONE images default 2 MultipleHosts() NONE
添加class
(1)查看一下class列表 [root@node1 init.d]# mogadm class list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- files default 2 MultipleHosts() NONE images default 2 MultipleHosts() NONE (2).添加class操作(不知为什么这一步会导致只有一个副本) 需要这样添加 mogadm class add images jpeg --mindevcount=2 --replpolicy="MultipleHosts(2)" mogadm class add images jpeg mogadm class add images png mogadm class add images gif (3).再次查看列表 [root@node1 init.d]# mogadm class list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- files default 2 MultipleHosts() NONE images default 2 MultipleHosts() NONE images gif 2 MultipleHosts() NONE images jpeg 2 MultipleHosts() NONE images png 2 MultipleHosts() NONE
上传图片测试
(1)上传图片
mogupload --tracker=192.168.42.150:7001 --domain=images --key='1.gif' --file='/tmp/83e.gif' --class=gif
mogupload --tracker=192.168.42.150:7001 --domain=images --key='1.jpg' --file='/tmp/zhuoku194.jpg' --class=jpeg
mogupload --tracker=192.168.42.150:7001 --domain=images --key='2.jpg' --file='/tmp/1.jpg' --class=jpeg
(2)查看图片信息
mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.gif'
[root@node1 init.d]# mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.gif'
- file: /1.gif
class: gif
devcount: 1
domain: images
fid: 2
key: /1.gif
length: 499461
- http://192.168.42.151:7500/dev2/0/000/000/0000000002.fid
mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.jpg'
[root@node1 init.d]# mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.jpg'
- file: /1.jpg
class: jpeg
devcount: 1
domain: images
fid: 3
key: /1.jpg
length: 374408
- http://192.168.42.150:7500/dev1/0/000/000/0000000003.fid
(3)在浏览器访问
http://192.168.42.151:7500/dev2/0/000/000/0000000002.fid
http://192.168.42.150:7500/dev1/0/000/000/0000000003.fid
都是ok的
删除文件:
mogdelete --tracker=192.168.42.150:7001 --domain=images --key='1.jpg'
mogdelete --tracker=192.168.42.150:7001 --domain=images --key='2.jpg'
mogdelete --tracker=192.168.42.150:7001 --domain=images --key='1.gif'
删除class:
mogadm class delete images jpeg
mogadm class delete images gitj
mogadm class delete images png
删除域:
mogadm domain delete files
mogadm domain delete images
#配置Nginx
nginx 的mogilefs模块的官方文档地址：
http://www.grid.net.ru/nginx/mogilefs.en.html
wget http://www.grid.net.ru/nginx/download/nginx_mogilefs_module-1.0.4.tar.gz
tar xvf nginx_mogilefs_module-1.0.4.tar.gz
mv nginx_mogilefs_module-1.0.4 /usr/local/
cd /usr/local/src
wget http://192.168.42.26/nginx-1.11.8.tar.gz
tar xvf nginx-1.11.8.tar.gz
cd nginx-1.11.8
yum groupinstall "Development Tools" "Server Platform Developments"
yum install pcre-devel openssl-devel zlib-devel -y
groupadd -r nginx
useradd -r -g nginx nginx
我的编译安装方法
./configure --with-pcre --user=nginx --group=nginx --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_realip_module --add-module=/usr/local/nginx_mogilefs_module-1.0.4
make && make install
编译会可能报错:
cc1: all warnings being treated as errors
make[1]: *** [objs/addon/nginx_mogilefs_module-1.0.4/ngx_http_mogilefs_module.o] Error 1
make[1]: Leaving directory `/usr/local/src/nginx-1.11.8'
make: *** [build] Error 2
解决办法:
我们这里打开
vim objs/Makefile
#去掉警告
-Werror
#编辑nginx配置文件
user nginx;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream trackers {
server 192.168.42.150:7001 weight=1;
server 192.168.42.151:7001 weight=1;
server 192.168.42.152:7001 weight=1;
check interval=1000 rise=2 fall=5 timeout=1000;
}
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
location /images/ {
mogilefs_tracker trackers;
mogilefs_domain images;
mogilefs_methods GET;
mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
浏览器访问:
http://192.168.42.150/images/2.jpg http://192.168.42.150/images/1.gif 都OK
此时我们的trackers其实只有一个,但我们在nginx配置文件中写了三个,所以我们需要把另外两个节点也要 安装tracker
scp /etc/mogilefs/mogilefsd.conf 192.168.42.151:/etc/mogilefs/ scp /etc/mogilefs/mogilefsd.conf 192.168.42.152:/etc/mogilefs/ 启动mogilefsd:(两台节点) service mogilefsd start
三个节点都能查看内容列表
[root@node1 conf]# moglistkeys --trackers=192.168.42.150:7001 --domain=images 1.gif 1.jpg 2.jpg [root@node1 conf]# moglistkeys --trackers=192.168.42.151:7001 --domain=images 1.gif 1.jpg 2.jpg [root@node1 conf]# moglistkeys --trackers=192.168.42.152:7001 --domain=images 1.gif 1.jpg 2.jpg
原创文章，作者：srayban，如若转载，请注明出处：http://www.178linux.com/78103
评论列表（1条）
不错，很详细，干货。