小强哥博客

小强哥,小强哥博客,技术大咖

Mongo集群、分片、高可用(一)

本文主要记一下如何配置Mongo集群、分片、高可用,Mongo版本是基于Mongo3.4进行测试,涉及到的技术有些自己总结,有些是来自网络整理,如有不正确的地方欢迎指正。

概念

分片(sharding)是指将数据库拆分,将其分散在不同的机器上的过程。将数据分散到不同的机器上,不需要功能强大的服务器就可以存储更多的数据和处理更大的负载。基本思想就是将集合切成小块,这些块分散到若干片里,每个片只负责总数据的一部分,最后通过一个均衡器来对各个分片进行均衡(数据迁移)。通过一个名为mongos的路由进程进行操作,mongos知道数据和片的对应关系(通过配置服务器)。大部分使用场景都是解决磁盘空间的问题,对于写入有可能会变差(+里面的说明+),查询则尽量避免跨分片查询。使用分片的时机:

  • 机器的磁盘不够用了。使用分片解决磁盘空间的问题。
  • 单个mongod已经不能满足写数据的性能要求。通过分片让写压力分散到各个分片上面,使用分片服务器自身的资源。
  • 想把大量数据放到内存里提高性能。和上面一样,通过分片使用分片服务器自身的资源。

部署安装

如下所示,部署安装架构图,

image

在搭建分片之前,先了解下分片中各个角色的作用。

  • 配置服务器(configsvr)。是一个独立的mongod进程,保存集群和分片的元数据,即各分片包含了哪些数据的信息。最先开始建立,启用日志功能。像启动普通的mongod一样启动配置服务器,指定configsvr选项。不需要太多的空间和资源,配置服务器的1KB空间相当于真是数据的200MB。保存的只是数据的分布表。当服务不可用,则变成只读,无法分块、迁移数据。
  • 路由服务器(mongos)。起到一个路由的功能,供程序连接。本身不保存数据,在启动时从配置服务器加载集群信息,开启mongos进程需要知道配置服务器的地址,指定configdb选项。
  • 分片服务器(shardsvr)。是一个独立普通的mongod进程,保存数据信息。可以是一个副本集也可以是单独的一台服务器。

首先准备4台机器,如下,

172.17.0.2(configsvr,shardsvr)
172.17.0.3(configsvr,shardsvr)
172.17.0.4(configsvr,shardsvr)
172.17.0.5(mongos)

首先,在172.17.0.2,172.17.0.3,172.17.0.4,172.17.0.5分别创建mongo数据文件夹,如下命令,

mkdir -p /root/mongo/config/data \
&& mkdir -p /root/mongo/config/log \
&& mkdir -p /root/mongo/shard/data1 \
&& mkdir -p /root/mongo/shard/log1 \
&& mkdir -p /root/mongo/shard/data2 \
&& mkdir -p /root/mongo/shard/log2 \
&& mkdir -p /root/mongo/shard/data3 \
&& mkdir -p /root/mongo/shard/log3 \
&& mkdir -p /root/mongo/mongos/log

如下目录结构,

[root@27e6d7264fba ~]# tree mongo/
mongo/
|-- config
|   |-- data
|   `-- log
|-- mongos
|   `-- log
`-- shard
    |-- data1
    |-- data2
    |-- data3
    |-- log1
    |-- log2
    `-- log3

ConfigSVR服务

在172.17.0.2,172.17.0.3,172.17.0.4中分别启动configsvr服务,如下命令,

mongod --configsvr --replSet configRepl  --port 2000 --bind_ip 0.0.0.0  --dbpath="/root/mongo/config/data" --logpath="/root/mongo/config/log/config.log" --fork --logappend

执行结果如下所示,

[root@27e6d7264fba ~]# mongod --configsvr --replSet configRepl  --port 2000 --bind_ip 0.0.0.0  --dbpath="/root/mongo/config/data" --logpath="/root/mongo/config/log/config.log" --fork --logappend
about to fork child process, waiting until server is ready for connections.
forked process: 53
child process started successfully, parent exiting

这里有个小问题,如果没有执行前一步的创建mong数据文件夹,执行这条命令会报错,mongod命令不会自动创建文件夹。

在mongo3.2以后规定,configsvr也必须是副本集模式,接着对configsvr创建副本集。在172.17.0.2,172.17.0.3,172.17.0.4中任何一台服务器执行mongo 127.0.0.1:2000登录并执行创建副本集命令,如下,

# 创建一个config变量
config={
    "_id":"configRepl",
    "members":[
        {"_id":0,"host":"172.17.0.2:2000"},
        {"_id":1,"host":"172.17.0.3:2000"},
        {"_id":2,"host":"172.17.0.4:2000"},
    ]
}

# 初始化副本集
rs.initiate(config)

如下所示执行结果,

[root@27e6d7264fba ~]# mongo 127.0.0.1:2000
MongoDB shell version v3.4.9
connecting to: 127.0.0.1:2000
MongoDB server version: 3.4.9
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
Server has startup warnings: 
2017-10-24T09:23:57.926+0000 I CONTROL  [initandlisten] 
2017-10-24T09:23:57.927+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-10-24T09:23:57.927+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-10-24T09:23:57.927+0000 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2017-10-24T09:23:57.927+0000 I CONTROL  [initandlisten] 
> config={
...     "_id":"configRepl",
...     "members":[
...         {"_id":0,"host":"172.17.0.2:2000"},
...         {"_id":1,"host":"172.17.0.3:2000"},
...         {"_id":2,"host":"172.17.0.4:2000"},
...     ]
... }
{
        "_id" : "configRepl",
        "members" : [
                {
                        "_id" : 0,
                        "host" : "172.17.0.2:2000"
                },
                {
                        "_id" : 1,
                        "host" : "172.17.0.3:2000"
                },
                {
                        "_id" : 2,
                        "host" : "172.17.0.4:2000"
                }
        ]
}
> rs.initiate(config)
{ "ok" : 1 }
configRepl:SECONDARY>

shardSVR服务

mongo3.2以后的版本规定shard服务同样需要配置副本集,为了提高高可用。

分别在172.17.0.2,172.17.0.3,172.17.0.4中执行如下命令创建分片shardR1pl1副本集服务,如下命令,

mongod --shardsvr --replSet shardR1pl1 --port 2001 --bind_ip 0.0.0.0 --logpath /root/mongo/shard/log1/shard.log --dbpath /root/mongo/shard/data1 --fork --logappend

执行结果如下,

[root@27e6d7264fba ~]# mongod --shardsvr --replSet shardR1pl1 --port 2001 --bind_ip 0.0.0.0 --logpath /root/mongo/shard/log1/shard.log --dbpath /root/mongo/shard/data1 --fork --logappend
about to fork child process, waiting until server is ready for connections.
forked process: 148
child process started successfully, parent exiting

同样的,分别在172.17.0.2,172.17.0.3,172.17.0.4中执行如下命令创建分片shardR1pl2副本集服务,如下命令,

mongod --shardsvr --replSet shardR1pl2 --port 2002 --bind_ip 0.0.0.0 --logpath /root/mongo/shard/log2/shard.log --dbpath /root/mongo/shard/data2 --fork --logappend

同样的,分别在172.17.0.2,172.17.0.3,172.17.0.4中执行如下命令创建分片shardR1pl3副本集服务,如下命令,

mongod --shardsvr --replSet shardR1pl3 --port 2003 --bind_ip 0.0.0.0 --logpath /root/mongo/shard/log3/shard.log --dbpath /root/mongo/shard/data3 --fork --logappend

可以通过netstat -nlp查看启动结果,如下,

[root@69a43ae0c6da /]# netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:2000            0.0.0.0:*               LISTEN      44/mongod           
tcp        0      0 0.0.0.0:2001            0.0.0.0:*               LISTEN      130/mongod          
tcp        0      0 0.0.0.0:2002            0.0.0.0:*               LISTEN      157/mongod          
tcp        0      0 0.0.0.0:2003            0.0.0.0:*               LISTEN      184/mongod     

在任何一台服务器上执行mongo 127.0.0.1:2001命令登录初始化shardR1pl1副本集,如下,

config={
    "_id":"shardR1pl1",
    "members":[
        {"_id":0,"host":"172.17.0.2:2001"},
        {"_id":1,"host":"172.17.0.3:2001"},
        {"_id":2,"host":"172.17.0.4:2001"},
    ]
}

rs.initiate(config)

同样的,在任何一台服务器上执行mongo 127.0.0.1:2002命令登录初始化shardR1pl2副本集,如下,

config={
    "_id":"shardR1pl2",
    "members":[
        {"_id":0,"host":"172.17.0.2:2002"},
        {"_id":1,"host":"172.17.0.3:2002"},
        {"_id":2,"host":"172.17.0.4:2002"},
    ]
}

rs.initiate(config)

同样的,在任何一台服务器上执行mongo 127.0.0.1:2003命令登录初始化shardR1pl3副本集,如下,

config={
    "_id":"shardR1pl3",
    "members":[
        {"_id":0,"host":"172.17.0.2:2003"},
        {"_id":1,"host":"172.17.0.3:2003"},
        {"_id":2,"host":"172.17.0.4:2003"},
    ]
}

rs.initiate(config)

Mongos

登录172.17.0.5启动mongos服务,如下命令,

mongos --configdb configRepl/172.17.0.2:2000,172.17.0.3:2000,172.17.0.4:2000 --logpath="/root/mongo/mongos/log/mongos.log" --port 4000 --fork --logappend

这里的--configdb参数需要指定configSVR副本集成员。

使用mongo 127.0.0.1:4000登录到mongos,如下,

[root@be9cf8c4ff08 ~]# mongo 127.0.0.1:4000
MongoDB shell version v3.4.9
connecting to: 127.0.0.1:4000
MongoDB server version: 3.4.9
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
Server has startup warnings: 
2017-10-24T09:53:20.612+0000 I CONTROL  [main] 
2017-10-24T09:53:20.612+0000 I CONTROL  [main] ** WARNING: Access control is not enabled for the database.
2017-10-24T09:53:20.612+0000 I CONTROL  [main] **          Read and write access to data and configuration is unrestricted.
2017-10-24T09:53:20.612+0000 I CONTROL  [main] ** WARNING: You are running this process as the root user, which is not recommended.
2017-10-24T09:53:20.612+0000 I CONTROL  [main] 
mongos> 

接着,在命令行添加副本集分片,如下命令,

sh.addShard("shardR1pl1/172.17.0.2:2001,172.17.0.3:2001,172.17.0.4:2001")
sh.addShard("shardR1pl2/172.17.0.2:2002,172.17.0.3:2002,172.17.0.4:2002")
sh.addShard("shardR1pl3/172.17.0.2:2003,172.17.0.3:2003,172.17.0.4:2003")

通过sh.status()可以查看副本成员,如下,

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("59ef0790b29da6dd002232dc")
}
  shards:
        {  "_id" : "shardR1pl1",  "host" : "shardR1pl1/172.17.0.2:2001,172.17.0.3:2001,172.17.0.4:2001",  "state" : 1 }
        {  "_id" : "shardR1pl2",  "host" : "shardR1pl2/172.17.0.2:2002,172.17.0.3:2002,172.17.0.4:2002",  "state" : 1 }
        {  "_id" : "shardR1pl3",  "host" : "shardR1pl3/172.17.0.2:2003,172.17.0.3:2003,172.17.0.4:2003",  "state" : 1 }
  active mongoses:
        "3.4.9" : 1
 autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
                Balancer lock taken at Tue Oct 24 2017 09:27:50 GMT+0000 (UTC) by ConfigServer:Balancer
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                No recent migrations
  databases:

到此,mongo分片已经配置完成,接着就启用分片。

启用副本集

mongo分片需要指定一个数据库,通过如下命令指定,

mongos> sh.enableSharding("quezon")
{ "ok" : 1 }

接着添加需要设置分片的文档,如下命令,

mongos> sh.shardCollection("quezon.student",{"_id":"hashed"})
{ "collectionsharded" : "quezon.student", "ok" : 1 }

接着执行测试10W条查看测试结果,如下命令,

for(var i=0;i<100000;i++){db.student.insert({"name":"zhangsan_"+i,"age":i*10})}

使用sh.status可以查看数据在每个片区的分布情况,如下,

mongos> db.student.stats()
{
        "sharded" : true,
        "capped" : false,
        "ns" : "quezon.student",
        "count" : 100000,
        "size" : 5988890,
 
        "shards" : {
                "shardR1pl1" : {
                        "ns" : "quezon.student",
                        "size" : 2003330,
                        "count" : 33450,
                        "avgObjSize" : 59,
                        "storageSize" : 688128,
                        "capped" : false,
                        
                        "nindexes" : 2,
                        "totalIndexSize" : 1249280,
                        "indexSizes" : {
                                "_id_" : 372736,
                                "_id_hashed" : 876544
                        },
                        "ok" : 1
                },
                "shardR1pl2" : {
                        "ns" : "quezon.student",
                        "size" : 1989415,
                        "count" : 33219,
                        "avgObjSize" : 59,
                        "storageSize" : 667648,
                        "capped" : false,
                        
                        "nindexes" : 2,
                        "totalIndexSize" : 1351680,
                        "indexSizes" : {
                                "_id_" : 376832,
                                "_id_hashed" : 974848
                        },
                        "ok" : 1
                },
                "shardR1pl3" : {
                        "ns" : "quezon.student",
                        "size" : 1996145,
                        "count" : 33331,
                        "avgObjSize" : 59,
                        "storageSize" : 671744,
                        "capped" : false,
                        
                        "nindexes" : 2,
                        "totalIndexSize" : 1380352,
                        "indexSizes" : {
                                "_id_" : 339968,
                                "_id_hashed" : 1040384
                        },
                        "ok" : 1
                }
        },
        "ok" : 1
}

数据分布基本上实现平均分布。

完。