目录

排查

mongod占用内存高

情况说明

  1. 排查时应用对mongod的写入操作已经停止
  2. 内存占用为1-2G

# top 信息
top - 17:14:15 up 277 days,  2:45,  2 users,  load average: 1.11, 1.31, 1.41
Tasks: 226 total,   1 running, 225 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.3 us,  0.2 sy,  0.0 ni, 96.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 33034460 total, 24934168 free,  4478848 used,  3621444 buff/cache
KiB Swap:  8257532 total,  6803968 free,  1453564 used. 27291200 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                     
37785 mongod    20   0 2770940   1.1g   5492 S   3.2  3.5  58111:28 mongod                                                                      
 6831 root      20   0 7930020 911192  14256 S   1.0  2.8   2:02.10 java                                                                        
 6765 root      20   0 6131528 523936  16452 S   0.3  1.6   0:28.81 java                                                                        
 6660 root      20   0   11.5g 443000  13532 S   0.0  1.3   0:19.36 java                                                                        
10089 root      20   0 6981044 323508  14720 S   0.0  1.0   0:12.48 java                                                                        
40742 mysql     20   0 7346416 179360   3740 S   0.7  0.5   3887:53 mysqld                                                                      
 4406 root      20   0 1010360  67092  25972 S   0.0  0.2   0:02.12 dockerd                                                                     
 3691 root      20   0  684476  63604  33292 S   1.7  0.2   1:15.36 rsyslogd
配置文件 
[root@centos77 ~]# cat mongod.conf
net: {port: 27018}
operationProfiling: {mode: slowOp, slowOpThresholdMs: 2000}
processManagement: {fork: true, pidFilePath: /var/mongodb/mongod.pid }
#replication: {oplogSizeMB: 256, replSetName: mg163-1}
#security: {authorization: enabled, keyFile: /var/mongodb/keyfile}
storage:
  dbPath: /var/mongodb/data
  directoryPerDB: true
  engine: wiredTiger
  journal: {enabled: true}
  syncPeriodSecs: 60
  wiredTiger:
    collectionConfig: {blockCompressor: snappy}
    engineConfig: {cacheSizeGB: 1, directoryForIndexes: true, journalCompressor: snappy,statisticsLogDelaySecs: 0}
    indexConfig: {prefixCompression: true}
systemLog: {destination: file, logAppend: true, logRotate: rename, path: /var/mongodb/logs/mongod.log}
security: {authorization: enabled}
insert query update delete 增删改查统计 通过mongostat查看insert query update delete 增删改查已完全停止，但注意到used（缓存占用高）

used = wiredTiger Storage Engine当前在用的缓存与配置的最大缓存的比值。
used 参考官方说明： mongostat used Only for WiredTiger Storage Engine. The percentage of the WiredTiger cache that is in use, calculated by wiredTiger.cache.bytes currently in the cache / wiredTiger.cache.maximum bytes configured. 

[root@centos77 ~]# mongostat -u username-p password --authenticationDatabase admin
insert query update delete getmore command dirty  used flushes vsize   res qrw arw net_in net_out conn                time
    *0    *0     *0     *0       0     2|0 15.2% 78.0%       0 2.64G 1.10G 0|0 1|0   159b   48.1k   14 Mar  7 17:13:03.247
    *0    *0     *0     *0       0     2|0 14.4% 78.0%       1 2.64G 1.10G 0|0 1|0   158b   47.6k   14 Mar  7 17:13:04.247
    *0    *0     *0     *0       0     1|0 12.9% 78.0%       0 2.64G 1.10G 0|0 1|0   157b   47.6k   14 Mar  7 17:13:05.247
    *0    *0     *0     *0       0     1|0 10.7% 78.0%       0 2.64G 1.10G 0|0 1|0   157b   47.6k   14 Mar  7 17:13:06.248
    *0    *0     *0     *0       0     2|0  8.8% 78.0%       0 2.64G 1.10G 0|0 1|0   158b   47.7k   14 Mar  7 17:13:07.247
    *0    *0     *0     *0       0     2|0  6.6% 78.0%       0 2.64G 1.10G 0|0 1|0   158b   47.6k   14 Mar  7 17:13:08.247
    *0    *0     *0     *0       0     1|0  5.6% 78.0%       0 2.64G 1.10G 0|0 1|0   157b   47.6k   14 Mar  7 17:13:09.247
    *0    *0     *0     *0       0     2|0  4.9% 78.0%       0 2.64G 1.10G 0|0 1|0   215b   47.8k   14 Mar  7 17:13:10.248
    *0    *0     *0     *0       0     4|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   274b   48.1k   14 Mar  7 17:13:11.247
    *0    *0     *0     *0       0     3|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   216b   47.9k   14 Mar  7 17:13:12.247
insert query update delete getmore command dirty  used flushes vsize   res qrw arw net_in net_out conn                time
    *0    *0     *0     *0       0     1|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   157b   47.6k   14 Mar  7 17:13:13.247
    *0    *0     *0     *0       0     1|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   157b   47.6k   14 Mar  7 17:13:14.248
    *0    *0     *0     *0       0     2|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   158b   47.7k   14 Mar  7 17:13:15.247
    *0    *0     *0     *0       0     2|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   158b   47.6k   14 Mar  7 17:13:16.247
    *0    *0     *0     *0       0     1|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   157b   47.6k   14 Mar  7 17:13:17.247
    *0    *0     *0     *0       0     6|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   712b   47.8k   15 Mar  7 17:13:18.254
    *0    *0     *0     *0       0     2|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   159b   48.0k   15 Mar  7 17:13:19.247
    *0    *0     *0     *0       0     2|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   215b   47.8k   15 Mar  7 17:13:20.247
    *0    *0     *0     *0       0     3|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   273b   48.0k   15 Mar  7 17:13:21.247
    *0    *0     *0     *0       0     3|0  1.8% 78.0%       0 2.64G 1.10G 0|0 1|0   216b   47.8k   15 Mar  7 17:13:22.247
^C2020-03-07T17:13:22.868+0800  signal 'interrupt' received; forcefully terminating

重启服务mongod后


操作有效
[root@centos77 ~]# mongostat -u username-p password --authenticationDatabase admin
insert query update delete getmore command dirty  used flushes vsize  res qrw arw net_in net_out conn                time
    *0    *0     *0     *0       0     2|0  5.0% 14.1%       0 1.24G 347M 0|0 1|0   160b   47.5k   13 Mar  7 17:36:53.102
    *0    *0     *0     *0       0     1|0  5.0% 14.1%       0 1.24G 347M 0|0 1|0   157b   46.9k   13 Mar  7 17:36:54.102
    *0    *0     *0     *0       0     1|0  5.0% 14.1%       0 1.24G 347M 0|0 1|0   157b   46.9k   13 Mar  7 17:36:55.102
    32    *0     *0     *0       0     1|0  5.1% 14.2%       0 1.24G 347M 0|0 1|0  6.95k   48.9k   13 Mar  7 17:36:56.103
    *0    *0     *0     *0       0     2|0  5.1% 14.2%       0 1.24G 347M 0|0 1|0   158b   47.0k   13 Mar  7 17:36:57.102
    *0    *0     *0     *0       0     4|0  5.1% 14.2%       0 1.24G 348M 0|0 1|0   274b   47.3k   13 Mar  7 17:36:58.102
    *0    *0     *0     *0       0     2|0  5.1% 14.2%       0 1.24G 348M 0|0 1|0   158b   46.9k   13 Mar  7 17:36:59.101
     4    *0     *0     *0       0     9|0  5.1% 14.2%       0 1.24G 348M 0|0 2|5  1.97k   49.0k   14 Mar  7 17:37:00.103
   230    *0     *0     *0       0     4|0  6.4% 14.4%       0 1.25G 354M 0|0 2|0  87.2k   84.9k   14 Mar  7 17:37:01.102
    27    *0     *0     *0       0  1879|0  6.3% 14.3%       0 1.25G 357M 0|0 1|0   387k    833k   14 Mar  7 17:37:02.101

  1. 查看du -sh /var/mongodb/data 目录大小，文件空间占用比较大。由于当前mongod数据仅为日志，考虑对数据库初始化。
  2. 备份data目录为data_bakup：重命名data目录，mkdir data , chown -R mongod.mongod data 初始化data目录
  3. 重新添加mongod登陆用户：
    • 因为之前的用户表信息存在于data_bakup中了。
    • 注意，先注释配置文件中的security: {authorization: enabled} ，重启mongod服务即可初始化data目录，
    • 重新创建用户信息。如下：
  4. 添加好用户后，取消注释security: {authorization: enabled}，重启服务。即可通过指定用户登陆

use admin;
db.createUser(
    {
    user:"username",
    pwd:"passwork",
    roles: [
        {role:"clusterAdmin",db:"admin"},
        {role:"readWriteAnyDatabase",db:"admin"}
    ]
    }
);

初始化后服务状态恢复

[root@centos77 ~]# mongostat -u username-p password --authenticationDatabase admin
insert query update delete getmore command dirty used flushes vsize   res qrw arw net_in net_out conn                time
    *0    *0     *0     *0       0    16|0  0.0% 0.1%       0  966M 48.0M 0|0 1|0  1.56k   51.9k    4 Mar  7 17:43:21.646
    *0    *0     *0     *0       0     9|0  0.0% 0.1%       0  968M 48.0M 0|0 1|0   851b   49.1k    5 Mar  7 17:43:22.646
    *0    *0     *0     *0       0     1|0  0.0% 0.1%       0  968M 48.0M 0|0 1|0   157b   46.8k    5 Mar  7 17:43:23.646
    *0    *0     *0     *0       0     2|0  0.0% 0.1%       0  968M 48.0M 0|0 1|0   158b   46.9k    5 Mar  7 17:43:24.645
    *0    *0     *0     *0       0     1|0  0.0% 0.1%       0  968M 48.0M 0|0 1|0   157b   46.9k    5 Mar  7 17:43:25.646
    *0    *0     *0     *0       0     1|0  0.0% 0.1%       0  968M 48.0M 0|0 1|0   157b   46.9k    5 Mar  7 17:43:26.646
    *0    *0     *0     *0       0     2|0  0.0% 0.1%       0  968M 49.0M 0|0 1|0   158b   46.9k    5 Mar  7 17:43:27.645
    *0    *0     *0     *0       0     1|0  0.0% 0.1%       0  968M 49.0M 0|0 1|0   157b   46.8k    5 Mar  7 17:43:28.646
    *0    *0     *0     *0       0     2|0  0.0% 0.1%       0  968M 49.0M 0|0 1|0   158b   46.9k    5 Mar  7 17:43:29.645

对比服务器的运行状态

正常的16G内存服务器 https://www.lagou.com/lgeduarticle/28686.html 

top - 17:24:04 up 8 days,  6:24,  1 user,  load average: 0.00, 0.02, 0.05
Tasks: 245 total,   1 running, 244 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  0.7 sy,  0.0 ni, 98.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16227724 total,  3019468 free,  7694916 used,  5513340 buff/cache
KiB Swap: 33554428 total, 33510140 free,    44288 used.  7929068 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 61474 root      20   0 9956.3m   1.8g   7868 S   0.0 11.8  19:56.24 java
113133 root      20   0 8234100   1.2g   7856 S   0.3  8.0  86:54.88 java
 58483 mysql     20   0   11.6g   1.1g   7000 S   0.0  7.3  31:22.69 mysqld
 63714 root      20   0 6934620 669344   8496 S   0.3  4.1  22:38.95 java
  2242 root      20   0 1221496 220884   7536 S   1.0  1.4 103:41.67 mongod
 62997 root      20   0 6399300 606504   9532 S   0.3  3.7  28:47.58 java
 62394 root      20   0 7921912 494676   8380 S   0.0  3.0  25:39.01 java
 61785 root      20   0 8265804 383536   5384 S   0.0  2.4  15:05.47 java
 64000 root      20   0 5171360 244916   6452 S   0.0  1.5  13:12.04 java


[root@centos77 ~]# mongostat -u username-p password --authenticationDatabase admin
insert query update delete getmore command dirty  used flushes vsize  res qrw arw net_in net_out conn                time
    *0    *0     *0     *0       0     2|0  0.0% 16.6%       0 1.16G 215M 0|0 1|0   160b   47.9k   12 Mar  7 17:23:26.059
    *0    *0     *0     *0       0     3|0  0.0% 16.6%       0 1.16G 215M 0|0 1|0   273b   47.4k   12 Mar  7 17:23:27.060
    *0    *0     *0     *0       0     1|0  0.0% 16.6%       0 1.16G 215M 0|0 1|0   157b   47.0k   12 Mar  7 17:23:28.061
    *0    *0     *0     *0       0     2|0  0.0% 16.6%       0 1.16G 215M 0|0 1|0   158b   47.1k   12 Mar  7 17:23:29.059
    *0    *0     *0     *0       0     2|0  0.0% 16.6%       0 1.16G 215M 0|0 1|0   158b   47.1k   12 Mar  7 17:23:30.059
    *0    *0     *0     *0       0     2|0  0.0% 16.6%       0 1.16G 215M 0|0 1|0   158b   47.1k   12 Mar  7 17:23:31.059
    *0    *0     *0     *0       0     1|0  0.0% 16.6%       0 1.16G 215M 0|0 1|0   157b   47.0k   12 Mar  7 17:23:32.059
    *0    *0     *0     *0       0     1|0  0.0% 16.6%       0 1.16G 215M 0|0 1|0   157b   47.0k   12 Mar  7 17:23:33.060
    *0    *0     *0     *0       0     3|0  0.0% 16.6%       0 1.16G 215M 0|0 1|0   273b   47.4k   12 Mar  7 17:23:34.062