操作系统杀死进程排查.md

发表于

Linux上进程意外宕机有内存OOM操作系统杀死 或者 进程内存越界被杀死两种场景。本文介绍了如何定位进程是因为哪种原因被操作系统杀死。

操作系统杀死进程排查

Linux上进程意外宕机有内存OOM操作系统杀死 或者 进程内存越界被杀死两种场景。

1
2
3
4
5
6
# 通过dmesg和/var/logs/messages 获取信息
# /var/logs/messages 文件包含内核和用户空间程序消息，数据持久化保存(主要)
cat /var/log/messages |grep -i "dump\|kill"

# dmesg 获取内核消息，服务器重启后消失
dmesg -T

OOM Killer

1
2
3
4
5
6
7
8
9
10
11
12
13
# 通过以下命令发现OOM Killed
dmesg -T| egrep -i 'killed process'
grep -i 'killed process' /var/log/messages
total-vm:19058680kB, anon-rss:14610188kB, file-rss:76kB

# sar -r 查看内存使用率情况确认
Sar 命令查看历史硬件使用率情况
1. 每10分钟刷新一次信息
2. 可以实时监测，也可以查看过去的信息。
3. 存储资源放在/var/log/sa路径，通过文件创建日期来判断数据时间，
sar -r -f /var/log/sa/sa23

# 解决方案：内存扩容或swap扩大

Core Dumped

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
# ======= 现象 ============
# 操作系统日志中有dump日志
cat /var/log/messages |grep -i "dump\|kill"
# 或者有如下日志
Segmentation fault(Core dumped)
原因：program tries to access a memory address outside the programs range.

# ======= 解决方案 ============
# 查询coredump历史，获取异常PID
coredumpctl list
# 查询coredump dbp分析（重要）,info指令可以看到简介，gdb指令包含info，并可以直接分析coredump进行debug，若因操作系统原因未生成coredump，则无法查看更多。
coredumpctl gdb $pid
# Note: 也可以直接找到coredump文件，使用gdb -c <coredump-file> 进行分析
coredumpctl info $pid

# 参考示例
           PID: 16042 (npc)
           UID: 1000 (yzy)
           GID: 100 (users)
        Signal: 6 (ABRT)
     Timestamp: Sun 2018-06-17 10:52:15 +08 (7h ago)
  Command Line: ./npc
    Executable: /home/yzy/Repos/naive-pascal-compiler/cmake-build-debug/npc
 Control Group: /user.slice/user-1000.slice/session-c2.scope
          Unit: session-c2.scope
         Slice: user-1000.slice
       Session: c2
     Owner UID: 1000 (yzy)
       Boot ID: 6d54757c981742b6984d6d89a59ba2a3
    Machine ID: adbc8ab8645546ffb9ef66ec15b02081
      Hostname: yzy-arch
       Storage: /var/lib/systemd/coredump/core.npc.1000.6d54757c981742b6984d6d89a59ba2a3.16042.1529203935000000.lz4
       Message: Process 16042 (npc) of user 1000 dumped core.
                
                Stack trace of thread 16042:
                #0  0x00007f8df8f4986b raise (libc.so.6)
                #1  0x00007f8df8f3440e abort (libc.so.6)
                #2  0x00007f8df8f342e0 __assert_fail_base.cold.0 (libc.so.6)
                #3  0x00007f8df8f42112 __assert_fail (libc.so.6)
                #4  0x0000563a871686f3 n/a (/home/yzy/Repos/naive-pascal-compiler/cmake-build-debug/npc)
                #5  0x0000563a87166c3c n/a (/home/yzy/Repos/naive-pascal-compiler/cmake-build-debug/npc)
                #6  0x0000563a8716afe3 n/a (/home/yzy/Repos/naive-pascal-compiler/cmake-build-debug/npc)
                #7  0x0000563a8716582b n/a (/home/yzy/Repos/naive-pascal-compiler/cmake-build-debug/npc)
                #8  0x0000563a8715e499 n/a (/home/yzy/Repos/naive-pascal-compiler/cmake-build-debug/npc)
                #9  0x00007f8df8f3606b __libc_start_main (libc.so.6)
                #10 0x0000563a8715e36a n/a (/home/yzy/Repos/naive-pascal-compiler/cmake-build-debug/npc)

GNU gdb (GDB) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/yzy/Repos/naive-pascal-compiler/cmake-build-debug/npc...done.
[New LWP 16042]
Core was generated by `./npc'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f8df8f4986b in raise () from /usr/lib/libc.so.6
(gdb)

Coredump未生成

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# 声明coredump生成路径及格式
/proc/sys/kernel/core_pattern
/proc/sys/kernel/core_pipe_limit
/proc/sys/kernel/core_uses_pid

# coredump未生成问题解决
# 查看操作系统限制
ulimit -a 
# 若core file size为0会导致coredump文件无法生成，如下
# core file size 0
# 修改为unlimited，仅对当前会话生效
ulimit -c unlimited
# 永久生效，重新登录会话
vi /etc/security/limits.conf
* hard core unlimited