heartbeat 및 mon-tool 로그 분석

[시스템로그]

 

Mon [stop | start]

- 시스템 로그 남지 않음

- 시스템 로그는 하트비트는 로컬만 하트비트 시작 / 정지 로그 남음

 

시스템 로그 하트비트 스탑 [마스터]

Apr 9 10:27:59 sulinux1 avahi-daemon[1933]: Withdrawing address record for 115.68.26.20 on eth0.
Apr 9 10:28:02 sulinux1 logd: [24861]: info: Waiting for pid=23934 to exit
Apr 9 10:28:02 sulinux1 logd: [23935]: info: logd_term_write_action: received SIGTERM
Apr 9 10:28:02 sulinux1 logd: [23935]: info: Exiting write process
Apr 9 10:28:03 sulinux1 logd: [24861]: info: Pid 23934 exited

 

 

시스템 로그 하트비트 스타트 [마스터]

Apr 9 10:28:53 sulinux1 logd: [24904]: info: logd started with default configuration.
Apr 9 10:28:53 sulinux1 logd: [24905]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Apr 9 10:28:53 sulinux1 logd: [24904]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Apr 9 10:28:57 sulinux1 avahi-daemon[1933]: Registering new address record for 115.68.26.20 on eth0.
Apr 9 10:28:57 sulinux1 avahi-daemon[1933]: Withdrawing address record for 115.68.26.20 on eth0.
Apr 9 10:28:57 sulinux1 avahi-daemon[1933]: Registering new address record for 115.68.26.20 on eth0.

 

 

시스템 로그 하트비트 스탑 [슬레이브]

Apr 9 11:29:33 sulinux2 logd: [19737]: info: Waiting for pid=22656 to exit
Apr 9 11:29:33 sulinux2 logd: [22657]: info: logd_term_write_action: received SIGTERM
Apr 9 11:29:33 sulinux2 logd: [22657]: info: Exiting write process
Apr 9 11:29:34 sulinux2 logd: [19737]: info: Pid 22656 exited

 

 

시스템 로그 하트비트 스타트 [슬레이브]

Apr 9 11:32:17 sulinux2 logd: [19995]: info: logd started with default configuration.
Apr 9 11:32:17 sulinux2 logd: [19996]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Apr 9 11:32:17 sulinux2 logd: [19995]: info: G_main_add_SignalHandler: Added signal handler for signal 15

 

 

하트비트 종료시 마스터

[root@sulinux1 ~]# cat /var/log/ha-log

 

 

하트비트 프로세스 종료
heartbeat[6139]: 2010/04/09_10:05:43 info: Heartbeat shutdown in progress. (6139)

 

 

자원을 넘겨줌
heartbeat[23557]: 2010/04/09_10:05:43 info: Giving up all HA resources.

ResourceManager[23570]: 2010/04/09_10:05:43 info: Releasing resource group: sulinux1 115.68.26.20
ResourceManager[23570]: 2010/04/09_10:05:43 info: Running /etc/ha.d/resource.d/IPaddr 115.68.26.20 stop

 

 

VIP 종료
IPaddr[23624]: 2010/04/09_10:05:43 INFO: ifconfig eth0:0 down
IPaddr[23607]: 2010/04/09_10:05:43 INFO: Success

 

 

모든 리소스 넘겨짐
heartbeat[23557]: 2010/04/09_10:05:43 info: All HA resources relinquished.

heartbeat[6139]: 2010/04/09_10:05:43 WARN: 1 lost packet(s) for [sulinux2] [107080:107082]
heartbeat[6139]: 2010/04/09_10:05:43 info: No pkts missing from sulinux2!
heartbeat[6139]: 2010/04/09_10:05:45 info: killing HBFIFO process 6142 with signal 15
heartbeat[6139]: 2010/04/09_10:05:45 info: killing HBWRITE process 6143 with signal 15
heartbeat[6139]: 2010/04/09_10:05:45 info: killing HBREAD process 6144 with signal 15
heartbeat[6139]: 2010/04/09_10:05:45 info: Core process 6142 exited. 3 remaining
heartbeat[6139]: 2010/04/09_10:05:45 info: Core process 6143 exited. 2 remaining
heartbeat[6139]: 2010/04/09_10:05:45 info: Core process 6144 exited. 1 remaining

 

 

하트비트 완전 종료
heartbeat[6139]: 2010/04/09_10:05:45 info: sulinux1 Heartbeat shutdown complete.

 

하트비트 종료시 슬레이브

heartbeat[22737]: 2010/04/09_09:59:26 info: Received shutdown notice from 'sulinux1'.
heartbeat[22737]: 2010/04/09_09:59:26 info: Resources being acquired from sulinux1.
heartbeat[18307]: 2010/04/09_09:59:26 info: acquire local HA resources (standby).
heartbeat[18308]: 2010/04/09_09:59:26 info: No local resources [/usr/share/heartbeat/ResourceManager listkeys sulinux2] to acquire.
heartbeat[18307]: 2010/04/09_09:59:26 info: local HA resource acquisition completed (standby).
heartbeat[22737]: 2010/04/09_09:59:26 info: Standby resource acquisition done [foreign].
harc[18333]: 2010/04/09_09:59:26 info: Running /etc/ha.d/rc.d/status status
mach_down[18348]: 2010/04/09_09:59:26 info: Taking over resource group 115.68.26.20
ResourceManager[18373]: 2010/04/09_09:59:26 info: Acquiring resource group: sulinux1 115.68.26.20
IPaddr[18399]: 2010/04/09_09:59:26 INFO: Resource is stopped
ResourceManager[18373]: 2010/04/09_09:59:26 info: Running /etc/ha.d/resource.d/IPaddr 115.68.26.20 start
IPaddr[18472]: 2010/04/09_09:59:26 INFO: Using calculated nic for 115.68.26.20: eth0
IPaddr[18472]: 2010/04/09_09:59:26 INFO: Using calculated netmask for 115.68.26.20: 255.255.255.224
IPaddr[18472]: 2010/04/09_09:59:26 INFO: eval ifconfig eth0:0 115.68.26.20 netmask 255.255.255.224 broadcast 115.68.26.31
IPaddr[18455]: 2010/04/09_09:59:26 INFO: Success
mach_down[18348]: 2010/04/09_09:59:26 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[18348]: 2010/04/09_09:59:26 info: mach_down takeover complete for node sulinux1.
heartbeat[22737]: 2010/04/09_09:59:26 info: mach_down takeover complete.
heartbeat[22737]: 2010/04/09_09:59:34 WARN: node sulinux1: is dead
heartbeat[22737]: 2010/04/09_09:59:34 info: Dead node sulinux1 gave up resources.
heartbeat[22737]: 2010/04/09_09:59:34 info: Link sulinux1:eth0 dead.

 

 

마스터 하트비트 종료후 다시 시작시 마스터쪽 로그

[root@sulinux1 ~]# cat /var/log/ha-log
heartbeat[24014]: 2010/04/09_10:13:40 info: Version 2 support: false
heartbeat[24014]: 2010/04/09_10:13:40 WARN: Logging daemon is disabled --enabling logging daemon is recommended
heartbeat[24014]: 2010/04/09_10:13:40 info: **************************
heartbeat[24014]: 2010/04/09_10:13:40 info: Configuration validated. Starting heartbeat 2.1.3
heartbeat[24015]: 2010/04/09_10:13:40 info: heartbeat: version 2.1.3
heartbeat[24015]: 2010/04/09_10:13:41 info: Heartbeat generation: 1268197589
heartbeat[24015]: 2010/04/09_10:13:41 info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0
heartbeat[24015]: 2010/04/09_10:13:41 info: glib: ucast: bound send socket to device: eth0
heartbeat[24015]: 2010/04/09_10:13:41 info: glib: ucast: bound receive socket to device: eth0
heartbeat[24015]: 2010/04/09_10:13:41 info: glib: ucast: started on port 694 interface eth0 to 115.68.26.12
heartbeat[24015]: 2010/04/09_10:13:41 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[24015]: 2010/04/09_10:13:41 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[24015]: 2010/04/09_10:13:41 info: G_main_add_SignalHandler: Added signal handler for signal 17
heartbeat[24015]: 2010/04/09_10:13:41 info: Local status now set to: 'up'
heartbeat[24015]: 2010/04/09_10:13:43 info: Link sulinux2:eth0 up.
heartbeat[24015]: 2010/04/09_10:13:43 info: Status update for node sulinux2: status active
harc[24024]: 2010/04/09_10:13:43 info: Running /etc/ha.d/rc.d/status status
heartbeat[24015]: 2010/04/09_10:13:43 info: Comm_now_up(): updating status to active
heartbeat[24015]: 2010/04/09_10:13:43 info: Local status now set to: 'active'
heartbeat[24015]: 2010/04/09_10:13:44 info: remote resource transition completed.
heartbeat[24015]: 2010/04/09_10:13:44 info: remote resource transition completed.
heartbeat[24015]: 2010/04/09_10:13:44 info: Local Resource acquisition completed. (none)
heartbeat[24015]: 2010/04/09_10:13:44 info: sulinux2 wants to go standby [foreign]
heartbeat[24015]: 2010/04/09_10:13:45 info: standby: acquire [foreign] resources from sulinux2
heartbeat[24042]: 2010/04/09_10:13:45 info: acquire local HA resources (standby).
ResourceManager[24055]: 2010/04/09_10:13:45 info: Acquiring resource group: sulinux1 115.68.26.20
IPaddr[24081]: 2010/04/09_10:13:45 INFO: Resource is stopped
ResourceManager[24055]: 2010/04/09_10:13:45 info: Running /etc/ha.d/resource.d/IPaddr 115.68.26.20 start
IPaddr[24154]: 2010/04/09_10:13:45 INFO: Using calculated nic for 115.68.26.20: eth0
IPaddr[24154]: 2010/04/09_10:13:45 INFO: Using calculated netmask for 115.68.26.20: 255.255.255.224
IPaddr[24154]: 2010/04/09_10:13:45 INFO: eval ifconfig eth0:0 115.68.26.20 netmask 255.255.255.224 broadcast 115.68.26.31
IPaddr[24137]: 2010/04/09_10:13:45 INFO: Success
heartbeat[24042]: 2010/04/09_10:13:45 info: local HA resource acquisition completed (standby).
heartbeat[24015]: 2010/04/09_10:13:45 info: Standby resource acquisition done [foreign].
heartbeat[24015]: 2010/04/09_10:13:45 info: Initial resource acquisition complete (auto_failback)
heartbeat[24015]: 2010/04/09_10:13:46 info: remote resource transition completed.

 

 

마스터 하트비트 종료후 다시 시작시 슬레이브쪽 로그

[root@sulinux2 ~]# cat /var/log/ha-log
heartbeat[22737]: 2010/04/09_10:07:24 info: Heartbeat restart on node sulinux1
heartbeat[22737]: 2010/04/09_10:07:24 info: Link sulinux1:eth0 up.
heartbeat[22737]: 2010/04/09_10:07:24 info: Status update for node sulinux1: status init
heartbeat[22737]: 2010/04/09_10:07:24 info: Status update for node sulinux1: status up
harc[18590]: 2010/04/09_10:07:24 info: Running /etc/ha.d/rc.d/status status
harc[18605]: 2010/04/09_10:07:24 info: Running /etc/ha.d/rc.d/status status
heartbeat[22737]: 2010/04/09_10:07:26 info: Status update for node sulinux1: status active
harc[18620]: 2010/04/09_10:07:26 info: Running /etc/ha.d/rc.d/status status
heartbeat[22737]: 2010/04/09_10:07:26 info: remote resource transition completed.
heartbeat[22737]: 2010/04/09_10:07:26 info: sulinux2 wants to go standby [foreign]
heartbeat[22737]: 2010/04/09_10:07:27 info: standby: sulinux1 can take our foreign resources
heartbeat[18635]: 2010/04/09_10:07:27 info: give up foreign HA resources (standby).
ResourceManager[18648]: 2010/04/09_10:07:27 info: Releasing resource group: sulinux1 115.68.26.20
ResourceManager[18648]: 2010/04/09_10:07:27 info: Running /etc/ha.d/resource.d/IPaddr 115.68.26.20 stop
IPaddr[18702]: 2010/04/09_10:07:27 INFO: ifconfig eth0:0 down
IPaddr[18685]: 2010/04/09_10:07:27 INFO: Success
heartbeat[18635]: 2010/04/09_10:07:27 info: foreign HA resource release completed (standby).
heartbeat[22737]: 2010/04/09_10:07:27 info: Local standby process completed [foreign].
heartbeat[22737]: 2010/04/09_10:07:28 WARN: 1 lost packet(s) for [sulinux1] [10:12]
heartbeat[22737]: 2010/04/09_10:07:28 info: remote resource transition completed.
heartbeat[22737]: 2010/04/09_10:07:28 info: No pkts missing from sulinux1!
heartbeat[22737]: 2010/04/09_10:07:28 info: Other node completed standby takeover of foreign resources.