로드발랜싱 LVS 중 랜 카드 장애

랜카드 r8169 메인보드 기본칩셋 온보드

ldirectord (LVS) 분배 약 360Mbps 유입시 아래와 같은 메세지 (1분 내 순단 / 프리징 현상 발생)

 

BPS 는 다 합쳐서 360Mbps 정도

 

session은 10000개 넘어가면 위험범위 인듯...
session web 2대 : MAX 3000개 정도 (Source - Hashing)
session image 3대: MAX : 9000개 정도 (round -robin)

cat /var/log/message

Apr 8 18:31:29 localhost kernel: ------------[ cut here ]------------
Apr 8 18:31:29 localhost kernel: WARNING: at net/sched/sch_generic.c:226 dev_watchdog+0x11f/0x1ac()
Apr 8 18:31:29 localhost kernel: NETDEV WATCHDOG: eth0 (r8169): transmit timed out
Apr 8 18:31:29 localhost kernel: Modules linked in: ip_set ip_vs_rr nls_utf8 nls_cp949 cifs ip_vs_sh ip_vs iptable_filter ip_tables x_tables sbs sbshc battery ac sg serio_raw r8169 i2c_i801 button i2c_core iTCO_wdt iTCO_vendor_support usb_storage ata_piix libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd
Apr 8 18:31:29 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.28.10 #4
Apr 8 18:31:29 localhost kernel: Call Trace:
Apr 8 18:31:29 localhost kernel: [] warn_slowpath+0xf3/0x111
Apr 8 18:31:29 localhost kernel: [] __slab_alloc+0x459/0x4a2
Apr 8 18:31:29 localhost kernel: [] nf_iterate+0x41/0x7d
Apr 8 18:31:29 localhost kernel: [] enqueue_task_fair+0x193/0x1a7
Apr 8 18:31:29 localhost kernel: [] enqueue_task+0x50/0x5b
Apr 8 18:31:29 localhost kernel: [] try_to_wake_up+0x23b/0x24d
Apr 8 18:31:29 localhost kernel: [] read_tsc+0x9/0x1c
Apr 8 18:31:29 localhost kernel: [] getnstimeofday+0x53/0xb2
Apr 8 18:31:29 localhost kernel: [] autoremove_wake_function+0x9/0x2e
Apr 8 18:31:29 localhost kernel: [] ktime_get_ts+0x22/0x4b
Apr 8 18:31:29 localhost kernel: [] dev_watchdog+0x11f/0x1ac
Apr 8 18:31:29 localhost kernel: [] sched_clock_cpu+0x123/0x12b
Apr 8 18:31:29 localhost kernel: [] read_tsc+0x9/0x1c
Apr 8 18:31:29 localhost kernel: [] dev_watchdog+0x0/0x1ac
Apr 8 18:31:29 localhost kernel: [] run_timer_softirq+0x163/0x1d4
Apr 8 18:31:29 localhost kernel: [] ktime_get+0xc/0x41
Apr 8 18:31:29 localhost kernel: [] __do_softirq+0x7a/0x13d
Apr 8 18:31:29 localhost kernel: [] call_softirq+0x1c/0x28
Apr 8 18:31:29 localhost kernel: [] do_softirq+0x2c/0x68
Apr 8 18:31:29 localhost kernel: [] irq_exit+0x3f/0x85
Apr 8 18:31:29 localhost kernel: [] smp_apic_timer_interrupt+0x93/0xac
Apr 8 18:31:29 localhost kernel: [] apic_timer_interrupt+0x6b/0x70
Apr 8 18:31:29 localhost kernel: [] tcp_poll+0x0/0x12e
Apr 8 18:31:29 localhost kernel: [] mwait_idle+0x41/0x44
Apr 8 18:31:29 localhost kernel: [] cpu_idle+0x4a/0x6d
Apr 8 18:31:29 localhost kernel: ---[ end trace 6d57e20ee1d36734 ]---
Apr 8 18:31:29 localhost kernel: r8169: eth0: link up
Apr 8 21:55:53 localhost kernel: r8169: eth0: link up

ifconfig

eth0 Link encap:Ethernet HWaddr 00:1D:92:B5:9E:B4
inet addr:xxx.68.xx.xx  Bcast:xxx.68.xx.xx  Mask:255.255.255.128
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2708834313 errors:0 dropped:0 overruns:0 frame:0
TX packets:2718263700 errors:0 dropped:92 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:470055594710 (437.7 GiB) TX bytes:472380922053 (439.9 GiB)
Interrupt:253

인터럽트가 253개가 발생한걸 볼수 있음...

이더넷 인터페이스 칩셋이 퍼포먼스를 못 따라가서 발생한것으로 판단됨.....

LVS서버가 더 필요하거나 퍼포먼스 더 나은 랜칩셋으로 바꿔볼것.!!!