Here is some interesting information from powertop. Looks like timer(softirq) is causing the issue and we have devices showing up in Device Stats in 3.2 that don't show up in 2.6. 2.6.32 kernel Device Stats: Usage Device name 76.9% CPU use 76.9% CPU use 2336696 pkts/s Network interface: eth0 (igb) 391.2 pkts/s Network interface: eth1 (igb) 100.0% USB device: UHCI Host Controller 100.0% USB device: Virtual Keyboard and Mouse (American Megatrends Inc.) 0.0% USB device: EHCI Host Controller 0.0% USB device: UHCI Host Controller 0.0% USB device: EHCI Host Controller 0.0% USB device: UHCI Host Controller 0.0% USB device: UHCI Host Controller 0.0% USB device: UHCI Host Controller 0.0% USB device: UHCI Host Controller 3.2.46 kernel Device Stats: Usage Device name 70.1% CPU use 70.1% CPU use 2586449 pkts/s Network interface: eth0 (igb) 294.1 pkts/s Network interface: eth1 (igb) 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 100.0% PCI Device: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device 100.0% PCI Device: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device 100.0% PCI Device: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device 100.0% PCI Device: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device 100.0% PCI Device: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 100.0% PCI Device: Intel Corporation 82576 Gigabit Network Connection 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) SMBus Controller 100.0% PCI Device: Intel Corporation 82576 Gigabit Network Connection 100.0% PCI Device: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2 100.0% PCI Device: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device 100.0% PCI Device: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device 100.0% PCI Device: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers 100.0% PCI Device: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 100.0% PCI Device: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 100.0% PCI Device: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 100.0% PCI Device: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 100.0% PCI Device: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 100.0% PCI Device: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device 100.0% USB device: Virtual Keyboard and Mouse (American Megatrends Inc.) 100.0% USB device: UHCI Host Controller 0.0% USB device: UHCI Host Controller 0.0% USB device: UHCI Host Controller 0.0% USB device: UHCI Host Controller 0.0% USB device: UHCI Host Controller 0.0% USB device: UHCI Host Controller 0.0% USB device: EHCI Host Controller 0.0% USB device: EHCI Host Controller 2.6.32 Overview Summary: 1036.6 wakeups/second, 0.0 GPU ops/second, 0.0 VFS ops/sec and 82.0% CPU use Usage Events/s Category Description 764.1 ms/s 382.1 Process php-fpm: pool www 10.4 ms/s 322.1 Process /usr/bin/nutcracker -d -s 22222 -i 30000 -c /etc/nutcracker.conf -p /var/run/nutcracker/nutcracker.pid -o /var/log/nutcracker/n 18.2 ms/s 151.1 Process /home/userx/mydomain.com/bin/uwsgi --ini /etc/uwsgi.ini --pidfile /var/run/uwsgi.pid 10.1 ms/s 136.8 Process nginx: worker process 68.2 us/s 9.6 Process [ksoftirqd/0] 49.3 us/s 4.8 Process /usr/sbin/iscsid 6.7 ms/s 0.00 Interrupt [3] net_rx(softirq) 146.1 us/s 2.0 Process /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock 34.6 us/s 2.0 Process /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d 141.7 us/s 1.9 Process /usr/bin/python /usr/bin/salt-minion 31.6 us/s 1.6 Process php-fpm: master process (/etc/php5/fpm/php-fpm.conf) 0.8 ms/s 1.2 Process /usr/bin/monit -c /etc/monit/monitrc 107.8 us/s 1.3 Process [kjournald] 3.1 ms/s 0.00 Interrupt [7] sched(softirq) 8.8 us/s 1.1 Process [events/4] 173.6 us/s 1.0 Process collectd -C /etc/collectd/collectd.conf -f 289.0 us/s 0.9 Process nrpe 26.1 us/s 1.0 Process /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 106:109 8.7 us/s 1.0 Process [events/5] 6.9 us/s 0.9 Process [events/14] 7.2 us/s 0.9 Process [events/13] 6.9 us/s 0.9 Process [events/7] 6.8 us/s 0.9 Process [events/12] 6.7 us/s 0.9 Process [events/15] 7.4 us/s 0.8 Process [events/6] 6.7 us/s 0.8 Process [events/11] 8.1 us/s 0.8 Process [events/10] 7.2 us/s 0.7 Process [events/9] 1.7 ms/s 0.00 Interrupt [1] timer(softirq) 18.0 us/s 0.6 Process /usr/lib/gamin/gam_server 5.2 us/s 0.6 Process [events/3] 1.2 ms/s 0.05 Process powertop 116.9 us/s 0.4 Process sh 1.2 ms/s 0.00 Interrupt [9] RCU(softirq) 283.7 us/s 0.3 Process sshd 3.3 us/s 0.4 Process [events/2] 46.0 us/s 0.3 Process /usr/sbin/rsyslogd -c5 2.9 us/s 0.3 Process [kblockd/4] 1.5 us/s 0.3 Process [ksoftirqd/1] 2.1 us/s 0.30 Process [events/1] 1.9 us/s 0.30 Process [ksoftirqd/11] 1.8 us/s 0.30 Process [ksoftirqd/10] 3.2.46 Overview Summary: 1591.3 wakeups/second, 0.0 GPU ops/second, 0.0 VFS ops/sec and 63.2% CPU use Usage Events/s Category Description 1.5 ms/s 702.7 Interrupt [1] timer(softirq) 550.1 ms/s 245.5 Process php-fpm: pool www 13.2 ms/s 210.5 Process /usr/bin/nutcracker -d -s 22222 -i 30000 -c /etc/nutcracker.conf -p /var/run/nutcracker/nutcracker.pid -o /var/log/nutcracker/n 19.4 ms/s 143.8 Process /home/userx/mysite.com/bin/uwsgi --ini /etc/uwsgi.ini --pidfile /var/run/uwsgi.pid 7.9 ms/s 104.6 Interrupt [3] net_rx(softirq) 12.8 ms/s 91.4 Process nginx: worker process 3.6 ms/s 54.0 Interrupt [7] sched(softirq) 0.7 ms/s 12.8 Timer tcp_delack_timer 274.1 us/s 4.5 kWork cache_reap 79.8 us/s 3.9 Process /usr/sbin/iscsid 52.4 us/s 1.9 Process /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d 125.2 us/s 1.5 Process /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock 3.9 ms/s 0.00 Timer delayed_work_timer_fn 172.0 us/s 1.4 Process /usr/bin/python /usr/bin/salt-minion 3.7 ms/s 0.00 kWork do_dbs_timer 269.1 us/s 1.3 Process collectd -C /etc/collectd/collectd.conf -f 39.1 us/s 1.2 Process php-fpm: master process (/etc/php5/fpm/php-fpm.conf) 2.3 ms/s 0.10 Process powertop 5.3 us/s 0.9 Timer clocksource_watchdog 345.9 us/s 0.8 Process nrpe 41.3 us/s 0.7 Process /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 106:109 20.3 us/s 0.6 Interrupt [67] eth0-TxRx-2 19.2 us/s 0.6 kWork edac_mc_workq_function 1.4 ms/s 0.00 Interrupt [0] timer/0 24.1 us/s 0.5 Interrupt [65] eth0-TxRx-0 28.1 us/s 0.5 Interrupt [69] eth0-TxRx-4 26.3 us/s 0.5 Interrupt [66] eth0-TxRx-1 1.0 ms/s 0.00 Interrupt [9] RCU(softirq) 149.5 us/s 0.30 Process sh 20.3 us/s 0.3 Interrupt [68] eth0-TxRx-3 3.4 us/s 0.3 kWork edac_pci_workq_function 31.1 us/s 0.30 Process /usr/bin/monit -c /etc/monit/monitrc 0.8 ms/s 0.00 Process [kworker/4:0] 0.7 ms/s 0.00 Process [kworker/10:1] 0.7 ms/s 0.00 Process [kworker/7:1] 685.7 us/s 0.00 Process [kworker/6:1] 659.3 us/s 0.00 Process [kworker/5:1] 24.5 us/s 0.25 Interrupt [72] eth0-TxRx-7 8.6 us/s 0.25 Process [kjournald] 636.4 us/s 0.00 Process [kworker/9:1] 3.0 us/s 0.25 Process [watchdog/6] 558.9 us/s 0.00 Process [kworker/15:1] |