Weird STDOUT problems

Support for the other architectures (X86_64, s390(x) and PowerPC)
chad3814
Posts: 2
Joined: 2009/07/31 14:45:11
Contact:

Weird STDOUT problems

Postby chad3814 » 2009/07/31 14:53:54

I'm seeing weird things from a couple CentOS 4.5 (kernel 2.6.9-55.ELsmp x86_64) boxes. Basically they get into a state where about 1/4 of the commands issued have their STDOUT output delayed. The output isn't lost, it shows up before/after/mixed in with the next command's output.

Here's an example
[ua@xfire-utility-1 log]$ ls
anaconda.log boot.log.4 cups kickstart maillog.3 ppp secure snmpd.log.2 spooler.4
anaconda.syslog compliance.log dell kickstart.log maillog.4 prelink.log secure.1 snmpd.log.3 ssclp.log
audit cron dmesg lastlog messages rpmpkgs secure.2 snmpd.log.4 ssevt.log
boot.log cron.1 dsmerror-i.log mail messages.1 rpmpkgs.1 secure.3 spooler wtmp
boot.log.1 cron.2 dsm_om_connsvc32dIO.log maillog messages.2 rpmpkgs.2 secure.4 spooler.1 wtmp.1
boot.log.2 cron.3 dsm_om_connsvc32dIO.log.bak maillog.1 messages.3 rpmpkgs.3 snmpd.log spooler.2 yum.log
boot.log.3 cron.4 dsmsched-i.log maillog.2 messages.4 rpmpkgs.4 snmpd.log.1 spooler.3
[ua@xfire-utility-1 log]$ ls
[ua@xfire-utility-1 log]$ echo hi
hi
[ua@xfire-utility-1 log]$ ping xfire-spare-1
PING xfire-spare-1.811.mtvi.com (172.16.4.224) 56(84) bytes of data.
anaconda.log boot.log.4 cups kickstart maillog.3 ppp secure snmpd.log.2 spooler.4
anaconda.syslog compliance.log dell kickstart.log maillog.4 prelink.log secure.1 snmpd.log.3 ssclp.log
audit cron dmesg lastlog messages rpmpkgs secure.2 snmpd.log.4 ssevt.log
boot.log cron.1 dsmerror-i.log mail messages.1 rpmpkgs.1 secure.3 spooler wtmp
boot.log.1 cron.2 dsm_om_connsvc32dIO.log maillog messages.2 rpmpkgs.2 secure.4 spooler.1 wtmp.1
boot.log.2 cron.3 dsm_om_connsvc32dIO.log.bak maillog.1 messages.3 rpmpkgs.3 snmpd.log spooler.2 yum.log
boot.log.3 cron.4 dsmsched-i.log maillog.2 messages.4 rpmpkgs.4 snmpd.log.1 spooler.3
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=0 ttl=61 time=1.65 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=1 ttl=61 time=0.180 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=2 ttl=61 time=0.212 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=3 ttl=61 time=0.187 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=4 ttl=61 time=0.185 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=5 ttl=61 time=0.191 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=6 ttl=61 time=0.161 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=7 ttl=61 time=0.184 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=8 ttl=61 time=0.148 ms


See the first ls has normal output (I'm in /var/log), the second has no apparent output. Shell commands (like echo that don't fork()/exec()) are not effected. Then when I ping you see the first line of the ping response, followed by the ls response, then the rest of the ping.

We first saw this 12 days ago on two machines. We reboot them (one had been up for 740 days, the other 172 days), and everything was working again. Now this morning both of them are exhibiting the problem again. Clearly rebooting our machines twice a month isn't an option.

Has anyone seen this before?

NedSlider
Forum Moderator
Posts: 2886
Joined: 2005/10/28 13:11:50
Location: UK

Weird STDOUT problems

Postby NedSlider » 2009/07/31 15:13:24

Please don't double post:

http://www.centos.org/modules/newbb/vie ... mpost82223

I'll lock this post