Weird STDOUT problems

Support for the other architectures (X86_64, s390(x) and PowerPC)
Post Reply
chad3814
Posts: 2
Joined: 2009/07/31 14:45:11
Contact:

Weird STDOUT problems

Post by chad3814 » 2009/07/31 14:53:54

I'm seeing weird things from a couple CentOS 4.5 (kernel 2.6.9-55.ELsmp x86_64) boxes. Basically they get into a state where about 1/4 of the commands issued have their STDOUT output delayed. The output isn't lost, it shows up before/after/mixed in with the next command's output.

Here's an example
[quote]
[b][ua@xfire-utility-1 log]$ ls[/b]
anaconda.log boot.log.4 cups kickstart maillog.3 ppp secure snmpd.log.2 spooler.4
anaconda.syslog compliance.log dell kickstart.log maillog.4 prelink.log secure.1 snmpd.log.3 ssclp.log
audit cron dmesg lastlog messages rpmpkgs secure.2 snmpd.log.4 ssevt.log
boot.log cron.1 dsmerror-i.log mail messages.1 rpmpkgs.1 secure.3 spooler wtmp
boot.log.1 cron.2 dsm_om_connsvc32dIO.log maillog messages.2 rpmpkgs.2 secure.4 spooler.1 wtmp.1
boot.log.2 cron.3 dsm_om_connsvc32dIO.log.bak maillog.1 messages.3 rpmpkgs.3 snmpd.log spooler.2 yum.log
boot.log.3 cron.4 dsmsched-i.log maillog.2 messages.4 rpmpkgs.4 snmpd.log.1 spooler.3
[b][ua@xfire-utility-1 log]$ ls[/b]
[b][ua@xfire-utility-1 log]$ echo hi[/b]
hi
[b][ua@xfire-utility-1 log]$ ping xfire-spare-1[/b]
PING xfire-spare-1.811.mtvi.com (172.16.4.224) 56(84) bytes of data.
anaconda.log boot.log.4 cups kickstart maillog.3 ppp secure snmpd.log.2 spooler.4
anaconda.syslog compliance.log dell kickstart.log maillog.4 prelink.log secure.1 snmpd.log.3 ssclp.log
audit cron dmesg lastlog messages rpmpkgs secure.2 snmpd.log.4 ssevt.log
boot.log cron.1 dsmerror-i.log mail messages.1 rpmpkgs.1 secure.3 spooler wtmp
boot.log.1 cron.2 dsm_om_connsvc32dIO.log maillog messages.2 rpmpkgs.2 secure.4 spooler.1 wtmp.1
boot.log.2 cron.3 dsm_om_connsvc32dIO.log.bak maillog.1 messages.3 rpmpkgs.3 snmpd.log spooler.2 yum.log
boot.log.3 cron.4 dsmsched-i.log maillog.2 messages.4 rpmpkgs.4 snmpd.log.1 spooler.3
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=0 ttl=61 time=1.65 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=1 ttl=61 time=0.180 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=2 ttl=61 time=0.212 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=3 ttl=61 time=0.187 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=4 ttl=61 time=0.185 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=5 ttl=61 time=0.191 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=6 ttl=61 time=0.161 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=7 ttl=61 time=0.184 ms
64 bytes from xfire-spare-1.811.mtvi.com (172.16.4.224): icmp_seq=8 ttl=61 time=0.148 ms
[/quote]

See the first [b]ls[/b] has normal output (I'm in /var/log), the second has no apparent output. Shell commands (like [b]echo[/b] that don't [i]fork()[/i]/[i]exec()[/i]) are not effected. Then when I [b]ping[/b] you see the first line of the [b]ping[/b] response, followed by the [b]ls[/b] response, then the rest of the [b]ping[/b].

We first saw this 12 days ago on two machines. We reboot them (one had been up for 740 days, the other 172 days), and everything was working again. Now this morning both of them are exhibiting the problem again. Clearly rebooting our machines twice a month isn't an option.

Has anyone seen this before?

NedSlider
Forum Moderator
Posts: 2887
Joined: 2005/10/28 13:11:50
Location: UK

Weird STDOUT problems

Post by NedSlider » 2009/07/31 15:13:24

Please don't double post:

http://www.centos.org/modules/newbb/viewtopic.php?topic_id=21495&forum=27&post_id=82223#forumpost82223

I'll lock this post

Post Reply