No Login on ttyS0 after console redirection

[dtesone]

Okay - here's the story...I have custom hardware running on an Intel processor where UART commands ultimately go through a FPGA that we developed in house (possible cause). - I have installed fresh base installation of CentOS-7.4 and selected "Infrastructure Server" during install with additional libs of "Compatible Libraries" and "Development Tools". I have tested the same hardware using CentOS 6.5 and was able to get the log in prompt on ttyS0 no problem so hardware checks-out. I reinstall using CentOS-7.4 and modify /etc/sysconfig/grub file with the following settings:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=<...> <--default value...too long to type!
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap intel_pstate=disable i915.enable_rc6=0"
GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200n8 console=tty1 plymouth.enable=0"
GRUB_DISABLE_RECOVERY="true"
So before you ask - the intel_pstate=disable and i915.enable_rc6=0 are there to circumvent power issues when loading the intel graphics driver. Without this I have to resort to using the VESA driver ... my ttyS0 problem happens in either case. The Plymouth.enable=0 entry is used because I found that if you remove "rhgb quiet" from Kernel cmd line, the boot process gets stuck. I have also found the order of console= commands matters. In my case, if I have console=tty1 first before console=ttyS0,115200n8, my board begins to boot properly and then drastically slows down. Eventually 4 or 5 minutes later it finishes - ran a "systemd-analyze blame" command and found system-vconsole-setup.service was taking over 3 minutes. Anyhow reversing the order seems to clear up that problem. When I boot - I get early grub menu and all boot messages coming out of serial (ttyS0) while booting. As soon as my VGA shows login prompt...I get no login prompt on ttyS0..but I still do see messages coming through the serial like when I am shutting down the system so I know the com port is working. If I switch to using a USB to Serial cable adapter and change my grub setting to console=ttyUSB0 and I enable the serial-getty@ttyUSB0.service, all works fine..I get login prompt. I have also tried on different (commercial) hardware...desktop pc and there is no problem with getting a login prompt out of ttyS0. So one would believe it is our hardware, but the fact that things work fine on our hardware using CentOS 6.5 sort of suggests hardware is OKAY. The only thing I can think of is that there is some timing difference between CentOS 6.5 and CentOS 7.4 and our FPGA is not doing the right thing. I will start debugging that aspect ...but in the meantime if anyone has had any similar behavior please chime in if you think you can help. Also - I have checked log messages (all over the place) and everything from what I can tell points to the fact that the operating system thinks agetty is running on the com port and there are no errors indicating otherwise. Thanks for listening!

[hunter86_bg]

I usually set the console like:

console=ttyS0,115200 console=tty0

And it never failed me.

[TrevorH]

You seem to have added your arguments to /etc/default/grub (not the file you mention but I assume that's a typo) in a variable that I don't have specified and I am not sure is used. You've added it to GRUB_CMDLINE_LINUX_DEFAULT= but I only have GRUB_CMDLINE_LINUX= which you haven't changed.

[dtesone]

Thanks for the replies - To answer a few questions...Yes I have tried using console=tty0 rather than console=tty1 .. neither one appears to cause any difference. Also - I did modify /etc/sysconfig/grub and ran grub2-mkconfig -o /boot/grub2/grub.cfg to officially update grub. I have also reinstalled using UEFI just to see if there was any difference between Legacy and UEFI and the result was the same. We are currently focusing on our FPGA - I think the problem is in there that we pass "normal" serial messages through the UART but some command that getty is sending I think is being dropped by our FPGA ... hence never seeing a login. I will report back when this is confirmed or denied.

[dtesone]

OK - so our FPGA that deals with 16550 does not handle interrupts and we see with a logic analyzer that agetty is trying to setup interrupts. This would explain why we are having trouble getting a login prompt to appear over serial. My question now is ... why do things work with CentOS 6.5? Same hardware and same FPGA. Has agetty changed to use interrupts in CentOS 7.x? Is there a way I can configure agetty to not use interrupts in 7.x? Can I replace agetty with some other login app that does not use interrupts?
Thanks for any and all thoughts. Adding serial interrupt logic to our FPGA at this point risks more than it buys...so we are looking for another alternative.

[TrevorH]

Look at the linux16/linuxefi line in /boot/grub2/grub.cfg and make sure that the parameter string there includes the changes you think it should have.

[hunter86_bg]

Are you really sure that there is no login on the serial console?
What happens when you press enter multiple times?

[dtesone]

Hi, Yes - I am positive /boot/grub2/grub.cfg has my changes. I even perform a ps -aux | grep agetty and see the getty process has started so the CentOS 7.4 system thinks the login prompt is up and is awaiting input. My Serial console on my Windows 7 box (TeraTerm) shows not login. If I press enter multiple times....nothing happens. I am pretty positive configuration is correct - I can switch over to a USB to Serial adapter cable and changing the grub configuration to use ttyUSB0 instead of ttyS0 ... everything works as expected and our FPGA is not in the loop when we go through USB. We found yesterday that our FPGA which really just passes along the serial commands, does not handle interrupts and we do see that agetty is trying to setup some of the interrupt registers. What doesn't make sense to me is why this same hardware and FPGA works in CentOS 6.5. I can't imagine that agetty's functionality changed from not using interrupts to using interrupts. I am hoping to run same analysis we ran yesterday with CentOS 6.5 today to see if it too is attempting to setup interrupt registers or whether it performs different functionality. Thanks again for the suggestions. I have been at this for longer than I would like and I feel like I have tried almost everything.

[hunter86_bg]

I'm pretty confident that you have run 'grub2-mkconfig -o /path/to/grub.cfg' , so I won't ask ...
Actually I'm out of ideas.

[dtesone]

OK - so here's the deal. Finally figured out what was going on. This is probably pretty much a one-off so I am not sure how much help this will be for anyone else..but for those interested...In our case, the BIOS enabled the UART 0 at the default address and assigned an IRQ of 4 (also default for most operating systems). As I mentioned earlier, we have custom FPGA that decodes the UART commands coming from the processor and routes them to an actual 16550 UART device. Well our FPGA does not yet handle interrupts. Data is slow enough not to require interrupts being implemented. Well apparently under CentOS 6.5, the UART drivers actually check to see if the UART IRQ as defined by the BIOS is actually working. If it is detected to not be hooked up, it assigns the IRQ value internally to zero which forces the UART to go into polling mode. This is why CentOS 6.5 ultimately shows the login prompt over the Serial port on the same hardware we are having a problem with CentOS 7.4. Apparently in CentOS 7.4, no such check is performed and the driver takes the BIOS' word for it. Since the BIOS said the IRQ tied to UART 0 is 4, the driver tries to setup the IRQ and make use of it. Our fix (for now), was to have our BIOS vendor modify the BIOS to disable the IRQ for the UART. This way both CentOS 6.5 and 7.4 see that the IRQ is disabled (coming from the BIOS) and they both go into polling mode rather than attempt to make use of the IRQ (which is not implemented in the FPGA). Good Times! All is good in the world now - we are able to get the login prompt up over serial in both operating system versions. Yippee!
Code Found in CentOS 6.5 UART driver:
found_port:
i = pcmcia_request_irq(link, &link->irq);
if (i != 0) {
cs_error(link, RequestIRQ, i);
link->irq.AssignedIRQ = 0;
}
No such "pcmcia_request_irq" found in CentOS 7.4 so the irq that gets passed around will have the value of "4" in CentOS 7.4 rather than the value of "0"!