App Segment Fault with 7.2's kernel and 7.3's glibc

[tengent]

Hi, team

My system was CentOS 7.2, but my colleague made a mistake to update the glibc and gcc (libgcc and libstdc++) with CentOS 7.3 repo where the glibc verison is glibc-2.17-157 and gcc version is gcc-4.8.5-11. And then my app running well in CentOS7.2 went crashed with a Segmentation Fault.

I traced my app's linking libraries as below

Code: Select all

	#ldd test
        linux-vdso.so.1 =>  (0x00007fffb97ed000)
        libmysqlclient.so.20 => /usr/local/mysql/lib/libmysqlclient.so.20 (0x00007f65b53dd000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f65b5015000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f65b4df9000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f65b4bf5000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f65b49ec000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f65b46e4000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f65b43e2000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f65b41cb000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f65b59fa000)
  

And I use

Code: Select all

rpm -qf <sofile>

to get my linking libraries belong to packages as

• glibc-2.17-106.el7.x86_64
  libgcc-4.8.5-4.el7.x86_64
  libstdc++-4.8.5-4.el7.x86_64

I comfirm my colleage has already update this 3 packages to 7.3 series

• glibc-2.17-157.el7.x86_64
  libgcc-4.8.5-11.el7.x86_64
  libstdc++-4.8.5-11.el7.x86_64

but my kernel version stays at CentOS 7.2 with version as

• kernel-3.10.0-327.36.3.el7.x86_64

When we update kernel to 3.10.0-514.6.1, my app behave normal

So My Question is why glibc and libgcc in CentOS 7.3 can't be compatible with kernel-3.10.0-327.36.3 in CentOS 7.2? And how can i foresee this incompatibility , kinda with kernel-abi-whitelist? Any clues or references about this will be appreciate a lot.

Thanks

gdb info:

Code: Select all

[zhenma@dev2282 scdong]$ gdb test core.18156
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/zhenma/scdong/test...done.
[New LWP 18156]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./test'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f364995af5f in stmt_fetch_row (row=<optimized out>, stmt=0x1fc6140) at /home/upsql/chsun/build/upsql2/upsql-2/libmysql/libmysql.c:4267
4267    /home/upsql/chsun/build/upsql2/upsql-2/libmysql/libmysql.c: No such file or directory.
(gdb) set substitute-path /home/upsql/chsun/build/upsql2/upsql-2 /home/zhenma/upsql-2
(gdb) l
warning: Source file is more recent than executable.
4262      for (my_bind= stmt->bind, end= my_bind + stmt->field_count,
4263             field= stmt->fields ;
4264           my_bind < end ;
4265           my_bind++, field++)
4266      {
4267        *my_bind->error= 0;
4268        if (*null_ptr & bit)
4269        {
4270          /*
4271            We should set both row_ptr and is_null to be able to see
(gdb) p *my_bind->error
$1 = 6 '\006'
(gdb) p my_bind
$2 = (MYSQL_BIND *) 0x1fc66b0
(gdb) p *my_bind
$3 = {length = 0x7ffee4b68bb8, is_null = 0x1fc6717 "", buffer = 0x7ffee4b68b80, error = 0x7f36496e8800 <intel_02_known> "\006\004 ", 
  row_ptr = 0x7f3649580120 "\372'", store_param_func = 0x7f364a16bb58, fetch_result = 0x7f3649956e60 <fetch_result_with_conversion>, 
  skip_result = 0x7f3649955a30 <skip_result_with_length>, buffer_length = 49, offset = 0, length_value = 140732735589736, param_number = 0, pack_length = 32566, 
  buffer_type = MYSQL_TYPE_STRING, error_value = -2 '\376', is_unsigned = 127 '\177', long_data_used = 0 '\000', is_null_value = 0 '\000', extension = 0x0}
(gdb) 

we notice that my_bind->error is abnormal show with " <intel_02_known>" , Any knows what does this mean?

[TrevorH]

Sorry but point releases are an all or nothing thing.

[tengent]

Hi, all.

The problem is solved.

Actually , kernel-3.10.0-36.3 for C7.2 can work well with glibc and libgcc in C7.3.

The real problem is that app's programmer didn't initialized the local variable which is on the stack arranged by glibc.

And the behave of glibc in 7.2 and 7.3 differ a little which cause the random value in stack memory assigned to pointer exceed the whole stack, which cause a segmentation fault when app try to access the wrong memory address.