INFO: task xfs-data/sdb:1615 blocked for more than 120 seconds.

[itsbig]

Anyone has experience/solution with this
OS : CentOS 7.9
Kernel : 3.10.0-1160.88.1.el7.x86_64

Sep 4 17:12:45 d2p-caching04 kernel: INFO: task xfs-data/sdb:1615 blocked for more than 120 seconds.
Sep 4 17:12:45 d2p-caching04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 4 17:12:45 d2p-caching04 kernel: xfs-data/sdb D ffff8970e21f9080 0 1615 2 0x00000000
Sep 4 17:12:45 d2p-caching04 kernel: Workqueue: xfs-data/sdb xfs_end_io [xfs]
Sep 4 17:12:45 d2p-caching04 kernel: Call Trace:
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97d8c3f9>] schedule+0x29/0x70
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97d8db35>] rwsem_down_write_failed+0x215/0x3c0
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc05eec3d>] ? __xfs_setfilesize+0x2d/0x100 [xfs]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97997fd7>] call_rwsem_down_write_failed+0x17/0x30
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97d8b90d>] down_write+0x2d/0x3d
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc0609021>] xfs_ilock+0xc1/0x120 [xfs]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc05eec3d>] __xfs_setfilesize+0x2d/0x100 [xfs]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc05eed56>] xfs_setfilesize_ioend+0x46/0x60 [xfs]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc05f0afb>] xfs_end_io+0x4b/0x90 [xfs]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff976bdfbf>] process_one_work+0x17f/0x440
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff976be514>] rescuer_thread+0x294/0x3c0
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff976be280>] ? process_one_work+0x440/0x440
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff976c5f91>] kthread+0xd1/0xe0
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff976c5ec0>] ? insert_kthread_work+0x40/0x40
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97d99df7>] ret_from_fork_nospec_begin+0x21/0x21
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff976c5ec0>] ? insert_kthread_work+0x40/0x40
Sep 4 17:12:45 d2p-caching04 kernel: INFO: task systemd-logind:1895 blocked for more than 120 seconds.
Sep 4 17:12:45 d2p-caching04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 4 17:12:45 d2p-caching04 kernel: systemd-logind D ffff8990ee2c1660 0 1895 1 0x00000084
Sep 4 17:12:45 d2p-caching04 kernel: Call Trace:
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97d8c3f9>] schedule+0x29/0x70
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97d8db35>] rwsem_down_write_failed+0x215/0x3c0
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97987642>] ? get_from_free_list+0x42/0x50
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97997fd7>] call_rwsem_down_write_failed+0x17/0x30
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97988920>] ? ida_get_new_above+0x230/0x2a0
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97d8b90d>] down_write+0x2d/0x3d
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff977d1851>] register_shrinker+0x21/0x50
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff978521f9>] sget_userns+0x489/0x4c0
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97851130>] ? get_anon_bdev+0x110/0x110
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff978522ad>] sget+0x7d/0xb0
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97851130>] ? get_anon_bdev+0x110/0x110
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff977d84a0>] ? shmem_rename+0x20/0x20
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97852310>] mount_nodev+0x30/0xb0
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff977d8128>] shmem_mount+0x18/0x20
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97852eae>] mount_fs+0x3e/0x1b0
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97871b47>] vfs_kern_mount+0x67/0x110
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff9787427f>] do_mount+0x1ef/0xd00
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff9784b18a>] ? __check_object_size+0x1ca/0x250
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff9782922c>] ? kmem_cache_alloc_trace+0x3c/0x200
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff978750d3>] SyS_mount+0x83/0xd0
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc06c7d87>] unload_network_ops_symbols+0x65b7/0x76d0 [falcon_lsm_pinned_14812]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97872474>] ? mntput+0x24/0x40
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc0816277>] unload_network_ops_symbols+0x7aa7/0x8c80 [falcon_lsm_pinned_15003]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc0911e13>] unload_network_ops_symbols+0x85d3/0x9720 [falcon_lsm_pinned_15110]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc092fcd3>] unload_network_ops_symbols+0x85d3/0x9720 [falcon_lsm_pinned_15309]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc094dcd3>] unload_network_ops_symbols+0x85d3/0x9720 [falcon_lsm_pinned_15402]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffffc096bcd3>] unload_network_ops_symbols+0x85d3/0x9720 [falcon_lsm_pinned_15508]
Sep 4 17:12:45 d2p-caching04 kernel: [<ffffffff97d99f92>] system_call_fastpath+0x25/0x2a

[TrevorH]

Decoded, that lot basically means "we waited for 2 whole minutes to get a response to an I/O we did to /dev/sdb and now I give up".

It means your /dev/sdb disk is not responding in a timely manner, possibly because it's massively overloaded, posssibly because it's failing and has hardware problems.

I also see mention of falcon_lsm_pinned* a lot in the stacktrace which appears to be part of a third party kernel module "Crowdstrike Falcon Sensor". That is not something that CentOS provides and since it's mentioned a lot in the stacktrace is a possible cause.