XFS Errors and Filesystem Issues

Issues related to hardware problems
willinit
Posts: 3
Joined: 2016/03/15 01:26:31

XFS Errors and Filesystem Issues

Postby willinit » 2016/03/15 01:39:22

I have a large ~72TB backup server running CentOS 5.11. For the past month or two the server's shown the occasional XFS related error but is now shutting down the file system which ultimately prevents it from being accessible. After digging through forums I've tried running a few different kernels (3.2.77-1.el5.elrepo, 2.6.39-2.el5.elrepo, 2.6.18-404.el5) thinking that maybe it was a kernel bug but still see the issue. Below are recent log outputs with the errors that appear right when the filesystem goes away. Any help with this would be greatly appreciated!


Mar 14 17:02:31 kernel: Filesystem dm-0: XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c. Caller 0xffffffff883e34fe
Mar 14 17:02:31 kernel:
Mar 14 17:02:31 kernel:
Mar 14 17:02:31 kernel: Call Trace:
Mar 14 17:02:31 kernel: [<ffffffff883ddd95>] :xfs:xfs_trans_cancel+0x55/0xfa
Mar 14 17:02:31 kernel: [<ffffffff883e34fe>] :xfs:xfs_mkdir+0x429/0x465
Mar 14 17:02:31 kernel: [<ffffffff883ec399>] :xfs:xfs_vn_mknod+0x161/0x215
Mar 14 17:02:31 kernel: [<ffffffff800ed95a>] vfs_mkdir+0xe3/0x152
Mar 14 17:02:31 kernel: [<ffffffff800edf30>] sys_mkdirat+0xa3/0xe4
Mar 14 17:02:31 kernel: [<ffffffff800a7f3f>] sys_futex+0x129/0x14a
Mar 14 17:02:31 kernel: [<ffffffff8005ddf9>] error_exit+0x0/0x84
Mar 14 17:02:31 kernel: [<ffffffff8005d116>] system_call+0x7e/0x83
Mar 14 17:02:31 kernel:
Mar 14 17:02:31 kernel: xfs_force_shutdown(dm-0,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff883dddae
Mar 14 17:02:31 kernel: Filesystem dm-0: Corruption of in-memory data detected. Shutting down filesystem: dm-0
Mar 14 17:02:31 kernel: Please umount the filesystem, and rectify the problem(s)
Mar 14 17:02:53 kernel: Filesystem dm-0: xfs_log_force: error 5 returned.
Mar 14 17:04:23 kernel: Filesystem dm-0: xfs_log_force: error 5 returned.
Mar 14 17:05:23 last message repeated 2 times
Mar 14 17:06:23 last message repeated 2 times






Mar 14 18:02:42 kernel: Filesystem dm-0: XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c. Caller 0xffffffff883e34fe
Mar 14 18:02:42 kernel:
Mar 14 18:02:42 kernel:
Mar 14 18:02:42 kernel: Call Trace:
Mar 14 18:02:42 kernel: [<ffffffff883ddd95>] :xfs:xfs_trans_cancel+0x55/0xfa
Mar 14 18:02:42 kernel: [<ffffffff883e34fe>] :xfs:xfs_mkdir+0x429/0x465
Mar 14 18:02:42 kernel: [<ffffffff883ec399>] :xfs:xfs_vn_mknod+0x161/0x215
Mar 14 18:02:42 kernel: [<ffffffff800ed95a>] vfs_mkdir+0xe3/0x152
Mar 14 18:02:42 kernel: [<ffffffff800edf30>] sys_mkdirat+0xa3/0xe4
Mar 14 18:02:42 kernel: [<ffffffff800a7f3f>] sys_futex+0x129/0x14a
Mar 14 18:02:42 kernel: [<ffffffff8005ddf9>] error_exit+0x0/0x84
Mar 14 18:02:42 kernel: [<ffffffff8005d116>] system_call+0x7e/0x83
Mar 14 18:02:42 kernel:
Mar 14 18:02:42 kernel: xfs_force_shutdown(dm-0,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff883dddae
Mar 14 18:02:42 kernel: Filesystem dm-0: Corruption of in-memory data detected. Shutting down filesystem: dm-0
Mar 14 18:02:42 kernel: Please umount the filesystem, and rectify the problem(s)
Mar 14 18:02:44 kernel: Filesystem dm-0: xfs_log_force: error 5 returned.
Mar 14 18:03:03 last message repeated 2 times
Mar 14 18:03:03 kernel: xfs_force_shutdown(dm-0,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xffffffff883e50fe
Mar 14 18:03:03 kernel: Filesystem dm-0: xfs_log_force: error 5 returned.
Mar 14 18:03:03 kernel: Filesystem dm-0: xfs_log_force: error 5 returned.
Mar 14 18:03:03 kernel: xfs_force_shutdown(dm-0,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xffffffff883e50fe

MartinR
Posts: 281
Joined: 2015/05/11 07:53:27
Location: UK

Re: XFS Errors and Filesystem Issues

Postby MartinR » 2016/03/15 10:02:44

It's not an XFS error, XFS is shutting down correctly to prevent corruption. You need to find out why:
Corruption of in-memory data detected

This looks at first glance to be a hardware problem, is your memory failing?

willinit
Posts: 3
Joined: 2016/03/15 01:26:31

Re: XFS Errors and Filesystem Issues

Postby willinit » 2016/03/15 12:37:35

MartinR: How would I find out? Any log specific items I should look for that would hint at a possible memory failure?

User avatar
TrevorH
Forum Moderator
Posts: 21209
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: XFS Errors and Filesystem Issues

Postby TrevorH » 2016/03/15 13:05:32

If you have a system with ECC RAM and it has out of band access (e.g iLO or iDRAC etc) then you may find errors have been logged on that. Otherwise you probably need to run memtest86+ for a day or two.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

aks
Posts: 2498
Joined: 2014/09/20 11:22:14

Re: XFS Errors and Filesystem Issues

Postby aks » 2016/03/15 17:25:51

So it seems the bork is at 0xffffffff883e34fe.
[<ffffffff883e34fe>] :xfs:xfs_mkdir+0x429/0x465
So while making a directory (for a nod, the next line)?

Anyway Googling around error 5 returned from xfs_log_force suggests a problem in unlinked/free lists.
The suggestions from the various xfs mail lists that Google can point at suggest doing an xfs_repair (but make sure that your backups work first).

My point is that the reference to in-memory may not be related to memory - it might mean that what's on disk ain't what's "cached" in memory.

Just my $7.50 worth...

willinit
Posts: 3
Joined: 2016/03/15 01:26:31

Re: XFS Errors and Filesystem Issues

Postby willinit » 2016/03/16 19:49:05

So just to make sure I addressed any sort of RAM issues I actually removed the old stuff and replaced it, doubling the former amount of memory to 24G. I have since started xfs_repair using the following command string:

Code: Select all

xfs_repair -vv -o bhash=16384 -o ihash=16384 -o ag_stride=16 /dev/mapper/cdp1raid-cdp1raid


This string did seem to drastically speed up the xfs_repair process to the point that it got much further in about 20 minutes than it did in 20 hours using just the basic command without all of the "-o" options. It was making great progress but for the past 10 hours has been constantly outputting the following line every 15 minutes:

rebuild AG headers and trees - 73 of 73 allocation groups done


I tried watching strace for the process and it's forks but have only seen this:

(root)>strace -vv -s 4000 -f -p 5603
Process 5603 attached with 7 threads - interrupt to quit
[pid 5604] rt_sigtimedwait([HUP ALRM], <unfinished ...>
[pid 8976] futex(0x2b4cb8916ea8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 8975] futex(0xe47d4f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 8974] futex(0xe47d4f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 8973] futex(0xe47d4f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 8972] futex(0xe47d4f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 5603] futex(0xe47d4f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 5604] <... rt_sigtimedwait resumed> 0, 0, 8) = 14
[pid 5604] write(2, " - 14:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done\n", 83) = 83
[pid 5604] rt_sigtimedwait([HUP ALRM], 0, 0, 8) = 14
[pid 5604] write(2, " - 14:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done\n", 83) = 83
[pid 5604] rt_sigtimedwait([HUP ALRM], 0, 0, 8) = 14
[pid 5604] write(2, " - 14:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done\n", 83) = 83
[pid 5604] rt_sigtimedwait([HUP ALRM], 0, 0, 8) = 14
[pid 5604] write(2, " - 14:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done\n", 83) = 83
[pid 5604] rt_sigtimedwait([HUP ALRM], 0, 0, 8) = 14
[pid 5604] write(2, " - 15:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done\n", 83) = 83
[pid 5604] rt_sigtimedwait([HUP ALRM], 0, 0, 8) = 14
[pid 5604] write(2, " - 15:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done\n", 83) = 83
[pid 5604] rt_sigtimedwait([HUP ALRM], 0, 0, 8) = 14
[pid 5604] write(2, " - 15:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done\n", 83) = 83
[pid 5604] rt_sigtimedwait([HUP ALRM], <unfinished ...>



At this point I'm wondering if it's actually still rebuilding AG headers and trees or if it's hung up on something and won't actually finish.


Here's the full output of the repair.log file just in case it's helpful.

Phase 1 - find and verify superblock...
- reporting progress in intervals of 15 minutes
Phase 2 - using internal log
- zero log...
zero_log: head block 2 tail block 2
- scan filesystem freespace and inode maps...
ir_freecount/free mismatch, inode chunk 0/631750720, freecount 9 nfree 0
- 15:33:35: scanning filesystem freespace - 73 of 73 allocation groups done
- found root inode chunk
libxfs_bcache: 0xe47d470
Max supported entries = 131072
Max utilized entries = 131072
Active entries = 131044
Hash table size = 16384
Hits = 0
Misses = 138716
Hit ratio = 0.00
MRU 0 entries = 131044 (100%)
MRU 1 entries = 0 ( 0%)
MRU 2 entries = 0 ( 0%)
MRU 3 entries = 0 ( 0%)
MRU 4 entries = 0 ( 0%)
MRU 5 entries = 0 ( 0%)
MRU 6 entries = 0 ( 0%)
MRU 7 entries = 0 ( 0%)
MRU 8 entries = 0 ( 0%)
MRU 9 entries = 0 ( 0%)
MRU 10 entries = 0 ( 0%)
MRU 11 entries = 0 ( 0%)
MRU 12 entries = 0 ( 0%)
MRU 13 entries = 0 ( 0%)
MRU 14 entries = 0 ( 0%)
MRU 15 entries = 0 ( 0%)
Hash buckets with 0 entries 12 ( 0%)
Hash buckets with 1 entries 68 ( 0%)
Hash buckets with 2 entries 229 ( 0%)
Hash buckets with 3 entries 562 ( 1%)
Hash buckets with 4 entries 1072 ( 3%)
Hash buckets with 5 entries 1584 ( 6%)
Hash buckets with 6 entries 2038 ( 9%)
Hash buckets with 7 entries 2234 ( 11%)
Hash buckets with 8 entries 2000 ( 12%)
Hash buckets with 9 entries 1790 ( 12%)
Hash buckets with 10 entries 1550 ( 11%)
Hash buckets with 11 entries 1173 ( 9%)
Hash buckets with 12 entries 784 ( 7%)
Hash buckets with 13 entries 524 ( 5%)
Hash buckets with 14 entries 308 ( 3%)
Hash buckets with 15 entries 201 ( 2%)
Hash buckets with 16 entries 115 ( 1%)
Hash buckets with 17 entries 66 ( 0%)
Hash buckets with 18 entries 23 ( 0%)
Hash buckets with 19 entries 16 ( 0%)
Hash buckets with 20 entries 11 ( 0%)
Hash buckets with 21 entries 4 ( 0%)
Hash buckets with 22 entries 2 ( 0%)
Hash buckets with >24 entries 18 ( 0%)
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- 15:33:35: scanning agi unlinked lists - 73 of 73 allocation groups done
- process known inodes and perform inode discovery...
- agno = 0
- agno = 48
- agno = 32
- agno = 16
- agno = 49
- agno = 17
- agno = 64
- agno = 18
- agno = 33
- agno = 65
- agno = 34
- agno = 50
- agno = 19
- agno = 35
- agno = 66
- agno = 51
- agno = 52
- agno = 53
- agno = 20
- agno = 67
- agno = 36
- agno = 54
- agno = 68
- agno = 21
- agno = 22
- agno = 37
- agno = 23
- agno = 69
- agno = 55
- agno = 70
- agno = 38
- agno = 56
- agno = 24
- agno = 71
- agno = 25
- agno = 72
- agno = 39
- agno = 57
- agno = 26
- agno = 40
- agno = 58
- agno = 27
- agno = 59
- agno = 41
- agno = 60
- agno = 42
- agno = 28
- agno = 61
- agno = 43
- agno = 29
- agno = 62
- agno = 30
- agno = 44
- agno = 63
- agno = 31
- agno = 45
- agno = 46
- agno = 47
- 15:33:44: process known inodes and inode discovery - 1796288 of 349428160 inodes done
- 15:33:44: Phase 3: elapsed time 9 seconds - processed 11975253 inodes per minute
- 15:33:44: Phase 3: 0% done - estimated remaining time 29 minutes, 1 second
- 15:48:43: process known inodes and inode discovery - 28051648 of 349428160 inodes done
- 15:48:43: Phase 3: elapsed time 15 minutes, 8 seconds - processed 1853633 inodes per minute
- 15:48:43: Phase 3: 8% done - estimated remaining time 2 hours, 53 minutes, 22 seconds
- 16:03:43: process known inodes and inode discovery - 56839360 of 349428160 inodes done
- 16:03:43: Phase 3: elapsed time 30 minutes, 8 seconds - processed 1886261 inodes per minute
- 16:03:43: Phase 3: 16% done - estimated remaining time 2 hours, 35 minutes, 6 seconds
- 16:18:43: process known inodes and inode discovery - 75489792 of 349428160 inodes done
- 16:18:43: Phase 3: elapsed time 45 minutes, 8 seconds - processed 1672595 inodes per minute
- 16:18:43: Phase 3: 21% done - estimated remaining time 2 hours, 43 minutes, 46 seconds
- 16:33:43: process known inodes and inode discovery - 86221632 of 349428160 inodes done
- 16:33:43: Phase 3: elapsed time 1 hour, 8 seconds - processed 1433840 inodes per minute
- 16:33:43: Phase 3: 24% done - estimated remaining time 3 hours, 3 minutes, 34 seconds
- 16:48:43: process known inodes and inode discovery - 94435264 of 349428160 inodes done
- 16:48:43: Phase 3: elapsed time 1 hour, 15 minutes, 8 seconds - processed 1256902 inodes per minute
- 16:48:43: Phase 3: 27% done - estimated remaining time 3 hours, 22 minutes, 52 seconds
- 17:03:43: process known inodes and inode discovery - 104917632 of 349428160 inodes done
- 17:03:43: Phase 3: elapsed time 1 hour, 30 minutes, 8 seconds - processed 1164026 inodes per minute
- 17:03:43: Phase 3: 30% done - estimated remaining time 3 hours, 30 minutes, 3 seconds
- 17:18:43: process known inodes and inode discovery - 122399040 of 349428160 inodes done
- 17:18:43: Phase 3: elapsed time 1 hour, 45 minutes, 8 seconds - processed 1164226 inodes per minute
- 17:18:43: Phase 3: 35% done - estimated remaining time 3 hours, 15 minutes
- 17:33:43: process known inodes and inode discovery - 139011904 of 349428160 inodes done
- 17:33:43: Phase 3: elapsed time 2 hours, 8 seconds - processed 1157146 inodes per minute
- 17:33:43: Phase 3: 39% done - estimated remaining time 3 hours, 1 minute, 50 seconds
- 17:48:43: process known inodes and inode discovery - 149658240 of 349428160 inodes done
- 17:48:43: Phase 3: elapsed time 2 hours, 15 minutes, 8 seconds - processed 1107485 inodes per minute
- 17:48:43: Phase 3: 42% done - estimated remaining time 3 hours, 22 seconds
- 18:03:43: process known inodes and inode discovery - 159843776 of 349428160 inodes done
- 18:03:43: Phase 3: elapsed time 2 hours, 30 minutes, 8 seconds - processed 1064678 inodes per minute
- 18:03:43: Phase 3: 45% done - estimated remaining time 2 hours, 58 minutes, 4 seconds
imap claims a free inode 631750766 is in use, correcting imap and clearing inode
imap claims a free inode 631750772 is in use, correcting imap and clearing inode
- 18:18:43: process known inodes and inode discovery - 168598976 of 349428160 inodes done
- 18:18:43: Phase 3: elapsed time 2 hours, 45 minutes, 8 seconds - processed 1020986 inodes per minute
- 18:18:43: Phase 3: 48% done - estimated remaining time 2 hours, 57 minutes, 6 seconds
- 18:33:43: process known inodes and inode discovery - 177444096 of 349428160 inodes done
- 18:33:43: Phase 3: elapsed time 3 hours, 8 seconds - processed 985070 inodes per minute
- 18:33:43: Phase 3: 50% done - estimated remaining time 2 hours, 54 minutes, 35 seconds
- 18:48:43: process known inodes and inode discovery - 187339904 of 349428160 inodes done
- 18:48:43: Phase 3: elapsed time 3 hours, 15 minutes, 8 seconds - processed 960061 inodes per minute
- 18:48:43: Phase 3: 53% done - estimated remaining time 2 hours, 48 minutes, 49 seconds
- 19:03:43: process known inodes and inode discovery - 197229632 of 349428160 inodes done
- 19:03:43: Phase 3: elapsed time 3 hours, 30 minutes, 8 seconds - processed 938592 inodes per minute
- 19:03:43: Phase 3: 56% done - estimated remaining time 2 hours, 42 minutes, 9 seconds
- 19:18:43: process known inodes and inode discovery - 205919616 of 349428160 inodes done
- 19:18:43: Phase 3: elapsed time 3 hours, 45 minutes, 8 seconds - processed 914656 inodes per minute
- 19:18:43: Phase 3: 58% done - estimated remaining time 2 hours, 36 minutes, 53 seconds
- 19:33:43: process known inodes and inode discovery - 214362560 of 349428160 inodes done
- 19:33:43: Phase 3: elapsed time 4 hours, 8 seconds - processed 892681 inodes per minute
- 19:33:43: Phase 3: 61% done - estimated remaining time 2 hours, 31 minutes, 18 seconds
- 19:48:43: process known inodes and inode discovery - 222800640 of 349428160 inodes done
- 19:48:43: Phase 3: elapsed time 4 hours, 15 minutes, 8 seconds - processed 873271 inodes per minute
- 19:48:43: Phase 3: 63% done - estimated remaining time 2 hours, 25 minutes
- 20:03:43: process known inodes and inode discovery - 232499072 of 349428160 inodes done
- 20:03:43: Phase 3: elapsed time 4 hours, 30 minutes, 8 seconds - processed 860682 inodes per minute
- 20:03:43: Phase 3: 66% done - estimated remaining time 2 hours, 15 minutes, 51 seconds
- 20:18:43: process known inodes and inode discovery - 241304832 of 349428160 inodes done
- 20:18:43: Phase 3: elapsed time 4 hours, 45 minutes, 8 seconds - processed 846287 inodes per minute
- 20:18:43: Phase 3: 69% done - estimated remaining time 2 hours, 7 minutes, 45 seconds
- 20:33:43: process known inodes and inode discovery - 248594688 of 349428160 inodes done
- 20:33:43: Phase 3: elapsed time 5 hours, 8 seconds - processed 828280 inodes per minute
- 20:33:43: Phase 3: 71% done - estimated remaining time 2 hours, 1 minute, 44 seconds
- 20:48:43: process known inodes and inode discovery - 256390464 of 349428160 inodes done
- 20:48:43: Phase 3: elapsed time 5 hours, 15 minutes, 8 seconds - processed 813593 inodes per minute
- 20:48:43: Phase 3: 73% done - estimated remaining time 1 hour, 54 minutes, 21 seconds
- 21:03:43: process known inodes and inode discovery - 268730560 of 349428160 inodes done
- 21:03:43: Phase 3: elapsed time 5 hours, 30 minutes, 8 seconds - processed 814006 inodes per minute
- 21:03:43: Phase 3: 76% done - estimated remaining time 1 hour, 39 minutes, 8 seconds
- 21:18:43: process known inodes and inode discovery - 278996288 of 349428160 inodes done
- 21:18:43: Phase 3: elapsed time 5 hours, 45 minutes, 8 seconds - processed 808372 inodes per minute
- 21:18:43: Phase 3: 79% done - estimated remaining time 1 hour, 27 minutes, 7 seconds
- 21:33:43: process known inodes and inode discovery - 289928000 of 349428160 inodes done
- 21:33:43: Phase 3: elapsed time 6 hours, 8 seconds - processed 805057 inodes per minute
- 21:33:43: Phase 3: 82% done - estimated remaining time 1 hour, 13 minutes, 54 seconds
- 21:48:43: process known inodes and inode discovery - 298228736 of 349428160 inodes done
- 21:48:43: Phase 3: elapsed time 6 hours, 15 minutes, 8 seconds - processed 794993 inodes per minute
- 21:48:43: Phase 3: 85% done - estimated remaining time 1 hour, 4 minutes, 24 seconds
- 22:03:43: process known inodes and inode discovery - 314882432 of 349428160 inodes done
- 22:03:43: Phase 3: elapsed time 6 hours, 30 minutes, 8 seconds - processed 807114 inodes per minute
- 22:03:43: Phase 3: 90% done - estimated remaining time 42 minutes, 48 seconds
- 22:18:43: process known inodes and inode discovery - 332009344 of 349428160 inodes done
- 22:18:43: Phase 3: elapsed time 6 hours, 45 minutes, 8 seconds - processed 819506 inodes per minute
- 22:18:43: Phase 3: 95% done - estimated remaining time 21 minutes, 15 seconds
- 22:33:43: process known inodes and inode discovery - 346065600 of 349428160 inodes done
- 22:33:43: Phase 3: elapsed time 7 hours, 8 seconds - processed 823704 inodes per minute
- 22:33:43: Phase 3: 99% done - estimated remaining time 4 minutes, 4 seconds
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- 22:39:44: process known inodes and inode discovery - 349428160 of 349428160 inodes done
- process newly discovered inodes...
- 22:39:44: process newly discovered inodes - 146 of 73 allocation groups done
libxfs_bcache: 0xe47d470
Max supported entries = 131072
Max utilized entries = 131072
Active entries = 131069
Hash table size = 16384
Hits = 53663876
Misses = 99522879
Hit ratio = 35.03
MRU 0 entries = 16 ( 0%)
MRU 1 entries = 0 ( 0%)
MRU 2 entries = 40 ( 0%)
MRU 3 entries = 556 ( 0%)
MRU 4 entries = 0 ( 0%)
MRU 5 entries = 0 ( 0%)
MRU 6 entries = 0 ( 0%)
MRU 7 entries = 28125 ( 21%)
MRU 8 entries = 0 ( 0%)
MRU 9 entries = 1446 ( 1%)
MRU 10 entries = 0 ( 0%)
MRU 11 entries = 81850 ( 62%)
MRU 12 entries = 0 ( 0%)
MRU 13 entries = 19036 ( 14%)
MRU 14 entries = 0 ( 0%)
MRU 15 entries = 0 ( 0%)
Hash buckets with 0 entries 15 ( 0%)
Hash buckets with 1 entries 100 ( 0%)
Hash buckets with 2 entries 390 ( 0%)
Hash buckets with 3 entries 719 ( 1%)
Hash buckets with 4 entries 1209 ( 3%)
Hash buckets with 5 entries 1573 ( 6%)
Hash buckets with 6 entries 1846 ( 8%)
Hash buckets with 7 entries 1996 ( 10%)
Hash buckets with 8 entries 1935 ( 11%)
Hash buckets with 9 entries 1710 ( 11%)
Hash buckets with 10 entries 1343 ( 10%)
Hash buckets with 11 entries 1088 ( 9%)
Hash buckets with 12 entries 789 ( 7%)
Hash buckets with 13 entries 588 ( 5%)
Hash buckets with 14 entries 387 ( 4%)
Hash buckets with 15 entries 273 ( 3%)
Hash buckets with 16 entries 169 ( 2%)
Hash buckets with 17 entries 99 ( 1%)
Hash buckets with 18 entries 66 ( 0%)
Hash buckets with 19 entries 37 ( 0%)
Hash buckets with 20 entries 25 ( 0%)
Hash buckets with 21 entries 10 ( 0%)
Hash buckets with 22 entries 7 ( 0%)
Hash buckets with 23 entries 2 ( 0%)
Hash buckets with 24 entries 4 ( 0%)
Hash buckets with >24 entries 4 ( 0%)
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- 22:39:51: setting up duplicate extent list - 73 of 73 allocation groups done
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 16
- agno = 64
- agno = 48
- agno = 32
- agno = 17
- agno = 65
- agno = 49
- agno = 33
- agno = 50
- agno = 66
- agno = 18
- agno = 34
- agno = 51
- agno = 67
- agno = 19
- agno = 52
- agno = 35
- agno = 53
- agno = 20
- agno = 68
- agno = 36
- agno = 54
- agno = 21
- agno = 37
- agno = 69
- agno = 22
- agno = 38
- agno = 55
- agno = 23
- agno = 56
- agno = 24
- agno = 57
- agno = 39
- agno = 25
- agno = 58
- agno = 40
- agno = 70
- agno = 41
- agno = 59
- agno = 71
- agno = 60
- agno = 42
- agno = 72
- agno = 61
- agno = 26
- agno = 43
- agno = 27
- agno = 44
- agno = 62
- agno = 28
- agno = 45
- agno = 63
- agno = 29
- agno = 46
- agno = 30
- agno = 47
- agno = 31
- 22:48:43: check for inodes claiming duplicate blocks - 20142720 of 349428160 inodes done
- 22:48:43: Phase 4: elapsed time 8 minutes, 59 seconds - processed 2242232 inodes per minute
- 22:48:43: Phase 4: 5% done - estimated remaining time 2 hours, 26 minutes, 51 seconds
- 23:03:43: check for inodes claiming duplicate blocks - 43836992 of 349428160 inodes done
- 23:03:43: Phase 4: elapsed time 23 minutes, 59 seconds - processed 1827810 inodes per minute
- 23:03:43: Phase 4: 12% done - estimated remaining time 2 hours, 47 minutes, 11 seconds
- 23:18:43: check for inodes claiming duplicate blocks - 67716672 of 349428160 inodes done
- 23:18:43: Phase 4: elapsed time 38 minutes, 59 seconds - processed 1737067 inodes per minute
- 23:18:43: Phase 4: 19% done - estimated remaining time 2 hours, 42 minutes, 10 seconds
- 23:33:43: check for inodes claiming duplicate blocks - 81305728 of 349428160 inodes done
- 23:33:43: Phase 4: elapsed time 53 minutes, 59 seconds - processed 1506126 inodes per minute
- 23:33:43: Phase 4: 23% done - estimated remaining time 2 hours, 58 minutes, 1 second
- 23:48:43: check for inodes claiming duplicate blocks - 91484608 of 349428160 inodes done
- 23:48:43: Phase 4: elapsed time 1 hour, 8 minutes, 59 seconds - processed 1326184 inodes per minute
- 23:48:43: Phase 4: 26% done - estimated remaining time 3 hours, 14 minutes, 30 seconds
- 00:03:43: check for inodes claiming duplicate blocks - 101952960 of 349428160 inodes done
- 00:03:43: Phase 4: elapsed time 1 hour, 23 minutes, 59 seconds - processed 1213966 inodes per minute
- 00:03:43: Phase 4: 29% done - estimated remaining time 3 hours, 23 minutes, 51 seconds
- 00:18:43: check for inodes claiming duplicate blocks - 116626240 of 349428160 inodes done
- 00:18:43: Phase 4: elapsed time 1 hour, 38 minutes, 59 seconds - processed 1178241 inodes per minute
- 00:18:43: Phase 4: 33% done - estimated remaining time 3 hours, 17 minutes, 35 seconds
- 00:33:43: check for inodes claiming duplicate blocks - 130431936 of 349428160 inodes done
- 00:33:43: Phase 4: elapsed time 1 hour, 53 minutes, 59 seconds - processed 1144307 inodes per minute
- 00:33:43: Phase 4: 37% done - estimated remaining time 3 hours, 11 minutes, 22 seconds
- 00:48:43: check for inodes claiming duplicate blocks - 142498304 of 349428160 inodes done
- 00:48:43: Phase 4: elapsed time 2 hours, 8 minutes, 59 seconds - processed 1104780 inodes per minute
- 00:48:43: Phase 4: 40% done - estimated remaining time 3 hours, 7 minutes, 18 seconds
- 01:03:43: check for inodes claiming duplicate blocks - 153540160 of 349428160 inodes done
- 01:03:43: Phase 4: elapsed time 2 hours, 23 minutes, 59 seconds - processed 1066374 inodes per minute
- 01:03:43: Phase 4: 43% done - estimated remaining time 3 hours, 3 minutes, 41 seconds
- 01:18:43: check for inodes claiming duplicate blocks - 163892800 of 349428160 inodes done
- 01:18:43: Phase 4: elapsed time 2 hours, 38 minutes, 59 seconds - processed 1030880 inodes per minute
- 01:18:43: Phase 4: 46% done - estimated remaining time 2 hours, 59 minutes, 58 seconds
entry "33" in shortform directory 631750746 references free inode 631750772
junking entry "33" in directory inode 631750746
entry "Gui" in shortform directory 631750749 references free inode 631750766
junking entry "Gui" in directory inode 631750749
entry ".." at block 0 offset 32 in directory inode 631750774 references non-existent inode 631748299
clearing inode number in entry at offset 32...
entry "cwd" at block 0 offset 48 in directory inode 631750774 references non-existent inode 631748521
clearing inode number in entry at offset 48...
entry "exe" at block 0 offset 64 in directory inode 631750774 references non-existent inode 631748524
clearing inode number in entry at offset 64...
entry "root" at block 0 offset 80 in directory inode 631750774 references non-existent inode 631748527
clearing inode number in entry at offset 80...
entry "fd" at block 0 offset 96 in directory inode 631750774 references non-existent inode 631748530
clearing inode number in entry at offset 96...
entry "fdinfo" at block 0 offset 112 in directory inode 631750774 references non-existent inode 631748533
clearing inode number in entry at offset 112...
entry "ns" at block 0 offset 136 in directory inode 631750774 references non-existent inode 631748535
clearing inode number in entry at offset 136...
entry "auxv" at block 0 offset 176 in directory inode 631750774 references non-existent inode 631750987
clearing inode number in entry at offset 176...
entry "cgroup" at block 0 offset 224 in directory inode 631750774 references non-existent inode 631750993
clearing inode number in entry at offset 224...
entry "clear_refs" at block 0 offset 280 in directory inode 631750774 references non-existent inode 631750996
clearing inode number in entry at offset 280...
entry "cmdline" at block 0 offset 336 in directory inode 631750774 references non-existent inode 631750999
clearing inode number in entry at offset 336...
entry "comm" at block 0 offset 384 in directory inode 631750774 references non-existent inode 631751001
clearing inode number in entry at offset 384...
entry "cpuset" at block 0 offset 432 in directory inode 631750774 references non-existent inode 631751003
clearing inode number in entry at offset 432...
entry "environ" at block 0 offset 488 in directory inode 631750774 references non-existent inode 631751005
clearing inode number in entry at offset 488...
entry "io" at block 0 offset 536 in directory inode 631750774 references non-existent inode 631751007
clearing inode number in entry at offset 536...
no .. entry for directory 631750774
- 01:33:43: check for inodes claiming duplicate blocks - 171412608 of 349428160 inodes done
- 01:33:43: Phase 4: elapsed time 2 hours, 53 minutes, 59 seconds - processed 985224 inodes per minute
- 01:33:43: Phase 4: 49% done - estimated remaining time 3 hours, 41 seconds
- 01:48:43: check for inodes claiming duplicate blocks - 181038208 of 349428160 inodes done
- 01:48:43: Phase 4: elapsed time 3 hours, 8 minutes, 59 seconds - processed 957958 inodes per minute
- 01:48:43: Phase 4: 51% done - estimated remaining time 2 hours, 55 minutes, 46 seconds
- 02:03:43: check for inodes claiming duplicate blocks - 193503552 of 349428160 inodes done
- 02:03:43: Phase 4: elapsed time 3 hours, 23 minutes, 59 seconds - processed 948624 inodes per minute
- 02:03:43: Phase 4: 55% done - estimated remaining time 2 hours, 44 minutes, 22 seconds
- 02:18:43: check for inodes claiming duplicate blocks - 202466624 of 349428160 inodes done
- 02:18:43: Phase 4: elapsed time 3 hours, 38 minutes, 59 seconds - processed 924575 inodes per minute
- 02:18:43: Phase 4: 57% done - estimated remaining time 2 hours, 38 minutes, 57 seconds
- 02:33:43: check for inodes claiming duplicate blocks - 210639936 of 349428160 inodes done
- 02:33:43: Phase 4: elapsed time 3 hours, 53 minutes, 59 seconds - processed 900234 inodes per minute
- 02:33:43: Phase 4: 60% done - estimated remaining time 2 hours, 34 minutes, 10 seconds
- 02:48:43: check for inodes claiming duplicate blocks - 220841728 of 349428160 inodes done
- 02:48:43: Phase 4: elapsed time 4 hours, 8 minutes, 59 seconds - processed 886973 inodes per minute
- 02:48:43: Phase 4: 63% done - estimated remaining time 2 hours, 24 minutes, 58 seconds
- 03:03:43: check for inodes claiming duplicate blocks - 231019200 of 349428160 inodes done
- 03:03:43: Phase 4: elapsed time 4 hours, 23 minutes, 59 seconds - processed 875127 inodes per minute
- 03:03:43: Phase 4: 66% done - estimated remaining time 2 hours, 15 minutes, 18 seconds
- 03:18:43: check for inodes claiming duplicate blocks - 242873856 of 349428160 inodes done
- 03:18:43: Phase 4: elapsed time 4 hours, 38 minutes, 59 seconds - processed 870567 inodes per minute
- 03:18:43: Phase 4: 69% done - estimated remaining time 2 hours, 2 minutes, 23 seconds
- 03:33:43: check for inodes claiming duplicate blocks - 253174080 of 349428160 inodes done
- 03:33:43: Phase 4: elapsed time 4 hours, 53 minutes, 59 seconds - processed 861185 inodes per minute
- 03:33:43: Phase 4: 72% done - estimated remaining time 1 hour, 51 minutes, 46 seconds
- 03:48:43: check for inodes claiming duplicate blocks - 264862272 of 349428160 inodes done
- 03:48:43: Phase 4: elapsed time 5 hours, 8 minutes, 59 seconds - processed 857205 inodes per minute
- 03:48:43: Phase 4: 75% done - estimated remaining time 1 hour, 38 minutes, 39 seconds
- 04:03:43: check for inodes claiming duplicate blocks - 278814272 of 349428160 inodes done
- 04:03:43: Phase 4: elapsed time 5 hours, 23 minutes, 59 seconds - processed 860582 inodes per minute
- 04:03:43: Phase 4: 79% done - estimated remaining time 1 hour, 22 minutes, 3 seconds
- 04:18:43: check for inodes claiming duplicate blocks - 290784768 of 349428160 inodes done
- 04:18:43: Phase 4: elapsed time 5 hours, 38 minutes, 59 seconds - processed 857814 inodes per minute
- 04:18:43: Phase 4: 83% done - estimated remaining time 1 hour, 8 minutes, 21 seconds
- 04:33:43: check for inodes claiming duplicate blocks - 299526208 of 349428160 inodes done
- 04:33:43: Phase 4: elapsed time 5 hours, 53 minutes, 59 seconds - processed 846159 inodes per minute
- 04:33:43: Phase 4: 85% done - estimated remaining time 58 minutes, 58 seconds
- 04:48:43: check for inodes claiming duplicate blocks - 318248896 of 349428160 inodes done
- 04:48:43: Phase 4: elapsed time 6 hours, 8 minutes, 59 seconds - processed 862502 inodes per minute
- 04:48:43: Phase 4: 91% done - estimated remaining time 36 minutes, 8 seconds
- 05:03:43: check for inodes claiming duplicate blocks - 338873536 of 349428160 inodes done
- 05:03:43: Phase 4: elapsed time 6 hours, 23 minutes, 59 seconds - processed 882521 inodes per minute
- 05:03:43: Phase 4: 96% done - estimated remaining time 11 minutes, 57 seconds
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- 05:12:27: check for inodes claiming duplicate blocks - 349428160 of 349428160 inodes done
libxfs_bcache: 0xe47d470
Max supported entries = 131072
Max utilized entries = 131072
Active entries = 131056
Hash table size = 16384
Hits = 106918098
Misses = 200477108
Hit ratio = 34.78
MRU 0 entries = 13 ( 0%)
MRU 1 entries = 0 ( 0%)
MRU 2 entries = 27 ( 0%)
MRU 3 entries = 1078 ( 0%)
MRU 4 entries = 0 ( 0%)
MRU 5 entries = 14 ( 0%)
MRU 6 entries = 0 ( 0%)
MRU 7 entries = 27152 ( 20%)
MRU 8 entries = 0 ( 0%)
MRU 9 entries = 2834 ( 2%)
MRU 10 entries = 0 ( 0%)
MRU 11 entries = 48679 ( 37%)
MRU 12 entries = 0 ( 0%)
MRU 13 entries = 51259 ( 39%)
MRU 14 entries = 0 ( 0%)
MRU 15 entries = 0 ( 0%)
Hash buckets with 0 entries 16 ( 0%)
Hash buckets with 1 entries 111 ( 0%)
Hash buckets with 2 entries 355 ( 0%)
Hash buckets with 3 entries 766 ( 1%)
Hash buckets with 4 entries 1162 ( 3%)
Hash buckets with 5 entries 1607 ( 6%)
Hash buckets with 6 entries 1897 ( 8%)
Hash buckets with 7 entries 1886 ( 10%)
Hash buckets with 8 entries 1901 ( 11%)
Hash buckets with 9 entries 1720 ( 11%)
Hash buckets with 10 entries 1410 ( 10%)
Hash buckets with 11 entries 1124 ( 9%)
Hash buckets with 12 entries 811 ( 7%)
Hash buckets with 13 entries 557 ( 5%)
Hash buckets with 14 entries 399 ( 4%)
Hash buckets with 15 entries 244 ( 2%)
Hash buckets with 16 entries 155 ( 1%)
Hash buckets with 17 entries 120 ( 1%)
Hash buckets with 18 entries 50 ( 0%)
Hash buckets with 19 entries 40 ( 0%)
Hash buckets with 20 entries 29 ( 0%)
Hash buckets with 21 entries 8 ( 0%)
Hash buckets with 22 entries 7 ( 0%)
Hash buckets with 23 entries 3 ( 0%)
Hash buckets with 24 entries 2 ( 0%)
Hash buckets with >24 entries 4 ( 0%)
Phase 5 - rebuild AG headers and trees...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- 05:18:43: rebuild AG headers and trees - 23 of 73 allocation groups done
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- agno = 32
- agno = 33
- agno = 34
- agno = 35
- agno = 36
- agno = 37
- agno = 38
- agno = 39
- agno = 40
- agno = 41
- agno = 42
- agno = 43
- agno = 44
- agno = 45
- agno = 46
- agno = 47
- agno = 48
- agno = 49
- agno = 50
- agno = 51
- agno = 52
- agno = 53
- agno = 54
- agno = 55
- agno = 56
- agno = 57
- agno = 58
- agno = 59
- agno = 60
- agno = 61
- agno = 62
- agno = 63
- agno = 64
- agno = 65
- agno = 66
- agno = 67
- agno = 68
- agno = 69
- agno = 70
- agno = 71
- agno = 72
- 05:28:39: rebuild AG headers and trees - 73 of 73 allocation groups done
- reset superblock...
libxfs_bcache: 0xe47d470
Max supported entries = 131072
Max utilized entries = 131072
Active entries = 131046
Hash table size = 16384
Hits = 106918098
Misses = 200590606
Hit ratio = 34.77
MRU 0 entries = 3 ( 0%)
MRU 1 entries = 0 ( 0%)
MRU 2 entries = 27 ( 0%)
MRU 3 entries = 1078 ( 0%)
MRU 4 entries = 0 ( 0%)
MRU 5 entries = 14 ( 0%)
MRU 6 entries = 0 ( 0%)
MRU 7 entries = 27152 ( 20%)
MRU 8 entries = 0 ( 0%)
MRU 9 entries = 2834 ( 2%)
MRU 10 entries = 0 ( 0%)
MRU 11 entries = 48679 ( 37%)
MRU 12 entries = 0 ( 0%)
MRU 13 entries = 51259 ( 39%)
MRU 14 entries = 0 ( 0%)
MRU 15 entries = 0 ( 0%)
Hash buckets with 0 entries 16 ( 0%)
Hash buckets with 1 entries 111 ( 0%)
Hash buckets with 2 entries 354 ( 0%)
Hash buckets with 3 entries 767 ( 1%)
Hash buckets with 4 entries 1162 ( 3%)
Hash buckets with 5 entries 1606 ( 6%)
Hash buckets with 6 entries 1897 ( 8%)
Hash buckets with 7 entries 1888 ( 10%)
Hash buckets with 8 entries 1904 ( 11%)
Hash buckets with 9 entries 1719 ( 11%)
Hash buckets with 10 entries 1409 ( 10%)
Hash buckets with 11 entries 1124 ( 9%)
Hash buckets with 12 entries 810 ( 7%)
Hash buckets with 13 entries 556 ( 5%)
Hash buckets with 14 entries 399 ( 4%)
Hash buckets with 15 entries 244 ( 2%)
Hash buckets with 16 entries 155 ( 1%)
Hash buckets with 17 entries 120 ( 1%)
Hash buckets with 18 entries 50 ( 0%)
Hash buckets with 19 entries 40 ( 0%)
Hash buckets with 20 entries 29 ( 0%)
Hash buckets with 21 entries 8 ( 0%)
Hash buckets with 22 entries 7 ( 0%)
Hash buckets with 23 entries 3 ( 0%)
Hash buckets with 24 entries 2 ( 0%)
Hash buckets with >24 entries 4 ( 0%)
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- agno = 0
- 05:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 05:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done
entry "Roboto-Thin-webfont.svg" in dir inode 358215663 inconsistent with .. value (631750763) in ino 631750769
will clear entry "Roboto-Thin-webfont.svg"
rebuilding directory inode 358215663
entry "slider.css.svn-base" in dir inode 358215673 inconsistent with .. value (631751273) in ino 631750763
will clear entry "slider.css.svn-base"
rebuilding directory inode 358215673
entry "Roboto-Thin-webfont.svg" in dir inode 359524752 inconsistent with .. value (631750763) in ino 631750769
will clear entry "Roboto-Thin-webfont.svg"
rebuilding directory inode 359524752
entry "slider.css.svn-base" in dir inode 359524762 inconsistent with .. value (631751273) in ino 631750763
will clear entry "slider.css.svn-base"
rebuilding directory inode 359524762
entry "slider.css.svn-base" in dir inode 363661536 inconsistent with .. value (631751273) in ino 631750763
will clear entry "slider.css.svn-base"
rebuilding directory inode 363661536
entry "Roboto-Thin-webfont.svg" in dir inode 363661680 inconsistent with .. value (631750763) in ino 631750769
will clear entry "Roboto-Thin-webfont.svg"
rebuilding directory inode 363661680
entry "Roboto-Thin-webfont.svg" in dir inode 366160230 inconsistent with .. value (631750763) in ino 631750769
will clear entry "Roboto-Thin-webfont.svg"
rebuilding directory inode 366160230
entry "slider.css.svn-base" in dir inode 366160236 inconsistent with .. value (631751273) in ino 631750763
will clear entry "slider.css.svn-base"
rebuilding directory inode 366160236
- 06:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done
entry "Roboto-Thin-webfont.svg" in dir inode 387225099 inconsistent with .. value (631750763) in ino 631750769
will clear entry "Roboto-Thin-webfont.svg"
rebuilding directory inode 387225099
entry "slider.css.svn-base" in dir inode 387225105 inconsistent with .. value (631751273) in ino 631750763
will clear entry "slider.css.svn-base"
rebuilding directory inode 387225105
entry "Roboto-Thin-webfont.svg" in dir inode 389807889 inconsistent with .. value (631750763) in ino 631750769
will clear entry "Roboto-Thin-webfont.svg"
rebuilding directory inode 389807889
entry "slider.css.svn-base" in dir inode 389807894 inconsistent with .. value (631751273) in ino 631750763
will clear entry "slider.css.svn-base"
rebuilding directory inode 389807894
- 06:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 06:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 06:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 07:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 07:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 07:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 07:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 08:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 08:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 08:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 08:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 09:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 09:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 09:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 09:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 10:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 10:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 10:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 10:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 11:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 11:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 11:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 11:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 12:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 12:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 12:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 12:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 13:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 13:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 13:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 13:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 14:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 14:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 14:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 14:48:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 15:03:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 15:18:43: rebuild AG headers and trees - 73 of 73 allocation groups done
- 15:33:43: rebuild AG headers and trees - 73 of 73 allocation groups done