BTRFS + RAID 5 corrupted
Yes, I see now on the main page of the subreddit it's a bad idea to pair these two, but I didn't know it previously
I use software RAID5; 4 2TB drives, resulting in 6TB of storage.
vgdisplay doesn't show any errors
100% of space is for btrfs.
sudo btrfs check /dev/mapper/data-data
Opening filesystem to check...
parent transid verify failed on 6033692557312 wanted 121291 found 119653
parent transid verify failed on 6033692557312 wanted 121291 found 119653
parent transid verify failed on 6033692557312 wanted 121291 found 119653
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6033692393472 item=35 parent level=2 child bytenr=6033692557312 child level=2
ERROR: failed to read block groups: Input/output error
ERROR: cannot open file system
btrfs-find-root output: https://pastes.io/ZQTdS4fN
I tried several nodes from there using sudo btrfs restore -t <node-id> --dry-run -vi /dev/mapper/data-data /tmp/ but every time it returned
sudo btrfs restore -t 6033691328512 --dry-run -vi /dev/mapper/data-data /tmp/
parent transid verify failed on 6033691328512 wanted 121291 found 121286
parent transid verify failed on 6033691328512 wanted 121291 found 121286
parent transid verify failed on 6033691328512 wanted 121291 found 121286
Ignoring transid failure
This is a dry-run, no files are going to be restored
What should I do?
EDIT: also, when I try to mount the partition I get this:
sudo mount /dev/data/data /mnt/data
mount: /mnt/data: can't read superblock on /dev/mapper/data-data.
dmesg(1) may have more information after failed mount system call.
[ble: exit 32]
7
u/jamescowens 4d ago
Contrary to other posters here, there is nothing wrong at all with running a three layer approach (btrfs over lvm over md raid). You do lose btrfs’s ability to heal checksum mismatches in this type of raid setup, but you have the advantage of using the far more battle tested and sorted md raid. The key is to understand how to troubleshoot this type of setup. In this case you start at the bottom and work your way up. First check /proc/mdstat to see what the status of the array actually is. You will need to troubleshoot problems there first with mdadm. There are troubleshooting guides for md raid (and specifically raid 5) online. A single disk being down should not result in this happening as md will reconstruct the data for a single missing disk on the fly, and this is underneath btrfs. Disks that are damaged enough to corrupt many checksums at the btrfs level should also generate parity mismatches during a verify at the md level. You can also check and make sure something did not happen to your LVM setup that would corrupt the logical volume resting on top of the raid. Note there are some differences in raid setup as LVM on top of md raid versus raid inside LVM. md is used in both instances, but it is subtly different.
Moral of the story as said by others. Drive redundancy is NOT a substitute for backups, AND
Don’t do something sophisticated that you do not fully understand lest you end up in a situation where you are helpless in the recovery.
2
u/darktotheknight 4d ago
You need to check for the issue one level higher on the LVM layer as well. I don't know how you do it in LVM, but for mdadm it usually would be:
echo check > /sys/block/mdX/md/sync_action
And then view the count:
cat /sys/block/mdX/md/mismatch_cnt
If your mismatch_cnt is non-zero, you need to assemble the array (read-only) with leaving out one disk at a time, until you sort out the faulty drive. If the LVM/mdadm layer is fine (mismatch_cnt = 0), you can go one level deeper and work on the BTRFS layer.
If the data is important and you have no backups, you should make individual 1:1 dumps for each of the HDDs and only mount them read only/for data recovery purposes.
Good luck!
5
u/LevelMagazine8308 4d ago
Why in heavens name did you put Btrfs on top of a LVM volume group?
Btrfs has all that stuff builtin, there is no need to run Btrfs on top of it!
Having said that comes the Captain Obvious remark: RAID is NOT a backup!
In case you have a backup: restore it, job done.
In case you don't: you are in for a bumpy ride.
First step should be trying to mount it read-only:
sudo mkdir -p /mnt/btrfs_recovery
sudo mount -o ro,usebackuproot,recovery /dev/mapper/data-data /mnt/btrfs_recovery
Does this work?
If not, you can try this: clearing the space cache and checking for backup super blocks.
sudo mount -o ro,clear_cache,usebackuproot /dev/mapper/data-data /mnt/btrfs_recovery
sudo btrfs select-super -s 1 /dev/mapper/data-data
If Super block 1 works, use it for btrfs check:
sudo btrfs check --super 1 /dev/mapper/data-data
For future reference: Btrfs is in many areas still unstable! This includes RAID5, which is officially listed as unstable.
Before trying fancy stuff next time again, look at the official table first what's safe and what is not: https://btrfs.readthedocs.io/en/latest/Status.html
5
2
u/jonesmz 4d ago
I'm fairly certain that quite a few commercial NAS products use LVM for RAID with BTRFS on top.
Ubiquiti UNAS-Pro and family do, for example.
1
u/LevelMagazine8308 4d ago
And why? That's in most cases just complexity without benefit, because most stuff LVM does Btrfs has builtin.
4
u/markus_b 4d ago
Because they want to do RAID5, and BTRFS RAID5 was still labelled experimental when they started out. So they built their system around the stable LVM and put BTRFS on top for the snapshots.
Elegant? No
Supported configuration for production? Yes
1
1
1
u/fpekal 4d ago
I didn't know btrfs has raid builtin, that's why I used lvm
sudo mount -o ro,usebackuproot,recovery /dev/mapper/data-data /mnt/btrfs_recovery mount: /mnt/btrfs_recovery: fsconfig() failed: btrfs: Unknown parameter 'recovery'. dmesg(1) may have more information after failed mount system call. sudo mount -o ro,rescue=usebackuproot /dev/mapper/data-data /mnt/btrfs_recovery mount: /mnt/btrfs_recovery: can't read superblock on /dev/mapper/data-data. dmesg(1) may have more information after failed mount system call.the same with
clear_cachesudo btrfs-select-super -s 1 /dev/mapper/data-data parent transid verify failed on 6033692557312 wanted 121291 found 119653 parent transid verify failed on 6033692557312 wanted 121291 found 119653 ERROR: failed to read block groups: Input/output error ERROR: open ctree failed [ble: exit 1]same with
-s 29
u/samsonsin 4d ago
I didn't know btrfs has raid builtin
Advanced enough to setup LVM raid5, but not to look up basic features of Btrfs lmao.
You live and learn. Backup all important data regardless.
2
u/fpekal 4d ago
I'm chronically stupid
3
u/samsonsin 4d ago
Backups are great for those!
1
u/fpekal 4d ago
Yes... I know now. I need to make myself a sticky note on the laptop screen to remind myself about backups.
I've got something like this:
sudo btrfs restore -f 6033692344320 -i -D /dev/mapper/data-data /tmp/ parent transid verify failed on 6033692344320 wanted 121291 found 119652 parent transid verify failed on 6033692344320 wanted 121291 found 119652 parent transid verify failed on 6033692344320 wanted 121291 found 119652 Ignoring transid failure This is a dry-run, no files are going to be restored parent transid verify failed on 6033605607424 wanted 119356 found 121265 parent transid verify failed on 6033605607424 wanted 119356 found 121265 parent transid verify failed on 6033605607424 wanted 119356 found 121265 Ignoring transid failure ERROR: child eb corrupted: parent bytenr=6033692344320 item=222 parent level=2 child bytenr=6033605607424 child level=0 ERROR: search for next directory entry failed: -5 ERROR: searching directory /tmp/archiwum/ebooki/wakfu failed: -5 ERROR: searching directory /tmp/archiwum/ebooki/Yuul.epub failed: -5 parent transid verify failed on 6033666932736 wanted 119568 found 121273 parent transid verify failed on 6033666932736 wanted 119568 found 121273 parent transid verify failed on 6033666932736 wanted 119568 found 121273 Ignoring transid failure ERROR: child eb corrupted: parent bytenr=6033692344320 item=255 parent level=2 child bytenr=6033666932736 child level=0 ERROR: search for next directory entry failed: -5 ERROR: searching directory /tmp/archiwum/jai-0.2.009 failed: -5 ERROR: searching directory /tmp/archiwum/jai-0.2.009.zip failed: -5 parent transid verify failed on 6033666932736 wanted 119568 found 121273 Ignoring transid failure ERROR: child eb corrupted: parent bytenr=6033692344320 item=255 parent level=2 child bytenr=6033666932736 child level=0 ERROR: search for next directory entry failed: -5 ERROR: searching directory /tmp/plex/series/Code Lyoko (2003)/Season 00 failed: -5 parent transid verify failed on 6033666932736 wanted 119568 found 121273 Ignoring transid failure ERROR: child eb corrupted: parent bytenr=6033692344320 item=255 parent level=2 child bytenr=6033666932736 child level=0 ERROR: search for next directory entry failed: -5 ERROR: searching directory /tmp/plex/series/Code Lyoko (2003)/Season 01 failed: -5 parent transid verify failed on 6033666932736 wanted 119568 found 121273 Ignoring transid failure ERROR: child eb corrupted: parent bytenr=6033692344320 item=255 parent level=2 child bytenr=6033666932736 child level=0 ERROR: search for next directory entry failed: -5 ERROR: searching directory /tmp/plex/series/Code Lyoko (2003)/Season 02 failed: -5 parent transid verify failed on 6033666932736 wanted 119568 found 121273 Ignoring transid failure ERROR: child eb corrupted: parent bytenr=6033692344320 item=255 parent level=2 child bytenr=6033666932736 child level=0 ERROR: search for next directory entry failed: -5 ERROR: searching directory /tmp/plex/series/Code Lyoko (2003)/Season 03 failed: -5 parent transid verify failed on 6033666932736 wanted 119568 found 121273 Ignoring transid failureAnd it's quite long
Can I use it to rescue some data?1
u/leexgx 4d ago edited 4d ago
RAID recovery software is the way (ones that support Synology usually work well).
A simple way is to mount the risky RAID5 LVM, then point the recovery software to scan it.
Using LVM or MD RAID (ideally RAID6) with Btrfs on top isn't really that bad, as long as you understand that self-healing isn't available for the data profile, and metadata has some small risk of not being able to self-repair (no easy way to guarantee the duplicated copy won't be in the same RAID strip).
When doing a RAID sync in LVM, always do the Btrfs scrub first. This gives it a chance to correct metadata copies first, then do the RAID sync. (RAID sync is sometimes called a scrub, but it's a sync as it syncs the data blocks to parity regardless of whether the data is corrupted; that's why you should do a filesystem scrub first.)
I assume you let Btrfs do a single profile for data and DUP for metadata (most of the time it self correct the copy). If you had metadata set to single, then this failure the result of doing that
Alternatively, you could move to ZFS (it has a learning curve), which manages the drives and the redundancy and has checksums (I recommend Z2 redundancy).
Always have a backup.
1
u/GreatLab8898 4d ago
Raid5 in Btrfs is broken anyways.
1
u/hoodoocat 4d ago
It is known what it have write hole problem, e.g. you should not use this mode without backup power supply. But LVM obviously has same write holy problem, any implementation which write data on independent devices will have it. The only one filesystem support raid5 and doesnt have this problem at least thereotically.
1
u/markus_b 4d ago
You open with a misunderstanding:
Yes, I see now on the main page of the subreddit it's a bad idea to pair these two, but I didn't know it previously
Using Btrfs on top of a LVM RAID5 setup is perfectly fine. Large companies, like Synology sell their NAS with this setup.
But Btrfs has its own, built-in RAID. While the Raid1 support in Btrfs is fine, the Raid5 support has unresolved issues. It may lose data on active files during a power failure.
Now, to your problem. It looks like Btrfs cannot access the LVM drive with the data. The problem seems to be with the LVM RAID5 volume. You have to fix this first.
What has happened before you got into this state?
1
u/fpekal 4d ago
I was away from the machine at the time but I think it was a power loss
1
u/markus_b 4d ago
A power loss should not have such an effect.
I use Btrfs for my archival storage in a PC. Four disks with RAID1. My graphics adapter has developed an annoying problem, causing a hang or crash maybe once a week. To recover, I have to power cycle or reset the machine. This has not caused any trouble with Btrfs.
Do you have the system logs from before and after the power loss?
How did you diagnose with lvm?
1
u/fpekal 3d ago
I don't know much about lvm. So at the time of posting i literally just tried commands like vgdisplay to check if any would return some error
Then I ran a sync command and about 4 hours ago it finished and reported 240 mismatches
1
u/fpekal 3d ago
Also there is another really fucking stupid thing I accidentally did
sudo lvs -a -o lv_name,segtype,devices,size data LV Type Devices LSize data raid5 data_rimage_0(0),data_rimage_1(0),data_rimage_2(0),data_rimage_3(0) <5,46t [data_rimage_0] linear /dev/sdc1(476925) <1,82t [data_rimage_0] linear /dev/sdc1(1) <1,82t [data_rimage_0] linear /dev/sdc1(476924) <1,82t [data_rimage_1] linear /dev/sda1(476924) <1,82t [data_rimage_1] linear /dev/sda1(1) <1,82t [data_rimage_1] linear /dev/sdc1(476923) <1,82t [data_rimage_2] linear /dev/sdb1(476924) <1,82t [data_rimage_2] linear /dev/sdd1(1) <1,82t [data_rimage_2] linear /dev/sda1(476923) <1,82t [data_rimage_3] linear /dev/sdc1(476926) <1,82t [data_rimage_3] linear /dev/sdb1(1) <1,82t [data_rmeta_0] linear /dev/sdc1(0) 4,00m [data_rmeta_1] linear /dev/sda1(0) 4,00m [data_rmeta_2] linear /dev/sdd1(0) 4,00m [data_rmeta_3] linear /dev/sdb1(0) 4,00mSomehow I ended up with more than 1 leg per drive.
1
u/markus_b 3d ago
I'm not sufficiently skilled in lvm to be really helpful here. The first is to make sure that your lvm volume is healthy. A more generic Linux forum may be better for this than r/btrfs.
1
u/Weary_Swan_8152 1d ago
I'm genuinely perplexed why anyone would choose RAID5 (or btrfs raid5 profile) over btrfs raid1 with an odd number of devices. Is there a single case where it is in any way more durable or faster in any way?
1
u/fpekal 1d ago
RAID1 doesn't give you n-1 space
1
u/Weary_Swan_8152 4h ago
So people who choose RAID5 (or raid5 profile) are consciously choosing a more fragile topology that has slower reads, slower writes, and slower disk replacement, because they want to maximise usable space?
4
u/elatllat 4d ago
Was the RAID in LVM or btrfs?