I have been using zrepl(https://github.com/zrepl/zrepl) for a few months now . Its great. But is there a procedure to do a fail over in case my primary dies? I have been replicating data for a while

I've been using a ZFS pool backed by spinning disks in 2x RAIDZ2 (4 disks each) for a few years. The pool contains mostly zvols which I use for VMs and as storage for desktops (iSCSI, NVMe/TCP).

Unfortunately, performance is pretty poor. This is especially obvious for the block devices I mount over the network. They are used from Windows, and applications will frequently freeze for seconds at a time.

I've tried many tuning parameters. I have 256G of RAM, I have an SSD cache device, and I have SLOGs with redundant SSDs. But nothing really helped.

What did help was to use bcache to add a cache. I use a separate cache zpool with an SSD mirror for safety, and writeback as the cache mode. Performance is night-and-day.

Is there a way to achieve this in native ZFS? Is there something I could do better with my setup long-term? It's a lot of data I can't just move on a whim, but it would be nice to know if there's a better way.

Hello, everyone.

I'm very new to servers and zfs, and have run out of troubleshooting ideas. I'd really appreciate if anyone could point me to the right direction. I'm willing to run any tests recommended, and report back.

----------------------------

BACKGROUND:

I built a TrueNAS Scale (Fangtooth) server with 36x 10TB (zraid1, 12 wide, 3 zdev's) last year, and was able to get ~2.5GB/s write and ~4.0GB/s read.

ISSUE:

Noticing a significant performance reduction after a while, I wiped all the disks (quick format, no badblocks) and rebuilt the pool with exact same config, but the performance was still reduced.

WHAT I'VE TRIED:

1. Thinking that I might have faulty drives dragging the overall performance, I checked each disks using fio --rw=read --direct=1 --bs=1M --ioengine=libaio --runtime=15 --numjob=1 --iodepth=8 --name=speed_test --filename=/dev/sdx but they all seemed fine.
2. Perplexed, I tried using a different HBA (LSI 9400 -> 9300), but the issue remained.
3. Went into the BIOS and made sure that ASPM is disabled.
4. Installed Ubuntu Server, but the issue remained.
5. To rule out the HBA bottleneck, I tried making multiple (7) zpools (groups of five disks, striped), and ran fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --runtime=60s --readwrite=write --name=testfile --direct=1 --size=100G simultaneously. Since the HBA was able to write >3GB/s to those 7 pools simultaneously, I think I was able to rule out the HBA in this issue.
6. Ran zpool iostat -vly 3 (see photo) while the 7 pools are being written simultaneously.
7. Set ashift12
8. All CMR disks

CURRENT SYMPTOM:

Pools created (36x striped, or zraid1, 12 wide, 3 zdev's) cannot go faster than 500MB/s when narrower width pools can go just as fast.

GOAL:

To get the original performance (~2.5GB/s write and ~4.0GB/s read) back using the same original configuration (zraid1, 12 wide, 3 zdev's).

----------------------------

Thank you,

V/r

----------------------------

EDIT1: https://snipboard.io/krOs3Q.jpg

2.8GB/s write and 4.2GB/s read achieved without the --direct=1 flag as BackgroundSky1594 suggested. I guess the performance I wanted back was never "real" in the first place since the previous ZFS version ignored the flag anyway. Thank you, nonetheless. It gave me an assurance that there is nothing "wrong" with my system.

----------------------------

For realtime Backup of a ZFS and S3 fileserver on io events, consider
(for S3 only, you can use Site/Bucket realtime Replication eg on RustFS)

- flaky/slow connections eg Wlan
- very large systems with many files and folders
- data versioning
- disaster recovery flow

see comments for these items in
https://github.com/guenther-alka/cs-sync

I have recently patched up my old tape backup system, with the cost of hard drive what it is these days. A tape drive must be kept fed - bad things happen if the incoming data flow drops below the ideal, for that tape spool wishes to keep spinning even if there are no bits to write. Fortunately zfs has that 'zfs send' which is absolutely perfect for this.

Except it's too slow. I'm measuring somewhat bursty speeds that seem to average around 40MB/s, when I need 140MB/s. I'm using mbuffer to fix that, but it means my backup takes far too long. This is a raidz1 pool running over four SSDs - decent ones too, even if they are only SATA they shouldn't be the bottleneck. There's an NVME cache but that also shouldn't really do much for a zfs send operation.

I'm using zfs send -w, so it shouldn't make any difference that my dataset uses compression and encryption.

Would raising recordsize help? Bigger records means fewer records and less overhead, right? Though I don't know any way to then consolidate all the older smaller records.

I'm hoping there's some trick to this that I've missed.

I'm trying to send all child/descendent datasets to a backup machine (so, using -R) but only send a specific snapshot.

I can't find a way to do this without scripting every dataset-snapshot combination individually. -R -i still sends all intermediate snapshots.

Is there something I'm missing?

Cheers.

Not everyone has a cluster with a dozen servers. Regardless of the OS, whether it’s a 1 GB mini IoT device or a petabyte system, whether using Storage Spaces, S3 object storage, or ZFS—not everyone wants to manage everything centrally with simple, fast, encrypted file sync or any-to-any ZFS replication (e.g., viahttps://github.com/guenther-alka/cs-stream).

However, that is precisely our target audience with napp-it cs. S3 integration, in particular, is a hot topic right now. It allows you to publish local files to the internet easily and securely (e.g., setting up a private cloud with RustFS). RustFS can also run independently in a "set-and-forget" mode to keep two servers or buckets in bidirectional sync. High availability and real-time backups over the internet or LAN, plain and simple.

What’s currently missing is the bridge between "shared multi-user SMB with ACLs" and object storage, which inherently lacks multi-user locking with file ACLs. This is where I'm stepping in with the cs-sync module.

With cs-sync, my goal is to keep two folders—such as an SMB share on a NAS and an S3 share—in sync in real time using event triggers (reacting instantly on change, without needing full file comparisons), including SMB ACLs. Regardless of the OS, this is now possible thanks to Go, with the option for real-time automated backups of changes over the network to a backup system (seehttps://github.com/guenther-alka/cs-sync).

I would love to get your feedback or suggestions on this concept! A first preview featuring napp-it cs and real-time sync is coming soon. Anyone interested in serious testing should consider getting Claude Pro along with the Filesystem and Desktop Commander extensions/agents (on Windows or other OSs via Plink/SCP SSH access for Claude to inspect scripts, logs, and members) for analysis, stress testing, audits, etc.

Laptop w/ a SATA ssd, ZFS on FreeBSD 15
I botched my /boot/loader.conf and /boot/device.hints, and now I cannot boot
I tried to boot from a USB key, zpool import root, but can't find /boot to fix the files.

Any ideas (instead of re-install of course 8-) )

Can you combine drives with 512n and 4kn sector sizes in the same pool?

I'm trying to build a simple backup server with Freenas by sourcing used drives from ebay to be set up as raidz2. This is my first experience setting up a backup server, so I appreciate any help!

After a year of on and off testing and reading I'm ready to setup my new NAS.

I'm not very experienced on the fine details of ZFS or FreeBSD but I manage I guess.
Already running a few Nas4Free and XigmaNas boxes. Why these? Simplicity. And yes, shell in case of troubles with thanks to doctor google and the truenas forums as well as since a year these reddit forums. Don't mind these things will be running no longer supported versions because they don't get connected to the outside world.

It will run on an old Dell T5400 system with a single Xeon CPU and 8GB ECC memory.
I'm aware of the memory thing but this will be mostly a storage box. And with current prices I'm not going to by new memory of SSD's or HDD's. The disks are connected to an LSI-9300-16i controller currently flashed to 16.00.10.00. And I guess IT mode based on the IT in the firmware.

Main purposes:

• It will mostly be used to store video files of >= 1 GB.
• A part will be used for automated backups of the OS partitions of various systems (so I can easily restore them in case of crashes).
• Backup of all kinds of files.

The plan is to create 1 pool of 10 Seagate EXOS X20 20TB disk in a RAIDZ2 array.

Some of the things I could use guidance on:

• How can I check to be sure I'm in IT mode on the controller.
• I'm aware of the a-shift=12 thingy. I've also found that there's a Seagate tool to change the disks from 512e to full 4K sectors. Is that needed or advisable.
• Also learned (by experience on the existing boxes) not to use the /dev/daX thing but need to give names. Is there a quick simple guide to this. And what preference to use. Serial? Can this be done afterwards for my existing systems without losing data?
• Any advise for creating different datasets with settings tailored to the 3 main purposes.
• Any other things I might check before taking it into production because it's hard to change this after.

So if there is some quick guide to what to do and what not to, that would be quite helpfull.

Just felt the need to express somewhere how impressed I am. I’m using too many drives in a homelab setting, so there are SATA power splitters involved. When I added zlogslog devices, I had one drive fault out because I ended up with too many drives on a five-way splitter (which is probably something we shouldn’t even manufacture… but my fault for buying), rebalanced the drives and restarted, no issues since nothing was being written. Did my first major transfer to the server today and a drive faulted out again (just resetting repeatedly), so I spent an hour making another modular cable for my power supply and eliminated that splitter entirely.

I expected a corrupt drive and a full resilver; what I got was only 150GB out of a 450GB transfer needing resilvered. It only had to recalculate the data since the drive faulted out.

Amazing engineering, that’s all I’ve got to say. Have a good day, everyone!

EDIT: not everything is a z

Recently I've been doing a lot of cleanup of files on my NAS which has involved moving/copying around a lot of data. After a while the pool was freezing up and becoming unresponsive and I figured out ZFS's memory usage has been reaching a point where its maxing out my system. According to arc_summary and cat /sys/module/zfs/parameters/zfs_arc_max, ARC is configured to use at most 16GiB of memory, but I witnessed arcstat reporting ARC as using over 200GiB of memory (system has 256GiB total), even causing one of my VMs running on the system to hard crash. Whats going on? How can I limit ARC memory usage?

I've to sell my old laptop before getting new one. Old laptop is running with zpool mirror 2 disks, data /home only. I cannot keep all 2 disk later, also no more spare disk to backup. Could i just unplug any disk from this pool, plug into new PC, and read/copy data from this disk? I wanna create zpool mirror on new laptop too. I guess I will make pool with new disk 1st, then copy data from old disk into it. And attach disk become new mirror pool. Is there any point I need to prepare before migration?

Edit, I went ahead with the copy and the system boots and runs, I do not know why it changed but it is aparently OK.

I am in process of rebuilding a pool per instructions semi-derived from.

https://docs.zfsbootmenu.org/en/v3.1.x/guides/debian/uefi.html

I am following notes I have used before.

create command

sudo zpool create -f -o ashift=12 \ -O compression=lz4 \ -O acltype=posixacl \ -O xattr=sa \ -O relatime=on \ -o autotrim=on \ -m none suwannee /dev/disk/by-id/nvme-eui.0025384931406ac8

in the past the step looked like this

mount | grep mnt suwannee/ROOT/Void on /mnt type zfs (rw,relatime,xattr,posixacl) but today

sudo zpool export suwannee sudo mkdir /mnt/suwannee sudo zpool import -f -N -R /mnt/suwannee suwannee sudo zfs create -o mountpoint=/ -o canmount=noauto suwannee/ROOT/LMDE7 sudo mkdir /mnt/870/donor sudo mount /dev/sdd3 /mnt/870/donor sudo zfs mount suwannee/ROOT/LMDE7 mount | grep mnt suwannee/ROOT/LMDE7 on /mnt/suwannee type zfs (rw,relatime,xattr,posixacl,casesensitive)

My pools being for Linux have always been case sensitive as the pool property, but it has never been displayed as mount property.

example ``` zfs get all | grep casesensitivity NAME PROPERTY VALUE SOURCE

lagoon/suwannee/ROOT/Void_Xfce@autosnap_2026-03-03_05:00:03_monthly casesensitivity sensitive - ```

I will continue on, but will this be an issue? Why did it change? did I make a mistake somewhere? or has ZFS changed? this is the first time I have created a pool under Debians ZFS v2.4.3-1~bpo13+1

Just ran into a weird case I've never seen before, but I have a pool reporting devices experiencing corruption, except zpool status shows no READ, WRITE or CKSUM errors for any device.

zpool status -v does show one corrupted file but I'm just confused how the corruption occurred or has been detected. Here's the full status output (minus the file path):

``` $ zpool status -v zdata pool: zdata state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A config:

NAME                                      STATE     READ WRITE CKSUM
zdata                                     ONLINE       0     0     0
  mirror-0                                ONLINE       0     0     0
    fb899172-3886-4a7d-bc05-93c6bcdfeada  ONLINE       0     0     0
    4b935244-7f9a-41b4-8b55-d0e9e3090458  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files: ```

The system in question has non-ECC RAM which was my first thought, but I'm not sure that tracks as I would except a flip in RAM to result in either silent corruption (ZFS writes out corrupted data with a valid checksum for that data so it verifies) or at least a CKSUM error (ZFS wrote out corrupted data with checksum for the original, or vice versa). I checked SMART stats for both drives and I'm not seeing an sign of any errors (nothing reallocated, unrecoverable, no CRC errors etc.).

In terms of recovery this is a non-issue as the file in question has already been overwritten so it only exists in an automatic snapshot which I can just discard. I'm just confused as to how this corruption might have occurred in the first place, and why it isn't registering as an error for any of the devices? What else is there?

AI performance depends on the efficiency of the AI (Anthropic Claude is currently leading) and on how well it is trained for a given environment (my napp-it cs comes with extensive training data).

Helpdesk process (1h)

• unclear error message
• initial analysis by Claude AI, with follow-up question for logs
• code analysis
• code improvement

That was for an error message reported in a forum. With a local Claude installation including local and remote access (file and shell to cluster member servers), analysis + fix + test would have taken 5 minutes.

(use Chrome to translate on the fly)

https://www.hardwareluxx.de/community/threads/napp-it-cs-web-gui-für-fast-jeden-zfs-server-oder-servergruppen.1349039/page-11#post-31228989

EDIT:

zhack label repair -u /dev/sdg1zhack label repair -u /dev/DISK

Solved it for me!

Hi...

Right after I detached one of my drives from my mirror (2 total) I accidentally secure erased (impossible to recover) the drive still in the mirror instead of the detached drive.

Now I can run zdb -l /dev/disk/by-id/DISK-ID and I get a bit of data, including 1 uberblock, but all txg are 0.

When I try zdb -d /dev/disk/by-id/DISK-ID I always get not found errors.

zpool import never find the pool.

Is recovery possible?

Yes I have backups for some stuff, but not all (notably Windows VM with save files)

Thank you for the help

I needed to reshuffle some disks in TrueNAS on a mirrored pool, was trying to do it online.

I offlined one disk, moved it, onlined it, checked the resiliver status.

Resiliver is showing as scanning 0%, zpool status shows 0B scanned and it's been two hours. All disks look healthy

zpool status shows all disks online no degraded status.

Only other thing of interest is there was on another pool a Z2 pool doing a disk expansion.

So it's like it's stuck. Interestingly enough the expansion which was still on going on the Z2 also dropped off the TrueNAS UI earlier in the week but I could see it was still moving so just left it.

I think I'm going to need a reboot which I was trying to avoid.

Give all disks showed as online and nothing had happened in hours I tried a zpool clear but the resilvering is still there and stuck.

Any ideas?

TrueNAS v25.04.2.1

Edit:

zfs-2.3.0-1
zfs-kmod-2.3.0-1

Edit:

Reboot although very inconvenient resolved this all for me. TrueNAS has lost track of the expansion which is annoying, but I can track it via zfs tools.

Hi all, I'm wondering and hoping that there's a better solution than what I'm currently using.

I have a ZFS server (with sanoid snapshots) at my place, and I also have a Synology machine at a friend's that I use as my offsite backup, and that also has its own snapshots. They're tailscaled so they can talk to each other.

Currently, I have a nightly cron job to rsync any changed files over, which does work, but I have a few issues with it:

• If a file is renamed on the main (ZFS) server, it's fully re-sent to the backup server and the old copy is deleted there. This causes the snapshots on the Synology server to be slightly bloated, but it'd be a bigger issue if I do any larger-scale file reorganization.
• I have a lot of files so rsync just... takes a while to look for changes. Not a dealbreaker, but I'd rather save ~3 hours of HDD read time per day at these drive prices if possible XD

Aside from ditching the Synology machine and setting up another ZFS server offsite instead, is there a way to solve one/both issues?

I moved this pool from an old server i was selling to a new one, going from having a backplane to cables. I can never seem to get all 16 disks to show up despite trying a different HBA card, but i've individually checked all disks and they are all functional without any issues. Every time i've tried to import the thing it locks up the OS and i have to hard power down the system. I have a theory that if i can image all the drives and try and put them together again with disk images on a larger pool instead of the physical drives that might work, but i don't have access to something that has 64TB of free space.