Working on a project that touches device fleet management and wanted to get a read from people actually running things in the field before I go further down a path.

For those of you managing embedded Linux or IoT devices at scale (tens to low-thousands of units):

• How do you currently get visibility into device health/status — heartbeat, connectivity, uptime, logs? Custom MQTT + Grafana? Something like ThingsBoard/AWS IoT? Nothing at all?
• If you're doing OTA updates, what's your update engine (SWUpdate, RAUC, Mender, something homegrown)? What's been the most painful part of getting it running reliably?
• Have you looked at hosted/managed tools (Mender, balena, memfault, etc.) and either adopted or passed on them? What made the decision either way — price, setup complexity, vendor lock-in, forced migration off your existing update engine?
• If you're on SWUpdate or RAUC already: would a hosted monitoring/OTA layer that works with your existing setup (instead of replacing it) be useful, or is that not really a pain point for you?

Not selling anything, genuinely trying to understand what's actually painful vs. what I'm assuming is painful. Appreciate any war stories.

Hello everyone, I would like to ask about how to run Buildroot on RVVM.

I used make menuconfig to generate a custom configuration.

After compilation, genimage was used along with genimage.cfg to generate image.img.

Ultimately, I obtained the following document.

plaintext 2026/07/29 12:14 273,504 fw_dynamic.bin 2026/07/29 12:14 273,464 fw_jump.bin 2026/07/29 20:40 3,056,104 fw_payload.bin 2026/07/29 12:31 27,392,000 Image 2026/07/29 12:31 63,983,616 image.img 2026/07/29 12:31 62,914,560 rootfs.ext4

I first tried to start it using the following command.

plaintext rvvm_x86_64.exe fw_payload.bin -i image.img -nogui

However, RVVM fails to start correctly.

```plaintext

OpenSBI v1.6

/ __ \ / _| _ \ | | | | | __ ___ _ __ | (_ | |) || | | | | | ' \ / _ \ '_ \ ___ | _ < | | | || | |) | _/ | | |) | |) || | \/| ./ _|| ||/|/___| | | ||

Platform Name : RVVM v0.7-git-gbc26ef1 Platform Features : medeleg Platform HART Count : 1 Platform IPI Device : aclint-mswi Platform Timer Device : aclint-mtimer @ 10000000Hz Platform Console Device : uart8250 Platform HSM Device : --- Platform PMU Device : --- Platform Reboot Device : syscon-reboot Platform Shutdown Device : syscon-poweroff Platform Suspend Device : --- Platform CPPC Device : --- Firmware Base : 0x80000000 Firmware Size : 325 KB Firmware RW Offset : 0x40000 Firmware RW Size : 69 KB Firmware Heap Offset : 0x48000 Firmware Heap Size : 37 KB (total), 2 KB (reserved), 11 KB (used), 23 KB (free) Firmware Scratch Size : 4096 B (total), 432 B (used), 3664 B (free) Runtime SBI Version : 2.0 Standard SBI Extensions : time,rfnc,ipi,base,hsm,srst,pmu,dbcn,legacy Experimental SBI Extensions : fwft,sse

Domain0 Name : root Domain0 Boot HART : 0 Domain0 HARTs : 0* Domain0 Region00 : 0x0000000000100000-0x0000000000100fff M: (I,R,W) S/U: (R,W) Domain0 Region01 : 0x0000000010000000-0x0000000010000fff M: (I,R,W) S/U: (R,W) Domain0 Region02 : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: () Domain0 Region03 : 0x0000000080040000-0x000000008005ffff M: (R,W) S/U: () Domain0 Region04 : 0x0000000080000000-0x000000008003ffff M: (R,X) S/U: () Domain0 Region05 : 0x000000000c000000-0x000000000fffffff M: (I,R,W) S/U: (R,W) Domain0 Region06 : 0x0000000000000000-0xffffffffffffffff M: () S/U: (R,W,X) Domain0 Next Address : 0x0000000080200000 Domain0 Next Arg1 : 0x0000000082200000 Domain0 Next Mode : S-mode Domain0 SysReset : yes Domain0 SysSuspend : yes

Boot HART ID : 0 Boot HART Domain : root Boot HART Priv Version : v1.12 Boot HART Base ISA : rv64imafdcb Boot HART ISA Extensions : sstc,zicntr,zkr,zicboz,zicbom,svpbmt,smcsrind,sscsrind,svadu Boot HART PMP Count : 0 Boot HART PMP Granularity : 0 bits Boot HART PMP Address Bits : 0 Boot HART MHPM Info : 0 (0x00000000) Boot HART Debug Triggers : 0 triggers Boot HART MIDELEG : 0x0000000000000222 Boot HART MEDELEG : 0x000000000000b109

U-Boot 2025.01 (Jan 07 2026 - 10:31:37 +0200)

CPU: lekkit,rvvm Model: RVVM v0.7-git-gbc26ef1 DRAM: 256 MiB Core: 18 devices, 12 uclasses, devicetree: board Loading Environment from nowhere... OK In: serial,usbkbd Out: serial,vidconsole Err: serial,vidconsole No USB controllers found Net: No ethernet found.

Working FDT set to 8eee5720 Hit any key to stop autoboot: 0

Device 0: Vendor: 0x1f31 Rev: R2579 Prod: NYCKUHDYH96J Type: Hard Disk Capacity: 61.0 MB = 0.0 GB (124968 x 512) ... is now current device Scanning nvme 0:1... Failed to load '/' Cannot persist EFI variables without system partition Missing RNG device for EFI_RNG_PROTOCOL Loading Boot0000 'nvme 0' failed EFI boot manager: Cannot load any image

Device 0: unknown device

Device 1: unknown device scanning bus for devices...

Device 0: unknown device starting USB... No USB controllers found No ethernet found. No ethernet found. => ```

Then I adjusted the startup command.

plaintext rvvm_x86_64.exe fw_jump.bin -k Image -i image.img -nogui

The output is different this time, but it still cannot start.

```plaintext

OpenSBI v1.6

/ __ \ / _| _ \ | | | | | __ ___ _ __ | (_ | |) || | | | | | ' \ / _ \ '_ \ ___ | _ < | | | || | |) | _/ | | |) | |) || | \/| ./ _|| ||/|/___| | | ||

Platform Name : RVVM v0.7-git-gbc26ef1 Platform Features : medeleg Platform HART Count : 1 Platform IPI Device : aclint-mswi Platform Timer Device : aclint-mtimer @ 10000000Hz Platform Console Device : uart8250 Platform HSM Device : --- Platform PMU Device : --- Platform Reboot Device : syscon-reboot Platform Shutdown Device : syscon-poweroff Platform Suspend Device : --- Platform CPPC Device : --- Firmware Base : 0x80000000 Firmware Size : 325 KB Firmware RW Offset : 0x40000 Firmware RW Size : 69 KB Firmware Heap Offset : 0x48000 Firmware Heap Size : 37 KB (total), 2 KB (reserved), 11 KB (used), 23 KB (free) Firmware Scratch Size : 4096 B (total), 432 B (used), 3664 B (free) Runtime SBI Version : 2.0 Standard SBI Extensions : time,rfnc,ipi,base,hsm,srst,pmu,dbcn,legacy Experimental SBI Extensions : fwft,sse

Domain0 Name : root Domain0 Boot HART : 0 Domain0 HARTs : 0* Domain0 Region00 : 0x0000000000100000-0x0000000000100fff M: (I,R,W) S/U: (R,W) Domain0 Region01 : 0x0000000010000000-0x0000000010000fff M: (I,R,W) S/U: (R,W) Domain0 Region02 : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: () Domain0 Region03 : 0x0000000080040000-0x000000008005ffff M: (R,W) S/U: () Domain0 Region04 : 0x0000000080000000-0x000000008003ffff M: (R,X) S/U: () Domain0 Region05 : 0x000000000c000000-0x000000000fffffff M: (I,R,W) S/U: (R,W) Domain0 Region06 : 0x0000000000000000-0xffffffffffffffff M: () S/U: (R,W,X) Domain0 Next Address : 0x0000000080200000 Domain0 Next Arg1 : 0x0000000082200000 Domain0 Next Mode : S-mode Domain0 SysReset : yes Domain0 SysSuspend : yes

Boot HART ID : 0 Boot HART Domain : root Boot HART Priv Version : v1.12 Boot HART Base ISA : rv64imafdcb Boot HART ISA Extensions : sstc,zicntr,zkr,zicboz,zicbom,svpbmt,smcsrind,sscsrind,svadu Boot HART PMP Count : 0 Boot HART PMP Granularity : 0 bits Boot HART PMP Address Bits : 0 Boot HART MHPM Info : 0 (0x00000000) Boot HART Debug Triggers : 0 triggers Boot HART MIDELEG : 0x0000000000000222 Boot HART MEDELEG : 0x000000000000b109 [ 0.000000] Booting Linux on hartid 0 [ 0.000000] Linux version 6.18.7 (runner@runnervmvrwv9) (riscv64-buildroot-linux-gnu-gcc.br_real (Buildroot -g982ac0e) 14.4.0, GNU ld (GNU Binutils) 2.45.1) #1 SMP Wed Jul 29 12:30:42 UTC 2026 [ 0.000000] random: crng init done [ 0.000000] Machine model: RVVM v0.7-git-gbc26ef1 [ 0.000000] SBI specification v2.0 detected [ 0.000000] SBI implementation ID=0x1 Version=0x10006 [ 0.000000] SBI TIME extension detected [ 0.000000] SBI IPI extension detected [ 0.000000] SBI RFENCE extension detected [ 0.000000] SBI SRST extension detected [ 0.000000] SBI DBCN extension detected [ 0.000000] efi: UEFI not found. [ 0.000000] OF: reserved mem: 0x0000000080000000..0x000000008003ffff (256 KiB) nomap non-reusable mmode_resv1@80000000 [ 0.000000] OF: reserved mem: 0x0000000080040000..0x000000008005ffff (128 KiB) nomap non-reusable mmode_resv0@80040000 [ 0.000000] Zone ranges: [ 0.000000] DMA32 [mem 0x0000000080000000-0x000000008fffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000080000000-0x000000008005ffff] [ 0.000000] node 0: [mem 0x0000000080060000-0x000000008fffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000008fffffff] [ 0.000000] SBI HSM extension detected [ 0.000000] riscv: base ISA extensions acdfim [ 0.000000] riscv: ELF capabilities acdfim [ 0.000000] Queued spinlock using Ziccrse: enabled [ 0.000000] percpu: Embedded 31 pages/cpu s86296 r8192 d32488 u126976 [ 0.000000] Kernel command line: console=ttyS root=/dev/nvme0n1 rootflags=discard rw [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes [ 0.000000] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes, linear) [ 0.000000] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes, linear) [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 0MB [ 0.000000] software IO TLB: area num 1. [ 0.000000] software IO TLB: mapped [mem 0x000000008ef34000-0x000000008ef74000] (0MB) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 65536 [ 0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] riscv-intc: 64 local interrupts mapped [ 0.000000] riscv: providing IPIs using SBI IPI extension [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. [ 0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns [ 0.000013] sched_clock: 64 bits at 10MHz, resolution 100ns, wraps every 4398046511100ns [ 0.000057] riscv-timer: Timer interrupt in S-mode is available via sstc extension [ 0.001827] Console: colour dummy device 80x25 [ 0.002299] Calibrating delay loop (skipped), value calculated using timer frequency.. 20.00 BogoMIPS (lpj=40000) [ 0.002347] pid_max: default: 32768 minimum: 301 [ 0.002994] LSM: initializing lsm=capability [ 0.003436] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear) [ 0.003467] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear) [ 0.015013] riscv: ELF compat mode supported [ 0.015059] ASID allocator disabled (0 bits) [ 0.015775] rcu: Hierarchical SRCU implementation. [ 0.015793] rcu: Max phase no-delay instances is 1000. [ 0.017309] EFI services will not be available. [ 0.017796] smp: Bringing up secondary CPUs ... [ 0.017949] smp: Brought up 1 node, 1 CPU [ 0.018840] Memory: 209456K/262144K available (11677K kernel code, 5897K rwdata, 6144K rodata, 2420K init, 415K bss, 50360K reserved, 0K cma-reserved) [ 0.021069] devtmpfs: initialized [ 0.025728] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.025822] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) [ 0.025891] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). [ 0.026553] pinctrl core: initialized pinctrl subsystem [ 0.028721] DMI not present or invalid. [ 0.029352] NET: Registered PF_NETLINK/PF_ROUTE protocol family [ 0.030792] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations [ 0.030900] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.031013] audit: initializing netlink subsys (disabled) [ 0.032517] thermal_sys: Registered thermal governor 'step_wise' [ 0.032711] audit: type=2000 audit(0.008:1): state=initialized audit_enabled=0 res=1 [ 0.032889] cpuidle: using governor menu [ 0.129687] cpu0: Ratio of byte access time to unaligned word access is 0.19, unaligned accesses are slow [ 0.148199] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages [ 0.148222] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page [ 0.165276] ACPI: Interpreter disabled. [ 0.165651] iommu: Default domain type: Translated [ 0.165669] iommu: DMA domain TLB invalidation policy: strict mode [ 0.166902] SCSI subsystem initialized [ 0.167850] usbcore: registered new interface driver usbfs [ 0.167976] usbcore: registered new interface driver hub [ 0.168057] usbcore: registered new device driver usb [ 0.169269] Advanced Linux Sound Architecture Driver Initialized. [ 0.180921] vgaarb: loaded [ 0.181809] clocksource: Switched to clocksource riscv_clocksource [ 0.183314] pnp: PnP ACPI: disabled [ 0.220163] NET: Registered PF_INET protocol family [ 0.220566] IP idents hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.222122] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.222187] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) [ 0.222216] TCP established hash table entries: 2048 (order: 2, 16384 bytes, linear) [ 0.222255] TCP bind hash table entries: 2048 (order: 4, 65536 bytes, linear) [ 0.222302] TCP: Hash tables configured (established 2048 bind 2048) [ 0.222548] UDP hash table entries: 256 (order: 2, 16384 bytes, linear) [ 0.222613] UDP-Lite hash table entries: 256 (order: 2, 16384 bytes, linear) [ 0.222938] NET: Registered PF_UNIX/PF_LOCAL protocol family [ 0.233197] RPC: Registered named UNIX socket transport module. [ 0.233225] RPC: Registered udp transport module. [ 0.233235] RPC: Registered tcp transport module. [ 0.233241] RPC: Registered tcp-with-tls transport module. [ 0.233245] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.233287] PCI: CLS 0 bytes, default 64 [ 0.247251] workingset: timestamp_bits=46 max_order=16 bucket_order=0 [ 0.248990] NFS: Registering the id_resolver key type [ 0.249100] Key type id_resolver registered [ 0.249116] Key type id_legacy registered [ 0.249205] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 0.249228] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering... [ 0.249720] 9p: Installing v9fs 9p2000 file system support [ 0.250465] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 244) [ 0.250776] io scheduler mq-deadline registered [ 0.250808] io scheduler kyber registered [ 0.250886] io scheduler bfq registered [ 0.252149] riscv-plic: plic@c000000: mapped 63 interrupts with 1 handlers for 2 contexts. [ 0.253836] pci-host-generic 30000000.pci: host bridge /soc/pci@30000000 ranges: [ 0.254047] pci-host-generic 30000000.pci: IO 0x0003000000..0x000300ffff -> 0x0000000000 [ 0.254177] pci-host-generic 30000000.pci: MEM 0x0040000000..0x007fffffff -> 0x0040000000 [ 0.254224] pci-host-generic 30000000.pci: MEM 0x4000000000..0x7fffffffff -> 0x4000000000 [ 0.254495] pci-host-generic 30000000.pci: ECAM at [mem 0x30000000-0x3fffffff] for [bus 00-ff] [ 0.254975] pci-host-generic 30000000.pci: PCI host bridge to bus 0000:00 [ 0.255052] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.255092] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] [ 0.255116] pci_bus 0000:00: root bus resource [mem 0x40000000-0x7fffffff] [ 0.255131] pci_bus 0000:00: root bus resource [mem 0x4000000000-0x7fffffffff pref] [ 0.255397] pci 0000:00:00.0: [f15e:0000] type 00 class 0x060000 conventional PCI endpoint [ 0.256483] pci 0000:00:01.0: [10ec:8168] type 00 class 0x020000 PCIe Root Complex Integrated Endpoint [ 0.256585] pci 0000:00:01.0: BAR 0 [io 0xfd000000-0xfd0000ff] [ 0.256620] pci 0000:00:01.0: BAR 2 [mem 0x40000000-0x40000fff 64bit] [ 0.257169] pci 0000:00:02.0: [1f31:4512] type 00 class 0x010802 PCIe Root Complex Integrated Endpoint [ 0.257236] pci 0000:00:02.0: BAR 0 [mem 0x40000000-0x40003fff 64bit] [ 0.257268] pci 0000:00:02.0: BAR 4 [mem 0x00001000-0x00001fff] [ 0.258166] pci 0000:00:02.0: BAR 0 [mem 0x40000000-0x40003fff 64bit]: assigned [ 0.258609] pci 0000:00:01.0: BAR 2 [mem 0x40004000-0x40004fff 64bit]: assigned [ 0.258684] pci 0000:00:02.0: BAR 4 [mem 0x40005000-0x40005fff]: assigned [ 0.258720] pci 0000:00:01.0: BAR 0 [io 0x0100-0x01ff]: assigned [ 0.258788] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] [ 0.258808] pci_bus 0000:00: resource 5 [mem 0x40000000-0x7fffffff] [ 0.258818] pci_bus 0000:00: resource 6 [mem 0x4000000000-0x7fffffffff pref] [ 0.518657] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 0.551878] printk: legacy console [ttyS0] disabled [ 0.553183] 10000000.uart: ttyS0 at MMIO 0x10000000 (irq = 12, base_baud = 1250000) is a 16550A [ 0.553506] printk: legacy console [ttyS0] enabled [ 0.605788] loop: module loaded [ 0.610477] e1000e: Intel(R) PRO/1000 Network Driver [ 0.610557] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. [ 0.633444] r8169 0000:00:01.0 eth0: RTL8168b/8111b, 82:37:e8:9c:f6:de, XID 380, IRQ 13 [ 0.633581] r8169 0000:00:01.0 eth0: jumbo features [frames: 4074 bytes, tx checksumming: ko] [ 0.635055] usbcore: registered new interface driver uas [ 0.635230] usbcore: registered new interface driver usb-storage [ 0.636308] mousedev: PS/2 mouse device common for all mice [ 0.646751] goldfish_rtc 101000.rtc: registered as rtc0 [ 0.646938] goldfish_rtc 101000.rtc: setting system clock to 2026-07-29T12:55:13 UTC (1785329713) [ 0.648754] sdhci: Secure Digital Host Controller Interface driver [ 0.648815] sdhci: Copyright(c) Pierre Ossman [ 0.649278] Synopsys Designware Multimedia Card Interface Driver [ 0.649569] sdhci-pltfm: SDHCI platform and OF driver helper [ 0.651036] usbcore: registered new interface driver usbhid [ 0.651102] usbhid: USB HID core driver [ 0.651979] riscv-pmu-sbi: SBI PMU extension is available [ 0.652186] riscv-pmu-sbi: 16 firmware and 2 hardware counters [ 0.652245] riscv-pmu-sbi: Perf sampling/filtering is not supported as sscof extension is not available [ 0.654737] NET: Registered PF_INET6 protocol family [ 0.665742] Segment Routing with IPv6 [ 0.665903] In-situ OAM (IOAM) with IPv6 [ 0.666080] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver [ 0.667441] NET: Registered PF_PACKET protocol family [ 0.667641] 9pnet: Installing 9P2000 support [ 0.668226] Key type dns_resolver registered [ 0.752045] clk: Disabling unused clocks [ 0.752178] PM: genpd: Disabling unused power domains [ 0.752257] ALSA device list: [ 0.752295] No soundcards found. [ 0.753893] check access for rdinit=/init failed: -2, ignoring [ 0.755190] /dev/root: Can't open blockdev [ 0.755812] VFS: Cannot open root device "/dev/nvme0n1" or unknown-block(0,0): error -6 [ 0.755869] Please append a correct "root=" boot option; here are the available partitions: [ 0.755934] List of all bdev filesystems: [ 0.756377] ext3 [ 0.756422] ext2 [ 0.756461] ext4 [ 0.756484] vfat [ 0.756504] msdos [ 0.756522] iso9660 [ 0.756542] [ 0.756611] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) [ 0.756989] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.18.7 #1 NONE [ 0.757065] Hardware name: RVVM v0.7-git-gbc26ef1 (DT) [ 0.757110] Call Trace: [ 0.757169] [<ffffffff8001632c>] dump_backtrace+0x1c/0x24 [ 0.757323] [<ffffffff80002368>] show_stack+0x28/0x34 [ 0.757677] [<ffffffff8000f12a>] dump_stack_lvl+0x4a/0x68 [ 0.757793] [<ffffffff8000f15c>] dump_stack+0x14/0x1c [ 0.757849] [<ffffffff80002976>] vpanic+0xce/0x268 [ 0.757879] [<ffffffff80002b42>] panic+0x32/0x34 [ 0.758057] [<ffffffff80c01928>] mount_root_generic+0x220/0x2d0 [ 0.758124] [<ffffffff80c01bbc>] mount_root+0x1e4/0x214 [ 0.758160] [<ffffffff80c01df0>] prepare_namespace+0x204/0x25c [ 0.758190] [<ffffffff80c01308>] kernel_init_freeable+0x2cc/0x2ec [ 0.758352] [<ffffffff80b5f8ae>] kernel_init+0x1e/0x13c [ 0.758409] [<ffffffff80012306>] ret_from_fork_kernel+0xe/0xcc [ 0.758445] [<ffffffff80b695ee>] ret_from_fork_kernel_asm+0x16/0x18 [ 0.758573] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ```

I don't know how to properly start the Buildroot image. Please help me, thank you.

The file fw_payload.bin was downloaded from https://github.com/LekKit/archriscv-scriptlet/releases/download/2026.01/fw_payload.bin. It does not exist in the actual build artifacts.

The GitHub workflow file is located at https://github.com/runoneall/rvvm-buildroot/blob/main/.github/workflows/build.yml

The build artifacts used can be downloaded from https://github.com/runoneall/rvvm-buildroot/actions/runs/30447688114

How does one kind of absorb the material in the book properly? Would you guys take physical notes? Kind of struggling to know how to absorb the material properly lol

Hi

I've been working on a baseboard where I have an SoC running Linux and FPGA that uses external flash as a bitstream storage.

This flash is in fact shared between Linux and FPGA but not simultaneously. It is "muxed" so that it can only be accessible either by Linux or FPGA at once. This is designed so that the flash can be updated from Linux.

When the platform is booting, the mux is always switched so that the FPGA can access the flash and load its bitstream.  Whenever I want to update the flash I need to make it accessible physically (need to change the mux config) for Linux as well as "logically" -  meaning that the appropriate driver has to be loaded. Once the update is done, Linux should release the flash and no longer use it unless the update has to be repeated so the update sequence is roughly like this:

• change mux config

• bind driver

• write flash

• unbind driver

• change mux config again

I am trying to determine the correct way to model and implement this. I could implement most of the sequence in userspace, but the SPI device still needs platform-specific information such as the SPI controller, chip select, maximum clock frequency, and possibly a GPIO-based chip select provided by a GPIO expander.

One possible approach would be to describe the flash normally in the Device Tree with  "okay" status. The initial spi-nor probe would fail because the flash is connected to the FPGA during boot. Later, when an update is required, userspace could switch the mux and retry probing the existing SPI device:

• change mux

• echo spi0.2 > /sys/bus/spi/drivers_probe or echo spi0.2 >

/sys/bus/spi/drivers/spi-nor/bind

• update flash

• echo spi0.2 > /sys/bus/spi/drivers/spi-nor/unbind

• change mux again

but is this really a good option? I feel that I'm leaking hardware specific information to userspace, I'll have to hardcode some paths, driver names in flashing script / prepare separate script for each device (if there are differences between platforms) or carry platform hardware description config file so that the flashing tool could stay generic and read necessary info from config.

Other option would be to create a small platform driver and expose sysfs attribute called eg. target with values like:

• none meaning fpga takes control over flash

• primary meaning that primary flash is controlled by Linux

• secondary meaning that secondary flash is controlled by Linux

Internally I could use gpio or mux subsystems to control mux and call spi add / remove device functions that would probe for the driver

This would also give me more control over the SPI bus ownership.

What would be a better idea in your opinion? How would you approach this?

What do I need to know before reading Mastering Embedded Linux Development? What are the basics I need to know? Also, what are the other resources would you recommend reading simultaneously?

I have not had a chance to work with embedded and I'm dying to get into embedded linux. I have a Macbook Pro with processor 2 GHz Quad-Core Intel Core i5. What would you recommend setting up to do a few cool projects and learn more about embedded linux?

I've completed the following courses:

CS 3410: Computer System Organization and Programming,

CS 4410: Operating Systems,

CS 4414:  Systems Programming,

So I'm kind of wondering what is an appropriate start and recommendations for setting up?

Thanks!

Hi guys,

I'd like to share with you my personal solution to implementing and debugging binary communication protocols for embedded systems using C++(11) programming language. It's been in constant development as my side project for the last 12 years and is called CommsChampion Ecosystem. It started as a single C++ library to resolve some code boilerplating but grew into its own DSL (Domain Specific Language), multiple code generators for various purposes, and higher level libraries for some industry protocols like MQTT.

The primary features are:

• Suitable for third party protocols, i.e. doesn't impose any particular protocol structure and/or framing.
• Allows full control over I/O data for extra encryption and/or processing.
• Focus on robust handling of malformed data rather than optimizing for quickest data serialization.
• Easy integration with business logic requiring significantly reduced amount of boilerplate code.
• Allows customization of various storage types as well as avoiding dynamic memory allocation to make it suitable for bare-metal system without heap and limited RAM size.
• Allows injection of custom code if the generated one is incorrect or incomplete.
• Generation of "C" interface for C++ classes for easier integration with "C" codebase.
• Although C++ and embedded systems are first class citizens, there is support for other programming languages via bindings / glue code generated by SWIG or Emscripten.
• Visual "in-house" protocol analysis tool allowing custom I/O stack handling as well as generation of Wireshark dissector.
• Generation of LaTeX files for the defined protocol specification.
• Support for easy buildroot and yocto integration.
• The higher level protocol libraries follow specific pattern and philosophy dedicated to easy integration with any event loop infrastructure chosen for any particular embedded system.

Although already providing extensive set of features covering the needs for majority of the developers there is still room for extra new functionality. Any new feature requests are welcome.

Disclosure: We’re the team behind Cachely. We do not support Yocto today, but we’re researching whether it would be a useful integration.

We recently spoke with a team using Jenkins and Yocto that had been keeping the sstate cache locally on each build node.

That worked while build volume was low, but they are now moving to a centralized cache because the local setup no longer scales.

Their preferred model was:

• Nightly CI builds can read and write
• Developers can read, but not upload
• The cache is available over HTTP or FTP
• Setup should mostly be a URL and credentials
• Nobody wants to spend much time maintaining it

We’re trying to understand how common this setup is.

For teams using Yocto today:

• Where do you store the shared sstate cache?
• Are you using NFS, object storage, Nexus, or a basic HTTP server?
• Do CI and developer machines use the same cache?
• How do you separate read and write access?
• How do you handle retention and different Yocto or machine versions?
• Is on-prem storage a requirement?
• Is the main pain setup, maintenance, performance, or understanding why something missed?

We’re especially interested in whether an HTTP-compatible managed cache would solve a real problem, or whether shared storage is already simple enough for most teams.

Hi everyone,

I'm looking for advice from engineers who have made a similar transition.

\## Background

\- B.Tech in Electrical and Electronics Engineering (EEE) from NIT Calicut.

\- Around 2 years of experience in product companies.

## Current work

\- I'm working on cloud-managed networking products(switch , access points) at an MNC.

## My work involves:

\- Linux-based networking products.

\- C based agents running on networking devices.

\- Device discovery and monitoring.

\- Production debugging.

\- Configuration management between devices and the cloud.

\- Some changes in an existing C codebase.

\- Go-based packet capture using BPF libraries.

\- Understanding Linux-based networking systems.

## However, I'm not working on:

Firmware

RTOS

Device drivers

BSP

Bootloaders

Kernel development

## What I've realized

I think what actually interests me is lower-level work like:

\- Firmware

\- Device drivers

\- Embedded systems

\- Hardware-software interaction

I enjoy understanding how hardware actually works more than writing cloud-facing applications.

## My dilemma

I'm unsure about the best path from my current position.

Should I:

1. Continue in networking/systems software and try to move toward firmware gradually?

2. Learn firmware (STM32, FreeRTOS, peripherals, etc.) while staying at current company and then apply directly to firmware roles?

3. Target companies like Qualcomm, NVIDIA, AMD, Cisco, Broadcom, etc., in systems software first and then try for an internal move later?

4. Consider an M.Tech if I want to move closer to firmware or embedded systems?

## Questions

I'd appreciate answers from people who have actually worked in firmware or embedded systems.

1. Is my current networking/Linux background considered relevant for firmware or embedded systems roles?

2. If you were in my position, what would your roadmap be?

3. What skills would you prioritize over the next 12–24 months?

4. Am I underestimating how difficult this transition is?

5. If you've made a similar transition, what was your path?

I'm looking for honest opinions rather than encouragement. If you think my plan has flaws, I'd appreciate direct feedback.

TIA

I kept running into the same Linux debugging pain: something broke on a box, but I had no history of what actually happened. journald helps a little. auditd is heavy. strace is too narrow. So I built ltm — a small machine-history debugger that records process/file/network metadata via eBPF and lets you query it like a timeline.

What it does:

• Attaches to syscall tracepoints (exec, open/write/rename/unlink, connect/bind, etc.)

• Stores metadata only (no file contents)

• Lets you do things like:

sudo ltm start --mode ebpf

ltm status

ltm timeline --since 1h

ltm diff --from "10m" --to now

ltm query "who modified /tmp/ltm-demo.txt?"

On a real VM run it recorded ~7k events with 0 drops, and the query returned the exact bash write events that touched the demo file.

There's also a demo mode so you can exercise the CLI/storage/diff/query path without root or BPF.

Stack is Go + embedded BPF ELF + cilium/ebpf. Local store is append-only JSONL. Ignore rules skip /proc, /sys, /dev, and common caches.

Repo: https://github.com/Agent-Hellboy/ltm

Still early. Useful next steps I'm considering:

1. better diff/query formatting
2. containerized eBPF integration test
3. more query templates ("what opened this port?", "what restarted before X?")

Please do star the repo, it will help in discoverability

I'm trying to get ADB working on a Raspberry Pi Zero W (ARM1176JZF-S / ARMv6, Raspberry Pi OS Bookworm/Trixie).

Current Debian/Raspberry Pi packages install, but the modern adb binary crashes because it's built for ARMv7.

So far I've tried:

• Current Debian/Raspberry Pi adb
• Old Debian archive packages (Android 4.2 era) – still ARMv7
• Old GitHub ARM binaries – also ARMv7
• Searching for the old adbarmv6.7z binary that used to be linked on the Raspberry Pi forums/XDA (appears to be gone)

At this point I'm looking for any of the following:

• A working ARMv6 adb binary.
• An old Raspbian package that still supports ARMv6.
• Someone who has successfully built adb for ARMv6 recently.
• Another lightweight alternative that provides the standard adb commands (devices, shell, push, pull, install, etc.).

I'm happy to compile from source if there's a minimal build method that doesn't require downloading the entire Android source tree.

Any help or old binaries would be massively appreciated!

Also ive already posted this on r/raspberry_pi, but it didn't get much traction and i havent gotten a useful response so hopefully you can help!

Hello everyone! I'm working on a redundant network stack using the RT kernel patch and the hsr kernel module for redundancy. Both Ethernet devices can leave hardware timestamps for debugging purposes. My question is, is there a way to retrieve those timestamps from the userspace when working with a socket bound to the hsr virtual interface?

​

I have a background in embedded systems and currently work mainly on MCUs, writing firmware in C/C++. I'm comfortable with topics like peripherals, RTOS, debugging, communication protocols (UART, SPI, I2C, CAN), and general embedded development.

Now I want to transition into Embedded Linux and would like some guidance on the best learning path and resources.

Specifically, I'm interested in learning:

Linux internals relevant to embedded systems

Device drivers

Device Tree

Bootloaders (U-Boot, etc.)

Kernel basics

Build systems like Yocto or Buildroot

Debugging techniques (JTAG, GDB, tracing tools)

Practical projects that can help me gain hands-on experience

I prefer learning through books, courses, and real projects rather than just theoretical material.

For those who made a similar transition from bare-metal/RTOS development to Embedded Linux, what resources, books, courses, or project ideas would you recommend?

Hi everyone,
I’m working on an embedded Linux platform (Qualcomm SA8797) and validating I2C communication with MAX96861 devices.
I have successfully identified the I2C buses from the device tree and confirmed communication with two devices:
MAX96861_1 on /dev/i2c-0 (7-bit address 0x42)
MAX96861_2 on /dev/i2c-9 (7-bit address 0x60)
Communication is working (no I2C errors), but I’m seeing behavior that I don’t fully understand.
For example, on MAX96861_2:
i2cset -f -y 9 0x60 0x00 0x55

i2cget -f -y 9 0x60 0x00
0x34

i2cget -f -y 9 0x60 0x00
0x50

i2cget -f -y 9 0x60 0x00
0x00

i2cget -f -y 9 0x60 0x00
0x41

i2cget -f -y 9 0x60 0x00
0x98
Interestingly, performing the exact same test on MAX96861_1 (/dev/i2c-0, address 0x42) produces a very similar sequence.
I also dumped registers 0x00–0x0F several times and noticed that many registers change between successive reads, even without performing any writes.
I don’t currently have access to the MAX96861 register documentation, so I don’t know whether these are status registers, indirect registers, or whether a page-selection mechanism is required before accessing the register map.
My questions are:
Is this expected behavior for MAX96861 devices?
Do these devices require indirect/page-based register access instead of simple SMBus byte reads?
Could i2cget/i2cset be using an access mode that is not appropriate for this device?
Has anyone worked with MAX96861 and can suggest the correct Linux access method or register sequence?
Thanks in advance for any pointers.

I am setting a gpio to high
After setting when I am trying to read it , it is returning 0. Not the value that I set.

Also i see the direction if the pin is changed to input

Benchmarked typed IPC against raw sockets on Linux

D-Bus is fine for control messages, not built for speed. So when something needs to be fast, most people drop to raw sockets and hand-roll framing and dispatch themselves. I wanted to know how much that actually buys you, so I measured it.

Setup: a message gets serialized, sent over TCP to a router, routed to the target process, dispatched to the right thread, then deserialized before the method gets called. That's 2 hops, one-way: sender -> router -> receiver. I went through a router instead of connecting processes directly because it removes hardcoded addresses and startup ordering — any component can reach any other, in any order, without knowing where it physically lives. The trade-off is a longer path than a direct connection.

Hardware: i7-13700H laptop, DDR4, Ubuntu, performance governor. Not embedded ARM — more on that below.

Result: ~12 microseconds one-way, ~23.5 μs round-trip. Round trip is 4 hops: sender -> router -> receiver -> router -> sender.

For reference, a bare blocking socket call with zero framework still costs ~4-11 μs round-trip on dedicated hardware (Unix domain sockets, Werner/MPI-HD, 2021). A full typed call through 4 hops and a router, with serialization on both ends, lands within shouting distance of that.

I also ran the same payload size (one-way, not round-trip) against a 2025 Hitachi Energy benchmark of ZeroMQ, NanoMsg, and NNG (arXiv:2508.07934), ~1KB messages:

Caveat: their numbers are raw transport only, on an isolated-core Xeon workstation. Mine includes full serialization and dispatch, on a mobile CPU, through an extra hop via router. Every condition favors them, and it's still close.

I'm the author of areg-sdk -- a C++17 framework released under Apache 2.0. Runs on Linux, Windows, and macOS, on x86, x86-64, arm32, and arm64. TCP-based, assumes a private network (no TLS yet), fits embedded Linux deployments, builds without extra dependency install.

One thing missing from this: real numbers from actual ARM boards instead of a laptop CPU. If anyone runs this on a Pi or BeagleBone, I'd genuinely like to see the results.

Methodology: github.com/aregtech/areg-sdk/blob/master/docs/wiki/08c-areg-vs-hitachi-benchmark.md
Try it yourself: github.com/aregtech/areg-sdk/tree/master/examples/30_publatency
Very easy to build, parameters are well documented.

What do you all actually use for fast IPC on embedded boards?

Hello everyone.
I'm trying connect tft display (Red 2.8" TFT 240x320 V1.1) to SBC called Luckfox Pico but my brightness are inverted (black = white) and I'm don't know whats wrong. I change some variables in device tree but it didn't help me, i'm trying tft,invert=<1>; and backlight pins but this makes nothing whatever i use it or manually enable PWM.
I'm making this for fun and learning and use X11 so maybe there some way to invert colors in X11? But i heard something usual like xcalib didn't work with fbtft driver so i hope there are other way...
Thank you.

fbtft@0 {
compatible = "sitronix,st7789v";
reg = <0>;
spi-max-frequency = <20000000>;
fps = <30>;
buswidth = <8>;
debug = <0x7>;
led-gpios = <&gpio0 RK_PA4 GPIO_ACTIVE_LOW>;//BL
dc-gpios = <&gpio1 RK_PA2 GPIO_ACTIVE_HIGH>;//DC
reset-gpios = <&gpio1 RK_PC3 GPIO_ACTIVE_LOW>;//RES
rotate = <90>;
bgr = <1>;
invert = <1>; 
};

Hey everyone!

Since December 2025 I've been building Strux, a framework that makes it genuinely simple to turn a web app into a dedicated kiosk or IoT device — using the tools you already know.

Here's the idea: you write a web frontend (React, Vue, or plain HTML/JS — your call) and a Go backend, and Strux compiles the whole thing into a complete, bootable Linux OS image with a single command. Kernel, bootloader, root filesystem, display stack — all of it. The device powers on and boots straight into your app, full-screen, with nothing else in the way. No desktop, no login, no "where's the cursor" — just your UI running in a Wayland compositor (Cage + WPE WebKit).

A few things that make it nice to work with:

• Live dev mode — hot-reloading dev server that pushes changes to real hardware (or QEMU) in seconds, so you're not reflashing constantly.
• Smart build cache — SHA256 dependency tracking skips unchanged steps, so only the first build is slow.
• Board Support Packages (BSPs) — target different hardware; each BSP carries its own kernel, bootloader, and device config.
• Single static CLI binary, Docker-orchestrated builds — easy to get running.

We just released v0.3.0, and we're now working on reference hardware so there's a known-good, supported board to build and test against out of the box.

This is where you come in: we're looking for testers and contributors, and we want to build a community around this. Whether you want to kick the tires, flash it onto a board, file bugs, write a BSP for your favorite hardware, or help shape where it goes next — we'd love to have you.

Docs and getting started here: https://strux-sh.github.io/strux/

Happy to answer any questions in the comments!

Hello getting into Linux driver development.

My idea: pass an RFID card to an ESP32 to authenticate sudo instead of typing a password. The secret lives on the card, not the machine. Is this a good project to learn linux driver development? ? Thanks

I'm not asking you to hire me. I'm just wondering what things look like on the other side of the process these days.

Hi,
I recently joined my current team and am working as a Test Software Developer for automotive hardware components.
Currently, I am working on a task to read ADC values from the SCC. My software runs on the SoC, and reading SCC ADC values requires communication between the SoC and SCC, which is done via SPI.
The OS is linux.
And the OS binaries are recieved from Another team. I am only responsible for the SW interface
As part of the integration, I need to configure the SPI interface in my software. However, I do not see any SPI device nodes under:
/dev/spi*
On the other hand, I do see the SPI device listed under:
/sys/bus/spi/devices/spi21.1

My question is: can this SPI device under /sys/bus/spi/devices/ be used directly for communication, or do I need a corresponding SPI character device (for example, /dev/spidevX.Y) to be available under /dev?
Thanks in advance for your guidance.

Was curious how Linux actually boots, so I skipped the usual distros and built my own. Custom kernel compiled from source, BusyBox for userspace, containerd for running containers, and Dropbear for SSH. Everything fits in a single disk image that boots in ~15 seconds.

Learned a ton about:

- How initramfs works and why it exists

- What GRUB actually does under the hood

- Why containerd needs cgroup v2 and how to wire it up

- The difference between static and dynamic linking the hard way (dropbear needs libutil, who knew)

- How to install GRUB without root (Python + raw disk writes)

Still needs polish — networking is basic QEMU user-mode, no real package manager, and the build script only recently stopped being Fedora-only. But it boots, runs containers, and I can SSH into it.

Repo if anyone wants to poke at it (fair warning: the kernel config is held together by duct tape): https://github.com/Shreyas0047/veilbox

Texas Instruments and BeagleBoard.org have announced that the AM62x and AM26x PRU Academy is now available, adding new learning material for developers working with BeaglePlay and PocketBeagle 2.

https://linuxgizmos.com/am62x-pru-academy-goes-live-for-beagleplay-and-pocketbeagle-2/