NAS Crash Report

Arc Bootloader Version

Component	Installed	Latest Stable	Latest Beta
Arc	3.1.0 beta	3.0.7 (2025-10-25)	3.1.0 (newer build exists)
LKMs	25.11.26	25.04.28	26.02.07
Modules	26.01.13	25.10.22	26.02.18
Addons	26.01.07	25.10.25	26.02.27
Configs	25.12.27	25.10.21	26.02.14
Patches	25.12.27	25.10.25	25.12.27
Custom	26.01.11	25.09.06	26.02.27

Running Arc 3.1.0 beta, ahead of stable 3.0.7. See stable releases and beta releases.

Arc Config Summary

Setting	Value
Model	DS36xx-class (masked)
Platform	BROADWELLNK
Kernel	4.4.302+ (DSM 7.3.x line)
LKM mode	prod
DSM	7.3.2-86009
Boot device	USB mass storage (masked)
Serial	[REDACTED]
Kernel panic	5 (auto-reboot after 5s)
Governor	conservative
Console	`ttyS0,115200n8`

Crash Timeline

Time (PST)	Event	Source
10:24:22	Last IPv6 SLAAC event logged	syslog
10:30:00	Last successful SNMP scrape (up=1)	Prometheus
10:32:00	Last temperature reading: 60°C	Prometheus SNMP
~10:33–10:38	CRASH Kernel NULL pointer dereference	inferred
10:40:00	First failed SNMP scrape (up=0)	Prometheus
10:50:00	Still down	Prometheus
11:00:00	Still down (BIOS POST + disk spin-up)	Prometheus
11:02:46	Kernel boots, redpill module loads	kern.log
11:03:50	Volumes mounted, services starting	syslog
11:03:51	`synocheckshare` fails on /volume1	syslog
11:17:58	Detects Improper Shutdown	shutdown log
11:18:42	Boot finishes: 1m16s kernel + 15m58s userspace	systemd
11:20:00	UP SNMP back online	Prometheus

Total downtime: ~48 minutes. Crash at ~10:33, SNMP restored at 11:20. Breakdown: ~22 min hardware reboot, ~16 min systemd snmpd deadlock (now fixed).

Root Cause

EFI pstore contains crash dumps from multiple boots showing the same bug — a NULL pointer dereference in the redpill module's virtual serial UART device.

Kernel Oops

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: _raw_spin_lock_irqsave+0xf/0x30
CPU: [masked]  PID: [masked]  Comm: insmod  Tainted: G  OE  4.4.302+ #86009

Call Trace:
  serial_remote_read+0x30/0x3b0 [redpill]    ← NULL deref here
  serial8250_config_port+0x304/0xd70
  uart_add_one_port+0x14f/0x320
  serial8250_register_8250_port+0x2e4/0x410
  vuart_add_device+0x257/0x3f0 [redpill]      ← virtual UART init
  register_pmu_shim+0x14/0x160 [redpill]
  init_+0xc0/0xeb [redpill]                   ← module load

serial_remote_read dereferences a NULL pointer during spinlock acquisition in the virtual UART path.

Pstore shows boot-time crashes during insmod, but today's crash happened ~15 days into uptime — the same vuart code path can be triggered at runtime by DSM processes interacting with serial console state.

The console=ttyS0,115200n8 kernel parameter keeps the serial console active, increasing vuart activity.

Shutdown History

Date	Type	Days Since Last Boot
Jan 22	Normal	—
Jan 23	Improper	1 day
Jan 24	Improper	1 day
Jan 26	Improper	2 days
Jan 27	Improper	1 day
Feb 3	Normal	7 days
Feb 3	Improper (same day)	<1 day
Feb 4	Improper	1 day
Feb 11	Normal	7 days
Feb 14	Normal	3 days
Mar 1	Improper	15 days

6 crashes in 5 weeks. Cluster in late January / early February, then stable until today.

What Was Ruled Out

Factor	Status	Evidence
Temperature	Normal	55–60°C via Prometheus SNMP
Disk failure	Passed	`smartctl -H` all drives
RAID degradation	Healthy	status=1; [UUU] / [UUUU]
OOM / memory	Clear	No OOM killer messages
Power loss	Unlikely	No UPS events; other hosts unaffected
Other kernel panic	No	Pstore shows redpill oops only
Disk realloc sectors	Unrelated	High POH, 0 pending, SMART passed

Recommendations

1

Update Arc to the latest beta build

Beta releases are where fixes land first. Arc beta releases →

2

Remove the serial console parameter

Drop console=ttyS0,115200n8 to reduce vuart activity, since the crash is in the virtual UART path.

3

Consider stable if crashes continue

Arc stable 3.0.7 or file an issue with the pstore dump. Arc stable releases →

4

Keep BTRFS volume headroom

Near-full volumes are risky during repeated dirty shutdowns.

Arc Beta Upgrade Guide

Update Arc beta → newer beta safely via either method below.

Encryption warning. Method A (re-flash) will cause hardware encryption keys to be lost. Plan accordingly.

Official Resources

Resource	Link
Arc beta releases	github.com/AuxXxilium/arc-beta/releases
Arc stable releases	github.com/AuxXxilium/arc/releases
Xpenology Wiki	xpenology.tech/wiki
Arc Loader page	xpenology.tech/arc-loader
Arc project repo	github.com/AuxXxilium/arc

Choose a Method

A: Replace Boot Image recommended

Maximum reliability. Re-flash a fresh Arc beta image onto the boot device.

B: Update Mode

Fastest path. Let Arc update itself via its built-in update workflow.

Preflight Checklist

Item	Why It Matters
Confirm platform format	Arc publishes `.img` `.vmdk` `.ova` `.vhd`
Loader UI access	Web UI on port 7080, terminal on 7681
SSH access	SSH as `root` / `arc`
Schedule downtime	At least one reboot, possibly two
Use USB 3.0+	Slow USB can cause kernel panic
Keep boot device attached	DSM depends on it every startup

Method A: Replace Boot Image

1

Download the correct artifact

Pick the newest release from Arc beta releases. Match image type to your environment.

2

Flash the new loader image

For bare metal: flash to USB, set BIOS first boot device. For VMs: attach at SATA 0:0.

Terminal

# Verify /dev/diskN carefully before running
diskutil list
diskutil unmountDisk /dev/diskN
sudo dd if=arc-beta.img of=/dev/rdiskN bs=4m conv=sync
diskutil eject /dev/diskN

3

Boot into Arc Config Mode

Select model → Build Loader → Boot Loader. Web UI at http://IP:7080.

4

Restore config cautiously

Not all settings may be compatible with new loaders. Re-check model, platform, addons, and PortMap.

Method B: Update Mode

1

Reboot into Update Mode

Terminal

sudo loader-reboot.sh update

2

Or Config Mode to review first

Terminal

sudo loader-reboot.sh config

3

Run the update flow

Use the "Update" option in Arc Config Mode. For advanced config, run arc.sh.

Verify & Rollback

Cross-check against published 3.0.7 metadata on the Arc Loader page. For rollback, grab stable artifacts from Arc stable releases.

Agent-Assisted Upgrade

You can use coding agents to generate a safe, auditable upgrade runbook — but keep a human in the loop for every destructive action (flash, write, reboot).

Tooling Overview

Tool	Description
Claude Code	Terminal-oriented agentic tool. Edits files, runs commands, works with git.
Conductor	Mac app to orchestrate parallel agents in isolated workspaces with review/merge workflow.
Google Antigravity	Agent-first IDE with async agent workflows and reviewable Artifacts.

What to Automate Safely

Have the agent produce audit artifacts: a plan.md runbook, a rollback.md, and a commands.sh that is print-only by default. Then you manually run only what you approve.

Claude Code Runbook

Install and start a session on your Mac:

Terminal

curl -fsSL https://claude.ai/install.sh | bash
mkdir -p arc-upgrade && cd arc-upgrade
claude

Create a CLAUDE.md with guardrails:

CLAUDE.md

# Arc beta → newer beta upgrade (HITL)

Non-negotiables:
- Do not run destructive commands automatically
  (dd, diskutil eraseDisk, wipefs, parted, mkfs, fdisk, zpool, mdadm)
- Always print the exact command and wait for explicit human approval
- Require the user to paste diskutil list output before any disk-write
- Produce artifacts: plan.md, rollback.md, verification.md, commands.sh

Deliverables:
1) plan.md — step-by-step runbook tailored to environment
2) commands.sh — SAFE (echo only); dangerous commands commented out
3) rollback.md — exact steps to revert if boot fails
4) verification.md — post-upgrade checks

Prompt to generate the upgrade plan:

Prompt

You are helping me upgrade Arc loader from an older beta to a newer beta.
Work in this folder only. First, ask me for:
(1) my platform (bare metal USB vs Proxmox/ESXi/etc)
(2) where the boot media is
(3) whether I can tolerate downtime
(4) whether I use encryption keys

Then create these files:
- plan.md (detailed, step-by-step)
- commands.sh (SAFE: echo-only; no destructive commands executed)
- rollback.md
- verification.md

Do not assume device names. Require me to paste diskutil list
before any write steps. Include official Arc links.

Conductor (Multi-Agent)

Separate concerns across parallel agent workspaces, then merge only what you approve:

Workspace	Agent Responsibility
`runbook-agent`	Write plan.md with environment questions + decision points
`safety-agent`	Write rollback.md + "what could brick me" warnings
`ops-agent`	Write verification.md (checks before/after reboot)

Docs: docs.conductor.build

Google Antigravity

If using Antigravity, treat it the same way: make it generate audit artifacts and require explicit confirmation before any disk-write or reboot step.

Key principle: Agents produce artifacts you can audit. You run only the commands you approve. Always keep a rollback path ready before the first reboot.

Arc Bootloader Version

Arc Config Summary

Crash Timeline

Root Cause

Shutdown History

What Was Ruled Out

Recommendations

Update Arc to the latest beta build

Remove the serial console parameter

Consider stable if crashes continue

Keep BTRFS volume headroom

Arc Beta Upgrade Guide

Official Resources

Choose a Method

A: Replace Boot Image recommended

B: Update Mode

Preflight Checklist

Method A: Replace Boot Image

Download the correct artifact

Flash the new loader image

Boot into Arc Config Mode

Restore config cautiously

Method B: Update Mode

Reboot into Update Mode

Or Config Mode to review first

Run the update flow

Verify & Rollback

Agent-Assisted Upgrade

Tooling Overview

What to Automate Safely

Claude Code Runbook

Conductor (Multi-Agent)

Google Antigravity

Sources