Wearable Sensor-Fusion Survey Rig

Subtitle: Architecture for hands-free ecological observation, co-registration, and in-situ situational awareness

Document ID: CNL-FN-2026-XXX (sequence auto-assigned by archive importer) Version: 0.1 Date: July 23, 2026 Author: Michael P. Hamilton, Ph.D. Visibility: Private

AI Assistance Disclosure: This field note was developed with assistance from Claude (Anthropic, Opus 4.8). The AI contributed to hardware specification review, architecture synthesis, and manuscript drafting during a design dialogue conducted July 23, 2026. The author takes full responsibility for the content, accuracy, and conclusions.

Abstract

This note records the conceptual architecture for a wearable sensor-fusion rig intended to support ecoSPLAT surveys at Canemah Nature Laboratory field sites. The rig pairs Brilliant Labs Halo augmented-reality glasses with an NVIDIA Jetson Orin fusion hub carried in a pack, together with acoustic and positional sense nodes, an existing 360° imaging rig, and an iPhone serving as voucher camera and network bridge. Halo functions not as an imaging device but as a head-pose sensor, a canopy-light probe, a capture trigger, and a heads-up display surface. The Jetson acts as master clock, disciplining all subordinate nodes to a single timebase so that heterogeneous streams — position, acoustic detections, imagery, head orientation, ambient luminance, and spoken field notes — can be co-registered after the walk. A latency-tolerant API path to Galatea provides deep-time archive context and non-real-time species identification. The note also specifies a privacy protocol appropriate to surveys conducted in public spaces, in which the rig is dark by default and armed only by deliberate physical action. Hardware acquisition has not yet occurred; this document precedes purchase and is intended to establish the design against which the Halo evaluation will be judged.

1. Introduction

1.1 Motivation

Current ecoSPLAT survey practice separates observation from interpretation. Imagery, acoustic detections, and positional tracks are captured in the field and reconciled at the workstation afterward. The observer therefore learns what they walked through only after leaving the site, and any anomaly worth investigating is discovered too late to investigate.

The proposal here is to close that gap: a rig that fuses its own streams in real time and presents a compact, glanceable summary to the observer while they are still standing in the habitat. The intent is not to automate the survey but to make ecological context perceptible in the moment, where field judgment can act on it.

1.2 Objectives

1. Track survey routes with position and, critically, facing — where the observer is looking, not merely where they stand.
2. Co-register heterogeneous backpack sensor streams against a single reliable timebase.
3. Present live situational awareness through a heads-up display.
4. Add a spoken field-notebook layer that is born geotagged and time-aligned to the sensor record.
5. Do all of the above under a privacy protocol defensible in public spaces.

1.3 Scope

This is an architecture and requirements note. It does not specify code, wire protocols, message schemas, or bill of materials. Those follow in a specification document once hardware is in hand.

1.4 What This Rig Is Not

The rig does not replace or supplement the 360° imaging rig as an ecoSPLAT capture instrument. Halo carries a VGA-class camera behind an 81.2° lens; Gaussian splatting quality scales with input pixel density and view overlap, and feeding it such frames would reverse the move to 360°-native single-pass splatting. The 360° rig remains the imaging instrument. Halo contributes positioned, oriented, metered observations that register into the Gaussian field the 360° rig produces.

2. Architecture

2.1 Layer Overview

The rig is a tiered federation rather than a monolith. Each node performs one function well and emits conclusions rather than raw data.

Table 1. Rig layers and responsibilities

2.2 The Fusion Hub

The Jetson Orin was selected over a Raspberry Pi as the hub for a specific architectural reason rather than raw capability. On a Pi, model inference consumes the CPU budget that fusion requires. On the Jetson, inference executes on the GPU, leaving the CPU free for time discipline, stream alignment, and the Bluetooth Low Energy conversation with Halo.

A second consequence is that the Halo NPU — an Arm Ethos-U55, currently without a public API in the Halo Lua interface — becomes irrelevant to the design. Any inference that might have been wished onto the glasses runs in the pack at far greater capacity, and only the result is pushed to the display. Halo remains a thin pose-and-glass client by design, not by limitation.

2.3 Distributed Sense Nodes

Dedicating a Pi-class node to each sensory modality yields three benefits: nodes may be added or swapped without modifying the hub; a node failure degrades rather than ends a survey; and each node can be developed and tested independently.

The commercially available bird-and-weather puck currently carried does not expose its GPS or detection stream in real time. This is disqualifying for live fusion. BirdNET-Pi provides an open equivalent that emits detections as they occur, and a companion Pi with a GNSS module supplies the position track over Bluetooth.

2.4 Deferred Inference

Not all identification needs to be immediate. BioCLIP 2 is a large vision transformer, poorly suited to an AI-hat workload and unnecessary in real time for plant identification. It is therefore hosted on Galatea and queried over the network, with the iPhone hotspot providing connectivity in the field. Results return to the session record rather than to the display.

This split — fast local inference for what must be acted on now, deferred remote inference for what can wait — keeps the pack light and the HUD responsive.

3. Time Discipline

3.1 The Central Problem

A distributed rig is fundamentally a temporal problem, not a spatial one. Every node carries its own clock. Co-registration quality is bounded by the worst-synchronized node in the federation, and the value of the entire architecture leaks away if that bound is loose.

3.2 Master Clock

The Jetson Orin serves as master clock. All subordinate nodes are time-slaved to it over the local link. Halo exposes a settable real-time clock, so the hub disciplines the glasses at session start along with everything else.

3.3 Known Seams

Two streams are not directly slaved and require attention:

• The 360° rig is an independent recorder. Aligning its footage to the hub clock remains the hardest unresolved seam in the design. A synchronization event at session start — an acoustic or visual marker observable to both — is the likely mitigation.
• The iPhone is not slaved at all (see §5.2), but its network time discipline is sufficient for the deferred join it participates in.

4. Halo: Role and Justification

4.1 Function

Halo occupies four roles simultaneously.

Pose sensor. GPS yields a breadcrumb; Halo's e-compass and accelerometer give every breadcrumb a facing. For registering a directional acoustic detection or a panoramic frame to the ground, orientation is the missing dimension, and no other node in the rig supplies it.

Canopy-light probe. The camera's metering path reports per-frame luminance histograms and per-channel averages. Because the Lua image pipeline is composable, automatic white balance and black-level correction can be omitted, yielding radiometrically comparable readings across a walk. Pointed upward, this is a continuous sky-fraction proxy — the quantity ecologists conventionally chase with hemispherical photography or a densiometer, here sampled continuously along the transect and timestamped against every detection.

Capture trigger. Button, tap, and head gesture provide hands-free control of the whole federation from the observer's face.

Display surface. A 256 × 256 drawable region within a 640 × 480 OLEDoS panel, addressable in 16 named palette colors.

4.2 The Two-Brain Convergence

The display's value is not that it shows sensor data. It is that it is the only place where two categorically different kinds of knowledge meet in the observer's field of view:

• Present tense, from the edge nodes: something is calling, at this bearing, right now.
• Deep time, from Galatea: what has been recorded at this location across prior surveys and seasons.

Because Halo knows heading and pitch, both classes register to where the observer is actually looking.

4.3 Structure–Composition Coupling as Design Case

The motivating observation comes from Waterboard Park, where the route passes between tall closed canopy and open gaps affording a vista. Bird composition tracks that structural gradient: eagles and turkey vultures (Cathartes aura) over open sky, robins, thrushes, and Bewick's wren (Thryomanes bewickii) beneath closed canopy.

Every term in this coupling is independently sensed by the rig. Halo's metering quantifies canopy openness; BirdNET-Pi quantifies the acoustic community. The HUD is where the two variables are watched moving together. Crossing from timber into a gap, the light metric rises and the detected guild shifts on the same glass.

The scientific value lies in the residuals — the gap where expected soaring birds are absent, the closed patch that is anomalously quiet, the ecotone where both guilds overlap. These are hypothesis-generating surprises, and they are only available to an observer who can perceive the pairing while standing in it.

4.4 Display Budget

A 256-pixel circle is a glance medium, not a dashboard. The design constraint is not what can be displayed but what earns a place in the visual field without becoming noise the observer learns to ignore. A working minimum for a survey point is two quantities: current canopy-openness reading, and current detected guild. Detailed HUD information budgeting is deferred to a subsequent note.

5. iPhone Integration

5.1 Roles

The iPhone contributes four things and is not treated as a real-time node.

1. Voucher camera. Apart from the 360° rig, it is the only competent imaging sensor in the federation. Any observation requiring a defensible image is captured here.
2. Species-identification node. iNaturalist and Seek provide on-device visual classification, addressing what is seen rather than what is heard.
3. Network bridge. The Jetson has no cellular radio. The iPhone hotspot carries Galatea queries and mid-walk uploads.
4. Backup position. A redundant GPS track independent of the Pi position node.

5.2 Integration by Timestamp, Not Stream

iNaturalist is a closed application with no live outbound feed, and contesting iOS background-execution limits to build one is not justified. Instead the phone participates through deferred temporal join: both the phone and the Jetson are network-time disciplined, every iNaturalist observation carries a timestamp and coordinates, and the session window is retrieved from the iNaturalist API after the walk. Each observation then lands in the fused record against the corresponding pose, luminance, acoustic detections, panoramic frames, and spoken notes.

The phone therefore need not communicate with the rig during the survey. It needs only to be honest about when.

5.3 Optional Event Marking

A Shortcuts automation firing on camera or iNaturalist launch could ping the Jetson, marking the fused timeline with "a deliberate observation occurred here" before the observation itself syncs. This is desirable but not required.

5.4 Conceptual Contribution

The sensor federation narrates continuously and indiscriminately. The phone records only the moments the observer chose to stop and voucher. That record of intent — where the observer decided something mattered — is itself a data layer, and it aligns naturally with the spoken-note channel described below.

6. Field Notebook Layer

6.1 Rationale

Every layer described so far is the instrument observing the world. The notebook layer places the observer in the record: reasoning, captured at the moment it occurred, in the same timebase as everything else. This is the layer that distinguishes a survey from a sensor log, and it is not currently integrated into ecoSPLAT.

6.2 Mechanism

Halo carries dual MEMS microphones. On trigger, audio streams to the Jetson, where transcription is performed locally or forwarded to Galatea. Because the Jetson is master clock, the note is stamped in alignment with position, pose, luminance, and detections at the instant of utterance. The note is born georeferenced; no post-hoc registration is required.

6.3 Mutual Annotation

Spoken notes and machine detections annotate one another. A remark such as "considerable vulture activity over the gap" coincides with a luminance excursion and whatever the acoustic node logged in the same second. Retrospective analysis can then ask which remarks the sensors corroborate and — more valuably — where observer attention and machine detection diverged. Those divergences are where three decades of field judgment reside, and the notebook layer is what prevents them from evaporating.

6.4 Audio Is Ground Truth

Speech-to-text reliably mangles taxonomic nomenclature; Bewick's and Swainson's are precisely the constructions that fail. The audio recording is retained as ground truth in all cases. The transcript is an index into the audio, never a replacement for it. The HUD should echo the transcription briefly so that an error can be caught in the field rather than puzzled over at the workstation.

7. Privacy Protocol

7.1 Operating Context

Survey sites are potentially public spaces. Existing CNL practice for 360° photography is to request permission from people nearby before capturing; consent is almost always granted. The wearable rig must extend that practice, not erode it.

7.2 Dark by Default

The rig records nothing until deliberately armed. This is stronger than continuous capture with subsequent discard: data never collected requires no justification and cannot be disclosed.

7.3 Wake Word Caveat

A true on-device wake word requires the microphone to sample continuously — the always-on acoustic activity detection path draws approximately 20 µA and never transmits. For most consent standards this is acceptable and equivalent to a phone awaiting its trigger phrase. For a stronger guarantee, the microphone should remain unpowered until a physical action energizes it.

7.4 Specified Protocol

Button arms; voice operates; glass shows state.

The physical press is the consent gesture: unambiguous, immune to false triggering by nearby conversation, and honestly narratable to a bystander. Within the armed window, trigger words drive the workflow — begin a note, drop a waypoint, stand down. The HUD displays capture state continuously so that the observer always knows, and can truthfully state, what is being recorded.

Trigger vocabulary is not yet specified and is deferred to a subsequent note.

8. Power

A large portable power station supports the pack indefinitely at survey timescales: days of runtime for a puck-class node, hours for the Jetson. The iPhone sustains a full day unaided. Power is therefore not a constraining factor for the distributed architecture.

The single exception is Halo itself, whose approximately 1.1 W·h internal cell cannot be conveniently supplemented mid-walk. Halo is consequently the only duty-cycle-disciplined node in an otherwise power-rich rig — which reinforces the dark-by-default protocol of §7 from a second direction. Vendor specification indicates all-day operation under a workload heavier than that proposed here (continuous voice interaction with streaming display), and this is accepted.

9. Pre-Purchase Position

9.1 Assessment

Halo is assessed as a buy for the pose-sensor, light-probe, trigger, and display roles described in §4. It is assessed as unsuitable for ecoSPLAT image capture (§1.4). No other commercially available device provides head-height, hands-free, orientation-tagged optical sampling with an openly composable image pipeline at this price point.

9.2 What Can Be Built Before Acquisition

Substantially everything. The Halo emulator embeds a full Lua 5.3 interpreter, stubs the device API, renders to a framebuffer, and supports injection of button, tap, IMU, and Bluetooth events under pytest. The complete on-device field protocol — trigger state machine, capture logic, standby and wake behavior, message parsing — can be developed and tested on Data without hardware. The Jetson fusion node, time discipline, and iNaturalist deferred join are independent of Halo entirely.

9.3 What Cannot

Two questions require hardware: the stability and linearity of metering data for radiometric use, and display legibility and power behavior under open-sky daylight at a survey duty cycle.

9.4 Platform Risk

Halo firmware is at 0.6.2-rc, the Lua namespace still reflects its Frame lineage, and the NPU has no public inference API. Acquisition is an investment in a moving platform. The tiered architecture mitigates this: no rig capability depends on Halo doing anything beyond pose, metering, trigger, and draw.

10. Open Items

1. Trigger vocabulary specification (§7.4).
2. HUD information budget for the 256 × 256 glance surface (§4.4).
3. 360° rig time synchronization to hub clock (§3.3).
4. Thermal behavior of a multi-node cluster inside a closed pack.
5. Jetson Orin model selection.
6. Message schema for hub-to-glass and glass-to-hub traffic.

References

[1] Brilliant Labs (2026). "Halo Hardware Documentation." https://docs.brilliant.xyz/halo/hardware/ (accessed July 23, 2026).

[2] Brilliant Labs (2026). "brilliant_sdk." https://github.com/brilliantlabsAR/brilliant_sdk (accessed July 23, 2026).

[3] Kahl, S., Wood, C. M., Eibl, M., & Klinck, H. (2021). "BirdNET: A deep learning solution for avian diversity monitoring." Ecological Informatics, 61, 101236. https://doi.org/10.1016/j.ecoinf.2021.101236

[4] McEntee, J. (2024). "BirdNET-Pi." https://github.com/mcguirepr89/BirdNET-Pi (accessed July 23, 2026).

[5] Stevens, S., et al. (2024). "BioCLIP: A Vision Foundation Model for the Tree of Life." Proceedings of CVPR 2024. arXiv:2311.18803.

[6] iNaturalist (2026). "iNaturalist API." https://api.inaturalist.org (accessed July 23, 2026).

[7] NVIDIA (2026). "Jetson Orin." https://developer.nvidia.com/embedded/jetson-orin (accessed July 23, 2026).

Document History

End of Field Note