CMU Moon-Miners: Autonomous Lunar-Mining Rover (NASA Lunabotics 2026)
I was the sole owner of the camera-perception stack on CMU's first NASA Lunabotics rover. I trained the rock-segmentation model, built the 2D obstacle mapping that fed the rover's Nav2 navigation, and localized the scoring zone from fiducial corner rods.
Role: Perception Engineer (sole owner, camera perception)
Organization: CMU Moon-Miners, Carnegie Mellon University (Pittsburgh, PA)
Timeline: Jan 2026 - May 2026
· YOLOv8-seg · ROS2 · RGB-D Perception · Nav2 · Python
The rock-segmentation model running on real arena footage, masking the obstacles the rover has to avoid.
What I Built
Owned the entire camera-perception stack solo, from data generation through to the navigation interface, on CMU's first-ever Lunabotics entry.
Solved a no-data problem with a two-stage training strategy. No arena footage existed at the start, so I bootstrapped a YOLOv8-seg model on screenshots pulled from prior-year competition videos on YouTube, then fine-tuned it on rosbags recorded in the real arena once we had access.
Turned 2D detections into a map the rover could drive on: projected segmented obstacles onto the ground plane using camera intrinsics and interpolated transforms, smoothed detections across a 5-frame window to kill flicker, and fed the result into Nav2 costmaps.
Localized the scoring zone from four fiducial corner rods to give the rover a target-region pose to navigate toward.
Prototyped a synthetic, rulebook-spec crater-detection dataset as an exploratory extension.
Results
Deployed in competition. The obstacle-detection and scoring-zone perception ran live on the rover in the real arena, driving navigation and targeting.
Fine-tuned segmentation reached ~0.65 mAP50-95 on real-arena data.
The bootstrap-then-fine-tune approach produced a working model despite zero arena training data existing when the project started.
A full run's obstacle map: every rock the pipeline detected along the rover's traverse, in the map frame used for planning.
The live pipeline in one frame. Left, the camera view with detected rocks. Right, those rocks projected into a top-down map with the rover's position, ready for Nav2 to plan against.
System Overview
NASA Lunabotics tasks a student team with building an autonomous rover that mines and deposits lunar-regolith simulant on dust-prone analog terrain. This was CMU's first entry. My job was the camera-perception stack: detect obstacles and the scoring zone from RGB-D and hand usable maps to navigation. The rover also carried a LiDAR subsystem owned by a separate teammate; my work was the camera pipeline.
Key Technical Decisions
Bootstrapping from public footage instead of waiting for arena data. With no arena footage to train on, starting from prior-year competition video got a working base model early, and fine-tuning on real rosbags later closed the gap to real conditions. It kept perception unblocked while the rover was still being built.
Ground-plane projection with temporal filtering over raw per-frame detections. Projecting detections onto the ground plane gave metric obstacle positions Nav2 could use, and 5-frame temporal filtering suppressed the flicker and false positives that would otherwise have corrupted the costmap.
Learnings & Limitations
The ~0.65 mAP reflects a small real-arena dataset. More labeled arena data would raise it.
Crater detection was prototyped on synthetic data only and not deployed.
Perception was camera-only; sensor fusion with the LiDAR subsystem was out of my scope.