Rock-Bolt Perception: 6-DoF Pose Estimation for Mining Automation
A perception pipeline that isolates partially occluded rock bolts on a mining conveyor and estimates their full 6-DoF pose to 0.055 m centroid error, accurate enough to hand off to a robot arm for removal; validated in an ABB arm simulation.
Role: Perception Engineer, 6-DoF Pose Pipeline
Organization: ABB-sponsored project, Umeå University (Sweden)
Timeline: Mar 2025 – Jul 2025
· Mask R-CNN · PointNet++ · RGB-D Perception · ROS2 · RViz
End-to-end run: the pipeline detects a bolt, estimates its pose, and feeds the centroid to an ABB arm in RViz, which reaches the bolt and removes it.
What I Built
Built one of two pose-estimation pipelines on a 10-person ABB project, aimed at cluttered, dusty conveyor scenes where bolts are half-buried and depth data is noisy.
Fine-tuned Mask R-CNN on 1,000+ RGB-D frames to segment individual bolts even under heavy occlusion.
Turned each segmented mask into an object-level point cloud using aligned depth and camera intrinsics, then trained a PointNet++ model to regress the full 6-DoF pose.
Ran the whole thing in ROS 2 and checked the pose output against an ABB arm in RViz to confirm a robot could actually act on it.
Results
0.055 m centroid pose error, evaluated against 10-frame manually-labeled ground truth.
Grasp-feasibility confirmed in ABB arm simulation (RViz) and pose output is usable for downstream robotic bolt removal.
In the team's comparative evaluation, the segmentation + PointNet++ approach held up better under clutter and partial occlusion than the alternative PCA-based pose method.
Pose-Estimate Workflow: detected bolt is segmented by Mask R-CNN, converted to point cloud, then passed to PointNet++ to output a full 6-DoF pose.
(a) segmented mask (Mask RCNN); (b) point cloud generation; (c) pose-estimation with color code
(white= centroid, green= bolt’s end points, yellow= camera center)
System Overview
The project automated rock-bolt removal from mine conveyors, a task that's unsafe and slow to do manually in dusty, cluttered conditions with heavy occlusion and poor depth quality. The perception effort was split into two parallel pose pipelines to compare robustness against efficiency. My pipeline processed RGB-D input through instance segmentation to isolate each bolt, projected the segmented region into 3D to build an object-level point cloud, and regressed 6-DoF pose with PointNet++.
Key Technical Decisions
Segmentation-first over detection-first: isolating each bolt by mask before extracting its point cloud produced cleaner object-level clouds under clutter than a bounding-box crop would.
Learning-based regression over geometric fitting: the alternative pipeline used PCA on extracted clouds. Lightweight, but dependent on geometric assumptions that break down under occlusion and noisy depth. Direct PointNet++ regression proved more robust to partial visibility and uneven depth quality.
Learnings & Limitations
Single-frame pose estimation is sensitive to severe occlusion and depth noise.
Temporal fusion or multi-view consistency would meaningfully improve robustness.
Active-perception strategies could reduce ambiguity before grasp.