Compute, sensors and perception

Perception answers four questions fast enough to score: where the rover is, what the object in front of it is, what the astronaut just commanded, and whether the view is good enough to drive. Each question has a number attached, and the numbers are smaller than they look. The most useful finding on this page is that the 2027 post tag is a last-five-metres sensor rather than a twenty-metre one, and that the mission design already accounts for it.

What the subsystem must do

The 2027 rules ask for four capabilities with numbers attached. The rover must stop within 1 m of a route target and within 3 m of the astronaut S-URC27-RULES §1.e.xvii. It must detect a 20 cm post tag, a 2 cm keyboard tag and a 1 cm key tag S-URC27-RULES §1.e.xii. It must recognise five astronaut commands, one of which scores 15 points only as a gesture S-URC27-RULES §1.e.vi. And it must give three of the four missions a driveable view, because those missions are teleoperated S-URC27-RULES §1.d.ii.

Two rules then set the architecture. The team must bring and run its own RTK base, with no correction service and no internet in the field S-URC27-QA RQ11. And the second route target sits behind a hill, out of radio line of sight, so anything score-critical has to run on the rover rather than at the C2 station S-URC27-RULES §1.e.xvi. A base-station computer may help, but it cannot be in the critical path S-URC27-QA AUQ2.

The numbers that size the design

Design space, in one table

The pixel budget is the subsystem

An ArUco 4x4_50 tag with a one-cell border is six modules across. With the cell fixed at 2.5 cm, the black extent on a 2027 post is 15 cm inside a 20 cm face, inferred from the face and cell dimensions in the rules. If the tag fills the whole face instead, every range below improves by a factor of 4/3, which makes one photograph of the real post worth taking.

Detection was measured rather than assumed. OpenCV 5.0.0 on a synthetic sunlit-sand background, with 24 noise seeds per point, decodes a clean tag at 12 px, needs 24 px under defocus, 40 px with 9 px of motion blur, and manages 18 to 24 px at 25 % contrast S-P407-ARUCO-TEST. That table converts directly into a design budget of 5 px per module.

Two consequences follow, and both are counter-intuitive.

Full sun is not the blur problem. At the roughly 10⁵ lux of a June afternoon the exposure is short: 1 m/s at 1/1000 s through a 1080p camera at 60° blurs 0.17 px at 10 m, which is invisible. The sun problems are contrast, where a shaded tag sits against bright ground, and flare, where the sun is near the field of view and no exposure setting helps. The answer is a hood on every camera and an exposure set so that the bright background is not clipped.

The keyboard is tractable and the key is not. The 2 cm keyboard tags are twice the 1 cm numbers, which is about 1 m through a 1080p wrist camera against the 0.2 to 0.5 m the key needs. That is why teams put a camera on the arm rather than relying on the mast, and it is why the wrist camera is an arm requirement created by a perception number.

The same arithmetic sizes the human tasks. A suited astronaut is 84 px tall at 20 m through 720p and 150 px through 1080p, which is enough for driving to the astronaut and for “Come!”, since both need only a bearing and a range. Pose and gesture models want 150 to 250 px, so the gesture belongs at 3 to 8 m. That is a team decision rather than a rule S-URC27-QA AUQ10, and voice is an explicitly encouraged fallback S-URC27-QA AUQ6. The hammer is a 0.91 kg object about 300 mm long, which makes it a 5 m object for detection and a 3 m object for grasp planning S-URC27-HARDWARE.

Depth is close range, and the base coordinate owns the whole metre

Stereo depth error grows as the square of range divided by focal length and by the distance between the two cameras. With the cameras 50 mm apart that is ±53 mm at 3 m and ±594 mm at 10 m, and the model assumes 0.2 px of disparity error while real hardware needs about 0.32 px, so the real error is roughly 1.6× the table S-P407-STEREO-CALC S-RS-D435. Three design rules follow.

Depth must not be the primary localiser at range. A D455 at 10 m is within ±0.31 m on one measurement, which is a third of the stop tolerance, and the worst case is worse than the tolerance.

Camera separation beats pixels. Moving the two cameras from 50 mm apart to 120 mm apart improves depth at 10 m by 2.4×. Moving from 720p to 1080p at the same field of view improves it by about 1.0×, because the field of view sets the focal length in pixels. A narrow lens and a wide camera spacing are the only route to centimetre depth at 10 m.

Rolling shutter is unusable on a moving stereo pair. At 1.5 m/s with a 15 ms readout the scene moves 22 mm during a frame, which is about 4.9 px of disparity error, which is 1.3 m of depth error at 3 m. Global shutter and hardware sync are not a luxury on a driving pair; they are the difference between depth and noise.

The 1 cm key task never uses stereo. A mono camera with ArUco pose estimation gives range directly from the known tag size, at about 0.3 % of range, or 1 mm at 0.3 m, which is enough for a 2 to 3 mm insertion.

RTK is only as good as the base station’s own coordinate, and a 1 m base error moves every waypoint by 1 m, which is the entire route-target tolerance. The mitigations in order of cost are a 30 to 60 minute survey-in, because 35 s is the time to first fix rather than to a decimetre absolute solution; reusing one base position file all week so the map and the rover share a single error; and letting the visual tag arbitrate at the end, which is what the task design already does.

One coupling matters most. The correction link is line-of-sight limited, so behind the hill the rover loses RTK corrections in the same shadow that blocks the C2 link. The metre at the second route target is bought with dead reckoning, a pre-loaded route and the tag, not with a live fix.

Compute is adequate; heat is the constraint

An Orin Nano Super 8 GB runs 67 INT8 TOPS at 7 to 25 W and needs no carrier board S-NV-JETSON-ORIN. In practice that is enough for person detection at 60 to 120 fps, pose at 30 to 60 fps, ArUco detection at 60 fps or better on the CPU, one stereo stream and a SLAM back-end. It is not enough to add a 4K stream and a second large model at the same time, and 8 GB is the ceiling rather than a margin.

The thermal arithmetic is the failure that goes unnoticed. A 40 W module in a sealed 300 × 250 × 150 mm box, with natural convection at about 5 W/m²K over 0.3 m², sits 27 K above ambient, or about 65 °C at 38 °C. That is before solar gain on the enclosure itself, which the electrical design also has to remove. Nothing crashes; the model simply runs slower, and the team reads the lost frames as a software problem S-URC27-RULES §3.d.iii.

The fixes are the ones the power page already asks for: size the enclosure for the sustained power mode rather than the peak, add a fan with a filtered inlet, which the rules permit S-URC27-QA RQ12, and soak-test at 40 °C for 40 minutes rather than 5.

The mast carries the consequences. It holds the cameras, the GNSS ground plane and often the radio antennas, it must fold inside the 1.2 m cube, and the terrain excites it at 2 to 15 Hz, so its first mode has to sit above 20 Hz to keep the images still. Every joint in a folding mast is a stiffness question and a cable-fatigue question at the same time.

Interfaces that bite

• Autonomy. Perception owes a tag identity, range, bearing, covariance and timestamp at 10 to 30 Hz, never a six-degree-of-freedom pose computed from a 30 px tag. It owes an obstacle map at 5 to 10 Hz with 0.1 to 0.2 m resolution over a 10 m radius.
• Arm. The 1 cm key tag creates the wrist camera, and the wrist camera lands on the arm’s mass and cable life. A cable through every joint is a fatigue item, so the camera’s driver board belongs in the wrist and the service loop has to be designed rather than discovered.
• Power. Compute draws 10 to 40 W with peaks near 60 W, and the kill has to cut it or the safety rule is not met. The classic coupling is motor stall current browning out the shared USB rail and taking the cameras with it.
• Communications. Three to six concurrent 720p30 streams at 100 to 300 ms is the video load, plus RTCM3 corrections at 0.5 to 2 kB/s. The second route target needs zero link, which is why the map and route must be on the rover.
• Structures and environment. The mast, the camera plate and the GNSS ground plane are one assembly, and the camera count is an operator-load decision as much as a perception decision: one documented 2026 rover carries 11 cameras S-WVU-SAR2026, which costs bandwidth and attention as well as mass.

Failure modes

Shortlist for a first build

Open problems

• Is the post tag 15 cm or 20 cm across? The rule fixes the face and the cell but not the margin. One photograph of a real post at the first practice task settles a 4/3 factor on every range on this page.
• What is the real detection range in Utah sun? The measured numbers are synthetic, with uniform degradations and no glare. A printed 20 cm tag at 5, 10, 15 and 20 m, logged against the time of day and the angle to the sun, replaces them.
• How accurate is the base coordinate? Survey-in for 30, 60 and 120 minutes at a known point, or run two bases and compare. The error maps one to one into the stop tolerance.
• Does the computer throttle in the real enclosure? The thermal figure is a one-dimensional estimate. A 40-minute soak at 40 °C with the real enclosure and the real workload is the test.
• Does gesture recognition work on a baggy suit at 3 to 8 m? No competition data exists. Film 200 gestures from a real suit at four ranges in sun and count the false positives per minute.
• Does a 360° lidar pay for itself? The teams using one also win, which is selection bias rather than proof. The same rover with and without it, on the hill route, measured by the share of runs reaching the 1 m stop, is the comparison.
• How often does a camera drop out? No team publishes a rate. Log every dropout across 20 hours of driving with the arm stalling.

Further reading

• WVU publishes the most detailed perception stack in the record, including a learned marker detector tested to 12 m S-WVU-SAR2023, and Monash shows the wrist-camera and mast arrangement in service. Missouri S&T documents the split between a central computer and motor controllers on CAN.
• Software stack and autonomy consumes every interface this page produces.
• Arm and end-effectors owns the wrist camera mount and the cable route that this page’s 1 cm requirement creates.
• Power, electrical architecture and safety owns the thermal design of the enclosure that the computer sits in.
• Communications and networking sets the video budget that the camera count has to fit.
• Interfaces and couplings follows the mast, camera and compute couplings with numbers.