In the previous article, we explained the implementation of Precision Landing. In this article, which is FC18, we will cover Optical Flow sensors. We will explain the principles of optical flow, which enables stable hovering in indoor or low-altitude environments where GPS cannot be used, as well as its implementation in ArduPilot and integrated design with range sensors.
What is Optical Flow?
Optical flow is a technology that estimates the relative velocity of an aircraft with respect to the ground based on the apparent motion of images observed by a downward-facing camera between consecutive frames.
Characteristic patterns on the ground (floor joints, textures, shadows, etc.) move on the camera image as the aircraft moves. The actual movement speed is calculated from this movement on the image (pixel displacement), the aircraft's altitude, and the camera's field of view.
The basic formula for optical flow relates velocity v, altitude h, focal length f, and angular velocity ω on the image:
ω = v / h (approximate formula, excluding changes in aircraft attitude)
In practice, it is necessary to remove the rotational component using a gyro. Since the apparent motion of the observed image includes both components from the aircraft's translational motion and rotational motion, the pure translational velocity is extracted by subtracting the rotational component using gyro data.
The difference from Visual Odometry, which was explained in NAV13, is that optical flow is a simple method that assumes the ground is a flat plane and combines it with an altitude sensor to determine absolute velocity. VO does not assume a 3D environmental structure and determines movement through triangulation of feature points, making it more complex but capable of operating in environments where the flat plane assumption does not hold.
Types of Optical Flow Sensors
We will organize the main optical flow sensors used in ArduPilot.
PX4FLOW is a dedicated open-source optical flow sensor. It completes optical flow calculations internally using a downward-facing low-resolution camera (752×480) and an embedded processor, and sends the flow velocity to the FC via I2C. It is designed to operate both under sunlight and indoors, and includes a built-in ultrasonic range sensor.
Optical flow modules for Cube/Pixhawk (manufactured by CUAV, etc.) are widely used as successors to PX4FLOW, equipped with higher-resolution sensors and improved image processing algorithms.
The Intel RealSense T265 is a dedicated tracking camera with built-in Visual Inertial Odometry, providing 6DOF pose estimation with higher accuracy than optical flow. Although more expensive than optical flow sensors, it offers higher robustness in complex environments.
A general-purpose camera + companion computer configuration involves connecting a USB camera to a Raspberry Pi or similar device and performing optical flow calculations via software. This is implemented using OpenCV's calcOpticalFlowFarneback or calcOpticalFlowPyrLK (see NAV13).
Optical Flow Calculation Algorithms
We will explain the optical flow calculation algorithms executed internally by dedicated sensors such as PX4FLOW.
The block matching method divides an image into small blocks and searches for the area most similar to each block from the previous frame within a search range in the current frame. SAD (Sum of Absolute Differences) or SSD (Sum of Squared Differences) is used as an indicator of similarity:
SAD(dx,dy) = Σ|I_t(x,y) - I_{t-1}(x+dx,y+dy)|
The (dx, dy) that gives the minimum SAD is the motion vector for that block. While the calculation is simple and easy to implement on embedded processors, mis-matching is prone to occur in areas with little texture.
The Phase Correlation method is a process in the frequency domain that performs high-precision motion estimation through correlation calculation using the FFT of the image. It provides sub-pixel accuracy and is highly robust to noise, but the computational cost is higher than that of block matching.
PX4FLOW performs block matching using a 5x4 grid of blocks and uses the median or mean of the motion vectors of each block as the final optical flow output. Robust statistics (median) are often used to suppress the influence of outliers.
Relationship with Altitude—Focal Length and Scale
Altitude information is essential for determining actual velocity from optical flow.
Conversion from angular velocity ω on the image to actual velocity v:
v = ω × h
h is the altitude above the ground. Even with the same angular velocity, if the altitude doubles, the actual velocity doubles. This corresponds to the intuition that at higher altitudes, the same image movement represents a larger actual displacement.
The accuracy requirement for the altitude sensor directly affects velocity estimation accuracy. Altitude error Δh propagates as velocity error Δv = ω × Δh. High accuracy is required for the rangefinder sensor combined with optical flow; a barometer (±1m) is insufficient, and an ultrasonic or LIDAR rangefinder (±a few cm) is necessary.
The maximum effective altitude depends on the sensor resolution and ground texture. As altitude increases, movement on the image becomes smaller, and the signal-to-noise ratio decreases. The practical maximum altitude for PX4FLOW is about 5 to 10m.
The minimum effective altitude is limited by increased image blur (motion blur) when the ground is too close, and the problem that ground texture may become monotonous beyond the field of view. Generally, accuracy decreases below 0.5m to 1m.
Optical Flow Configuration in ArduPilot
The optical flow-related parameters in ArduPilot are organized as follows.
FLOW_TYPE sets the type of optical flow sensor. 1 is for PX4FLOW, and values are provided for other I2C/MAVLink optical flow sensors.
FLOW_FXSCALER and FLOW_FYSCALER are for adjusting the sensor scale factors. They are used to correct deviations from factory calibration values. Adjustments are made by comparing with measured values during a calibration flight on the ground (round-trip movement over a known distance).
FLOW_ORIENT_YAW is the sensor mounting orientation (offset angle from the airframe axis). Set this if the sensor is not perfectly aligned with the forward direction of the airframe.
By setting EK3_SRC1_VELXY to OpticalFlow (=5), optical flow is used for EKF horizontal velocity estimation. In indoor environments where GPS cannot be used, this becomes the primary velocity source.
Set the rangefinder sensor with RNGFND_TYPE and combine it with optical flow. Many optical flow modules have built-in ultrasonic or LIDAR rangefinders, which should be configured as RNGFND1.
Optical Flow Observation Model in EKF
The optical flow EKF observation model is organized using mathematical formulas.
The observation equation compares the optical flow observations flow_x and flow_y in the airframe coordinate system with the predicted values:
flow_pred_x = (v_x × R31 + v_y × R32 + v_z × R33) / h - ω_y
R31 to R33 are elements of the rotation matrix, and ω is the gyro angular velocity (removal of rotational components). As this equation shows, optical flow directly observes the ratio of 'velocity/altitude,' and the EKF calculates velocity v in reverse by combining it with the estimated altitude h.
EK3_FLOW_M_NSE is the noise parameter for optical flow observation. The default is 0.25. Adjust it according to the sensor's noise characteristics, and increase the value in environments with high vibration.
Optical flow delay compensation is set via EK3_FLOW_DELAY_MS. Many optical flow sensors have a delay of several tens of ms in their internal processing; if this is not compensated for, systematic errors will occur in the EKF's velocity estimation.
Optical Flow and Vibration
Optical flow sensors are sensitive to vibration. The same considerations as the IMU vibration countermeasures in the previous series are required.
Camera motion blur is a phenomenon where images become blurred due to vibration, reducing the tracking accuracy of feature points (textures). Cameras with longer exposure times are more susceptible to blur, so selecting a sensor with a high frame rate (short exposure time) is advantageous in vibrating environments.
Mechanical vibration-damping mounts are also effective for optical flow sensors. Similar to the FC unit itself, isolate the sensor from vibrations using a damping mount.
The accuracy of gyro rotation compensation is directly linked to the accuracy of optical flow. If ω is not accurate in the aforementioned observation equation, the aircraft's rotation will be misidentified as translational motion, causing significant errors in velocity estimation. This is why the gyro noise and bias management explained in the IMU series also affects the accuracy of optical flow.
Coordination with GPS-Denied Navigation in Indoor Environments
We will organize the positioning of optical flow in GPS-denied environments (NAV1 to NAV19 series).
Optical flow is a velocity sensor and does not provide absolute position. If the EKF is operated using only optical flow, position estimation via velocity integration will accumulate drift, similar to INS-only operation during GPS denial. Optical flow serves to make this drift velocity significantly slower than with INS alone.
The combination of optical flow and a range sensor is suitable for 'local stabilization' specialized for hovering and low-speed movement. While its accuracy is inferior to indoor missions requiring absolute position accuracy (VIO in NAV11, LiDAR SLAM in NAV12), the computational load is extremely light (built into the dedicated sensor), enabling low-cost indoor hovering stabilization.
As a practical combination, designs that use basic hovering stabilization via optical flow in conjunction with absolute position correction via VIO or LiDAR SLAM are increasing in industrial drones. The division of roles is that optical flow provides high-frequency, low-latency velocity feedback, while VIO/LiDAR provides low-frequency, high-precision position correction.
Limitations of Optical Flow and Countermeasures
We will organize the situations where optical flow does not function or its accuracy decreases, along with their countermeasures.
In low-texture environments (uniform floors, water surfaces, reflective surfaces), feature points cannot be found, and optical flow does not function. As a countermeasure, it is necessary to create textures artificially by adding infrared illumination or to fall back to other sensors (VIO, LiDAR).
In low-light environments, image contrast in dark areas is insufficient. PX4FLOW has an auxiliary lighting option using built-in LEDs.
During high-speed movement or high-speed rotation, tracking fails due to image blur and exceeding the search range. Check the maximum supported angular velocity of the optical flow in the sensor specifications and restrict its use on aircraft that perform intense maneuvers.
On sloped ground or steps, the planar assumption collapses, causing velocity errors due to inconsistency with altitude estimation. Flying on stairs or ramps inside buildings is a factor that reduces the accuracy of optical flow.
Integration with ROS2—Coordination with px4_optical_flow
We will also touch upon optical flow integration in the ROS2 environment.
MAVROS2 publishes ArduPilot's OPTICAL_FLOW messages as ROS2 topics. This allows optical flow data to be integrated with other sensors in a sensor fusion node on the ROS2 side (such as robot_localization, see NAV9).
For custom optical flow implementations, it is possible to use a USB camera and OpenCV to implement a proprietary optical flow calculation node in ROS2. By using GPU-accelerated optical flow (cv::cuda::FarnebackOpticalFlow) on a Jetson, it is possible to achieve higher resolution and higher precision optical flow than with dedicated sensors. However, a design that considers the trade-off between computational load and latency is required.
Summary
Optical flow estimates relative velocity to the ground from image changes in a downward-facing camera and converts it to actual velocity by combining it with a ranging sensor. Motion vector calculation via block matching or phase correlation is performed inside the sensor, and removing rotational components using a gyro is the key to accuracy. In GPS-denied environments, it does not provide absolute position, so its role is limited to velocity feedback, but it contributes significantly to hovering stabilization with low cost and low computational load. Low texture, low light, and high-speed movement are the main causes of accuracy degradation, and it is necessary to combine it with VIO or LiDAR SLAM depending on the application. In the next installment, FC19, we will explain the mathematical principles of the L1 Controller—ArduPilot's path-following algorithm.