Member Avatar for tumblinmonkeym

I recently got accepted into a research program for the Computer Science department at my university and I was assigned with the project:

Augmented Reality for Small Unmanned Aerial Systems at CBRN Events
This project will augment video imagery from a small UAS with labels showing responders during a CBRN event where important locations are, such as schools that need to be evacuated or a hospital where victims can be transported to. Currently, a responder has to look at the image feed to see if they see anything that looks like a school or hospital versus a factory or warehouse or stop looking at the imagery, search on Google maps and then try to line up mentally what’s on the map with where the robot is looking. This project applies visualization principles to augment the image with information extracted from Google maps. The project results will benefit the human-robot interaction community by increasing the corpus of successful interface designs and will benefit society as the code will be added to the interface used for CRASAR rescue robots. The project is low risk as it involves trigonometry and simple image processing, with interfacing with Google maps as the only capability expected to be new for a computer science or visualization student.

I have been thinking of how to begin this project and was curious how other people would go about solving this program.

Thank you!

  • 2 Contributors
  • 1 Reply
  • 384 Views
  • 8 Hours Discussion Span
  • Latest Post Latest Post by rubberman

Dani AI

Generated

Brief roadmap and a practical starting plan for Augmented Reality on small UAS (ties to ’s CRASAR goal). The core technical problem is georeferencing: convert POI lat/lon from a map source into 2D image pixels using the UAS pose and the camera model. This is doable as a research project, but it has a few engineering pitfalls (sensor timing, altitude/DEM, lens distortion) that make the "trigonometry" more than toy math. (developers.google.com)

Suggested 4-step workflow to prototype quickly:

  1. Proof-of-concept on the ground: mount the camera at a known GPS coordinate, label several known POIs and verify overlays with the vehicle stationary.
  2. Acquire POIs from a map data source (Google Places / Places API or OpenStreetMap/Nominatim/Overpass as an open alternative) and get each POI’s lat/lon. (developers.google.com)
  3. Convert lat/lon/height -> local Cartesian (ECEF or local tangent / NED) with GeographicLib or pyproj, subtract UAV position to get relative 3D vectors, then apply the UAV->camera rotation/lever-arm transform and the camera intrinsics to project into pixels. (geographiclib.sourceforge.io)
  4. Render labels after filtering by distance, FOV and line-of-sight (use DEM/building heights when required). Iterate with moving flights once the static case is accurate.

Minimal projection sketch (Python-style pseudocode):

from pyproj import Transformer
# world -> ECEF (or use local cartesian)
T = Transformer.from_crs("EPSG:4326","EPSG:4978", always_xy=True)
poi_ecef = np.array(T.transform(lon, lat, alt))
uav_ecef = np.array(T.transform(u_lon, u_lat, u_alt))
rel = poi_ecef - uav_ecef
p_cam = R_uav_to_cam.dot(rel)   # rotate into camera frame (account for mount)
# project with intrinsics K (fx,fy,cx,cy), simple pinhole:
u = (K[0,0]*p_cam[0]/p_cam[2]) + K[0,2]
v = (K[1,1]*p_cam[1]/p_cam[2]) + K[1,2]

Key practical tips and cautions: calibrate the camera (intrinsics + distortion) and confirm with OpenCV’s calibration/project tools; synchronize timestamps between the video frames and the GNSS/IMU stream (MAVLink/companion-computer telemetry); be conservative about POI altitude — use DEMs (USGS/SRTM) or building footprints if overlay accuracy matters; consider RTK or local correction for critical tasks; review Google Maps Platform terms if planning to ship with Google data. (opencv2-python-tutorials.readthedocs.io)

Start with the static POC, log all intermediate transforms, and add one complexity at a time (moving camera, then occlusion handling, then automated semantic filtering). This sequence keeps debugging tractable and produces usable results for CRASAR integration.

Member Avatar for rubberman

I assume this is a graduate-level program? And you are still clueless? What about "research" don't you understand? Don't ask us to do your work for you! This sort of research project is not supposed to be easy or simple, otherwise any schlub could deal with it.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.