ubik 360 Optical Sensor

We wanted to be able to stream live video from all around the drone without using a motorized gimbal. Gimbals are great but in our case we are mostly interested on the unlimited, panoramic real time video viewing and monitoring instead of the stabilization part. On the other hand, gimbals come with a lot of overhead i.e. they add extra weight and power consumption, usually they are not plug-n-play but require case-by-case optimizations and finally they come at a cost. The short answer might be that we prefer the digital approach.

The ubik 360 Optical Sensor does not only cover the region below the drone as most 180 degrees FOV spherical cameras do but also a good part of the horizon where a lot of activity takes place and it is required for visual navigation. The 360×240 FOV is adequate since it covers 60 degrees on the horizon plane (+/-30 degrees above/below the horizon). Also, as the camera usually mounts at the drone fuselage belly, there was no reason to try to increase the FOV further as the mounting point will inevitably hide the remaining FOV.  Therefore, the 360×240 FOV was the optimum solution.

We wanted a single camera set-up, mostly for simplifying the integration. There are many challenges we had to overcome to make the single camera spherical video comparable to multi-camera rigs and these are related to the image rectification and georegistration algorithms. We finally managed to get a good balance between frame rate and resolution.

What are the benefits of using a single camera instead of multi-camera rigs?

• No real time image stitching is required. Image stitching requires additional algorithms that are CPU-hungry for real time video streaming. The single camera solution requires no external processors attached to the cameras.
• The smaller the number of cameras, the less bandwidth is used.
• Less power requirements
• (Significantly) Less weight
• (Significantly) Less expensive
• It is much easier to integrate to existing drones

What are the downsides of using a single camera for spherical real time video?

• Depending on the Fisheye FOV, wider FOV cameras require better algorithms to faster rectify the Region of Interest (ROI) the viewer is looking at, in order to produce undistorted video.
• Real time spherical video streaming is a very demanding process as the fisheye rectification algorithms are applied on every pixel of every frame multiple times per second i.e. at a rate of 25fps.
• To that extend, higher camera resolutions can stall the real time video rectification process and prevent real time footage. A balance must be maintained between the camera resolution and the frame rate to obtain a viable result.

The ubik 360 Optical Sensor software creates a video bubble and allows the viewer to either monitor it externally from any perspective as a “georegistered” object or get inside the bubble and have a more immersive experience. “Georegistered” means that the video bubble is projected on its real location on the map. Orientation of the sphere with respect to the drone is also possible but requires integration of MAVlink messages. We have achieved both location and orientation georegistration with the Pixhawk autopilot.

This process creates an FPV “spatial awareness” that is missing from current video monitoring techniques. As a direct outcome, we can click on objects that we see on the video and extract their geographic coordinates (in real time). Keep a note on that as the video georegistration allows new navigation functions like “fly-by-vision” mode we are currently developing, etc.

The algorithms used by the ubik 360 Optical Sensor have been written from scratch to allow the spherical projection without using exotic GPUs. The ubik 360 Optical Sensor spherical video can be streamed and viewed on conventional laptops and tablet PCs.

ubik 360 Optical Sensor
Preliminary Specs

Dimensions: 64x44mm
Weight with enclosure and lens: 62 gr

Operating temperature: -20 to +60 C
Power: 12VDC (3S battery compatible), <3W

Optical Properties

Field of View: Horizontal: 360,  Vertical: 240
Number of cameras: single camera

Lens: 1/2.3, f:1.2

Interface

Ethernet, 4 wires

Video Quality

1080p: 15 fps
720p: 25 fps

Compression: H.264/H.265
Buffer delay: 100-250ms

• Uniform 360×240 image without stitching (expandable to 360 x 280 with a different lens).
• No external GPU or image processing boards required.
• Projection method:
  • Georeferenced Spherical, Standalone spherical, Region of interest
  • Region of interest (ROI): Custom algorithms rectify the fisheye image in real time with virtually no distortion at 720p/25 fps to create a 60×60 degrees undistorted video window. The video window is scalable up to 120×120 degrees FOV.
• Compatible devices:
  • Main application runs on Windows and Android machines
  • VR Headsets: GearVR, Google Cardboard and any Android smartphone compatible VR headset
• Un-gimbaled operation: Select “Point-camera-here” in real time streaming to keep the ROI centered to the target regardless of the drone flight path/orientation.

• Real time transmission of the whole video sphere and real time ROI selectively
• Record and replay the whole video sphere and selectively navigate to the Region of Interest during playback without limitations. During playback of the recorded footage, the viewer can select different “Point-camera-here” targets! Go back in time and space and select different Regions of Interest (ROI) to center the camera Field of View (FOV) regardless of if they were tracked during the real time video streaming. Nothing stays hidden anymore.  For example, you can navigate during the playback of a recorded file to a flown waypoint and select to see what was happening at the 6 o’clock of the drone while in real time you were observing the 12 o’clock region!.
• Records the spherical video along with the whole flight track as a georegistered footage
• The camera can be placed underneath the platform (i.e. aerial drone) or on top of the platform (i.e. Rover, Boat). Its 240 Vertical FOV ensures that it covers not only the region above or below the drone as 180 FOV cameras do but a big part of the horizon (60 degrees) is also covered for navigation purposes.
• Its miniaturized size and especially its low weight (62 gr) will allow for better flight times (more endurance). A typical GoPro with a 3-Axis gimbal weights aprox. 300 gr.

Multirotors, fixed wing, Hybrid, Rovers, boats, Robotics

• An analog camera version (with reduced resolution though)
• An even wider FOV camera (360×280 degrees FOV). It would be heavier and bigger though than the current spherical camera. This version will be optimized for ground moving platforms.
• A vehicular mounted unit (VMU) that streams the georegistered spherical video over cellular networks (it has been tested and works surprisingly well)

• Spherical video broadcast: Multiple viewers (up to 10) will be able to monitor different ROIs in real time or during playback! This feature will allow multiple viewers monitoring different perspectives by using their VR headsets or laptops.
• Fly-by-vision: An upcoming VR-headset mode that will allow flying the drone and control its main functions (i.e. “Fly-here”, “Point-camera-here”, take-off, land, RTL) by only using a VR-headset like Google cardboard or Gear-VR.