Stereo cameras pass calibration but have offset -- effecting DFS? (2024)

I've been able to calibrate my stereo cameras properly (reprojection errors of 0.25-0.5, in the green), but I always notice an offset between the left and right cameras that would indicate that they are not horizontally aligned.

While digging around the forums, I came across this old thread where someone was having trouble with calibration and @Dobry-Kolacz advised to check to see if the tops of each camera's views align. I tried it for myself and discovered that there was a significant gap:

We have a bunch of M0015 stereo cameras in inventory. I've checked several of them and they all have similar offsets. Same thing occurs when I try to align with the bottom of the camera views.

I've been testing out depth-from-stereo for VOA and have been noticing way more false positives than I'd like. I've been playing around with filters and thresholds but it feels like a band-aid rather than an actual solution (in order to eliminate false positives, I have to restrict things so much that it starts ignoring true positives).

I wonder if this offset is hurting the quality of my DFS, and if there is any way to fix it? According to the old thread, it seems that it may be due to an issue with the internal construction of the image sensor being offset to its lens. It's concerning to me that this difference is observed on all the pairs that I've checked so far, despite my efforts to mount them perfectly each time. Should I start trying to mix and match individual sensors to make new pairs that match better?

@MattK ,

Interesting, it does appear that the images are shifted up down.

I am wondering what the principal point values look like for this example? Additionally, what is the output of the extrinsic calibration suggesting the vertical shift is?

If the sensor is offset with respect to the lens, this should be completely captured by the principal points of the intrinsic calibration (and should not affect extrinsic calibration (shift between two cameras)). I imagine that the vertical shift of the camera relative position would have to be quite significant in order to result in such a change in vertical view. So, i suspect the actual difference is either in principal points (shift of lens relative to sensor) or a rotation of lens relative to the sensor (which is not modeled by the standard camera calibration as far as i know, but maybe i am not correct).

I am suspecting that when you are running a stereo calibration, which optimizes intrinsics and extrinsics at the same time (I believe), the optimization may get into a local minimum where it thinks that cameras are physically offset vertically as opposed to lenses offset w.r.t sensor. In which case, something to try would be to calibrate intrinsics separately (using monocular camera calibration), lock them down and just calibrate extrinsics using the stereo calibration procedure. I believe opencv allows you to lock in the initial intrinsics values and not change them for stereo optimization. It would be interesting to compare the principal points from monocular calibration vs stereo calibration (unconstrained).

However, this does not change the fact that the cameras are not perfectly aligned. As far as affecting DFS, i believe this should not be an issue, since DFS will undistort and align the images first based on the calibration parameters, so that the pixel disparity can be calculated between points on the same line (I believe that is how the DFS algorithms work in general). If the cameras have different top/bottom views, then it will limit the combined field of view of the DFS output, as you can imagine.

I will double check about what i mentioned regarding cameras being vertically misaligned (or shifted principal points causing different vertical FOVs) - whether that can affect DFS performance in a case when calibration is actually good and valid.

Thanks @Alex-Kushleyev ,

Here's the summaries from the successful intrinsic/extrinsic calibrations that I've been using:

I tried doing the monocular calibration process and it worked for the intrinsic calibrations, but when I tried to then do the normal extrinsic calibration it no longer worked.

Monocular intrinsic calibration successes:

Following stereo extrinsic calibration failures:

Is this the process you had in mind? The re-projection error and distance between cameras calculations are so off that I figured something must be inherently wrong. At first I thought that you wanted me to try and calibrate each stereo camera individually (stereo_front_L, stereo_front_R, stereo_rear_L, stereo_rear_R), but there didn't seem to be an option for that.

Might be worth noting that my stereo_front cameras are still incorrectly rotated in config, which might have something to do with why it was mistakenly identified as a vertical pair (you gave me the solution to this one here, I just haven't gotten around to fixing it yet). Though I wasn't getting that problem before, so it still seems to me that I'm just not doing what you envisioned.

And yes, please let me know if you find anything out regarding if the field of view offset could be effecting the quality of DFS.

@MattK ,

Oh, i guess the process already calibrates intrinsics first separately for left and right, and then the stereo (extrinsics). That is what I meant and it already implemented.

However, take a look at the intrinsics for the Rear Right, the principal points ar around 347, 210. The expected principal points are typically in the center of the camera image, which would be 640/2, 480/2 = 320, 240. So for that camera, it does look like the lens is shifted with respect to the camera quite a bit, almost 30 pixels in each X and Y direction. That would potentially explain the vertical shift that you are seeing.

The Translation between cameras is roughly -0.085, 0, 0 which i am assuming is consistent with the baseline of 8.5 cm between the cameras. Is that not true?

If the calibration fails, then perhaps you just got not very good samples and you can try again. Are the successful and failed extrinsic calibrations done on the same cameras, just different attempts?

Also, i checked with the team, the shift in the image due to principal points is not an issue as long as it is calibrated correctly, which it seems like to be captured by the large offset in your principal points.

So i suspect that the DFS issues you are seeing may be due to something else. Are you able to compare this "offset" configuration to another pair that is not that much offset and see if performance is similar or one is much better than the other?

Alex

@MattK ,

We recently found that in releases starting 1.1.3, the sync line that is connected between left and right stereo cameras has been affected by a GPIO pin on VOXL2. The sync line between the cameras is also connected to a pin in VOXL2 and although that pin should have been floating, it was selected as output.

This can explain the poor performance of your DFS, since cameras could be out of sync.

You can fix this by doing this (after each boot). Note the GPIO number has to be added 1100 when using the gpio export etc in the sys fs.:

#set GPIO 113 to input (J6 camera sync)echo 1213 > /sys/class/gpio/exportecho in > /sys/class/gpio/gpio1213/direction#set GPIO 111 to input (J8 camera sync)echo 1211 > /sys/class/gpio/exportecho in > /sys/class/gpio/gpio1211/direction

I have confirmed the lack of sync if you boot up normally and the sync is restored if you run this command before starting the camera server (should be ok if camera server is already running too).

can you please try using the above fix and check if your DFS performance improves?

We will be implementing the fix in the latest release ASAP.

Alex

@Alex-Kushleyev

Oh, very fascinating. Thank you for the update!

I'd actually somewhat given up on DFS and started testing your new ToF beta product, but when I can I'll try boot my DFS rig back up and see if the fix helps.

Edit: Nevermind, I just checked and my DFS rig is on SDK v1.1.2, so this fix shouldn't be applicable. I might still try it out just in case, though.