Yes, the estimated depth error from the least squares estimation in fact has quite a good correlation with the general form of the estimated depth function.
This is why historically the error or depth is used as a rejection criterion.
Clustering and other discrimination criterion can help reduce the noise in the results in a more compelling way.