Incorrect parameter for imagingGL tests

Community Build and Deployment
1

Hello!

I was looking into some of the imagingGL tests and ran with a couple of tests that seem to have a faulty idiff parameters.

For example, the test for testUsdImagingGLPoints is passing 1 as FAIL argument, which means that the pixel has to be 100% different

pxr_register_test(testUsdImagingGLPoints
    COMMAND "${CMAKE_INSTALL_PREFIX}/tests/testUsdImagingGLBasicDrawing -offscreen -frameAll -stage points.usda -widgetSize 640 480 -write testUsdImagingGLPoints.png"
    IMAGE_DIFF_COMPARE
        testUsdImagingGLPoints.png
    FAIL 0.001
    FAIL_PERCENT 0.01
    PERCEPTUAL
    EXPECTED_RETURN_CODE 0
    TESTENV testUsdImagingGLPoints
)

This parameter virtually disables the test. My question is whether this is intentional and is an obscure way to disable this test or it wasn’t detected before and should be corrected to something more sensible as what is set in some of the other tests, for example testUsdImagingGLDomeLight_invis, which uses FAIL 0.01

2

Hey - this confused me at first too. You’ll notice that this test also has the “PERCEPTUAL” flag set, though, which adds an additional failure condition - it does a special “perceptual diff” test on each pixel, and will fail if more than FAIL_PERCENT pixels fail that test.
Because it’s an additional test, in general it can still fail if FAIL_PERCENT pixels are greater than FAIL absolute difference. By setting to FAIL to 1, you disable the absolute test, and rely solely on the perceptual test.

3

Their image differ either directly uses OpenImageIO’s idiff, or is based off it (I don’t remember which)… you can read docs about idiff here:

https://openimageio.readthedocs.io/en/latest/idiff.html

4

Thanks for sharing but, I still think it is wrong. I ran into this because I’m working on the HgiWebGPU backend and I noticed this test passing while I know that gl_pointSize is not supported.

In the case of my result, it only displayed the clear color and was marked as passing. I tried the test using metal(although disabled) and same result. The test was passing while the images where rather different:

image
image3386×594 71.1 KB

The left image is the baseline and the right is the new result.

Not sure about the perceptual difference calculation, if it takes into account the FAIL value or whether we are just being unlucky with the selection of colors.

And I assume we can also discard version differences in the idiff tool, because I’m using the one installed by the build_usd.py

5

But you are right with the perceptual difference that doesn’t use the fail value, just checked the implementation: https://github.com/AcademySoftwareFoundation/OpenImageIO/blob/05c951385b26b8528366d79593257c9e2ceb8877/src/idiff/idiff.cpp#L285

Now I’m more confused by the results but this is what I tried:

($) idiff -fail 0.75 -failpercent 1 -p baseline/testUsdImagingGLPoints_zoom_pan.png  tmp3_olf446/testUsdImagingGLPoints_zoom_pan.png
Comparing "baseline/testUsdImagingGLPoints_zoom_pan.png" and "tmp3_olf446/testUsdImagingGLPoints_zoom_pan.png"
  Mean error = 0.00363772
  RMS error = 0.0478421
  Peak SNR = 26.4038
  Max error  = 1 @ (155, 207, R)  values are 1, 0.5019608, 0.098039225, 1 vs 0, 1, 0, 1
  4888 pixels (1.59%) over 1e-06
  1755 pixels (0.571%) over 0.75
  4838 pixels (1.57487%) failed the perceptual test
FAILURE
($) idiff -fail 1 -failpercent 1 -p baseline/testUsdImagingGLPoints_zoom_pan.png  tmp3_olf446/testUsdImagingGLPoints_zoom_pan.png
Comparing "baseline/testUsdImagingGLPoints_zoom_pan.png" and "tmp3_olf446/testUsdImagingGLPoints_zoom_pan.png"
PASS

Why does it fail the perceptual test in the first case? I guess it would basically just ignore those parameters. Either way, I think empirically, makes the test always a false positive