"Bytes per sample mismatch" with ArtCNN when using fp16=True with the MIGraphX backend

[TheFeelTrain]

fp16=False with the exact same script/input works fine. 7900XTX on Arch Linux.

Traceback (most recent call last):
  File "src/cython/vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate
  File "src/cython/vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate
  File "/home/thefeeltrain/Encoding/test.py", line 72, in <module>
    scaled = ArtCNN.R8F64(backend=vsmlrt.BackendV2.MIGX(fp16=True)).scale(src, src.width * 2, src.height * 2)
  File "/usr/lib/python3.13/site-packages/vskernels/abstract/base.py", line 106, in _wrapped
    returned = method(self, *args, **init_kwargs | kwargs)
  File "/usr/lib/python3.13/site-packages/vsscale/onnx.py", line 200, in scale
    scaled = self.inference(wclip, **inference_kwargs)
  File "/usr/lib/python3.13/site-packages/vsscale/onnx.py", line 346, in inference
    return mlrt_ArtCNN(clip, self.tiles, self.tilesize, self.overlap, ArtCNNModel(self._model), self.backend)
  File "/home/thefeeltrain/.local/lib/python3.13/site-packages/vsmlrt.py", line 1818, in ArtCNN
    clip = inference_with_fallback(
        clips=[clip], network_path=network_path,
        overlap=(overlap_w, overlap_h), tilesize=(tile_w, tile_h),
        backend=backend
    )
  File "/home/thefeeltrain/.local/lib/python3.13/site-packages/vsmlrt.py", line 2764, in inference_with_fallback
    raise e
  File "/home/thefeeltrain/.local/lib/python3.13/site-packages/vsmlrt.py", line 2741, in inference_with_fallback
    ret = _inference(
        clips=clips, network_path=network_path,
    ...<4 lines>...
        batch_size=batch_size
    )
  File "/home/thefeeltrain/.local/lib/python3.13/site-packages/vsmlrt.py", line 2678, in _inference
    ret = core.migx.Model(
        clips, mxr_path,
    ...<2 lines>...
        **kwargs
    )
  File "src/cython/vapoursynth.pyx", line 3123, in vapoursynth.Function.__call__
vapoursynth.Error: operator(): bytes per sample mismatch

The FP16 version of the model works when using onnxruntime, so it's not a problem with the model.
For example this simple script runs without issue:

import onnxruntime as ort
import numpy as np
import cv2
import time
from pathlib import Path

# Load model and start session
model = Path("./ArtCNN_R8F64_FP16.onnx")
session = ort.InferenceSession(model, providers=["MIGraphXExecutionProvider"])

# Get model input details
input_name = session.get_inputs()[0].name
input_shape = session.get_inputs()[0].shape
input_type = session.get_inputs()[0].type

# Load input
input = cv2.imread("./test.png", cv2.IMREAD_GRAYSCALE)
input = np.clip(input.astype(np.float32) / 255.0, 0.0, 1.0)
input = np.expand_dims(input, axis=0) # Add channel dimension
input = np.expand_dims(input, axis=0) # Add batch dimension

# Run inference 100 times
pred = session.run(None, {input_name: input}) # Run once just to make sure we're not measuring startup time
start_time = time.time()
for _ in range(100):
    pred = session.run(None, {input_name: input})
end_time = time.time()
print(f"100 forward steps took {end_time - start_time} seconds!")

# Save output
pred = np.squeeze(np.array(pred))
cv2.imwrite(f"./{Path(model).stem}.png", np.around(pred * 255.0).astype(np.uint8))

[WolframRhodium]

The MIGX backend requires fp32 input/output, while it is very likely that your vs.VideoNode is of fp16.

[TheFeelTrain]

It seems to accept any format as long as fp16=False as far as I can tell. And the opposite is true with fp=True where it accepts nothing including depth(clip, 32). So I don't think the error is actually referring to the format of the input clip.

Unless is fp16=True forcing the input clip format to fp16 internally somewhere?

[WolframRhodium]

Unless is fp16=True forcing the input clip format to fp16 internally somewhere?

https://github.com/Jaded-Encoding-Thaumaturgy/vs-jetpack/blob/fd54b849bf29a5251ae766d5322e1d25c7eeb751/vsscale/onnx.py#L283-L291

[TheFeelTrain]

Alright if this is expected behavior I guess all is good. Right now jetpack defaults to fp16 with the MIGX backend so I'll get them to change it, they already did for NCNN.

Although, would it be possible to catch the input format before it actually runs? Right now you have to wait for MIGX to compile the ONNX model which even on my 7950X can take up to 45 minutes for ArtCNN, and on something like a 5600X it's double that. That's a long time to wait before it spits out an error.

If not I can also ask for it on the jetpack side.

[WolframRhodium]

Thanks for the suggestion. This could be an enhancement.