Issue with INT4 quantization

[NICESTM]

Hello,

I am trying to quantize in INT4 all the weights of a mobilenet.
My attempt crashes at this line:
quark/onnx/quantization/quant_utils.py", line 254, in tensor_type
raise ValueError(f"Unexpected value qtype={self!r}.")
ValueError: Unexpected value qtype=<ExtendedQuantType.QInt4: 5>.

For doing this, I needed to define my own class Int4Spec(QTensorConfig) and also my own type INT4
class Int4(BaseInt4):
onnx_proto_dtype: TensorProto.INT4
map_onnx_format = ExtendedQuantType.QInt4

Then I used
config = QConfig(global_config=QLayerConfig(activation=Int8Spec(), weight=Int4Spec())

and then I have something like this:
qk_quantizer = ModelQuantizer(config)
dr = ImageDataReader(quantization_samples=data, model_path=model_path)
print(f"[INFO] : Running ONNX quantization on {model_path}")
qk_quantizer.quantize_model(model_path, qk_quantized_model_path, dr)

Could you help?
BR

[ChaoLi-AMD]

Hi @NICESTM, thanks for the feedback. We will take a closer look. Could you share why INT4 is needed for MobileNet? For this type of model, INT4 usually causes a large accuracy drop.

For now, we recommend using INT8 quantization for MobileNet in the ONNX quantization flow.

[NICESTM]

Hi, you're right it may not be possible to consider INT4 for MobileNet topologies without severe accuracy degradation. However, it may be possible for other topologies. Therefore, I tend to think it would be nice to have INT4 quantization as well.

[ChaoLi-AMD]

Quark's ONNX workflow supports INT4 quantization for MatMulNBits, which can be used to quantize MatMul operations in some LLM models. You can refer to: https://github.com/amd/Quark/blob/release/0.11/examples/onnx/weights_only_quantization/int4_matmul_nbits/llama2/example_quark_onnx_weights_only_quant_int4_matmul_nbits_llama2.rst

For CNN-based models, INT4 quantization has not yet been well validated in terms of either hardware support or accuracy. It is not currently supported in the Quark ONNX quantization workflow.

If you have a specific use case or requirement, we'd be happy to discuss it and evaluate whether we can add support.

[fxmarty-amd]

Consider trying https://github.com/Xilinx/brevitas as well for INT4 CNN quantization