undefined symbol: _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_ib

[ZMY-0309]

I am using the following environment to perform SFT on the Qwen3.5 MoE model:

transformers=5.2.0
transformer-engine=2.12.0
flash-attn=2.8.3

The following error occurred when transformer-engine imported flash-attn:

Online research suggests this might be due to version incompatibility between transformer-engine and flash-attn. Could you please advise on how to adjust this, or could there be other causes?

[ptrendx]

This error suggests the incompatibility between the flash attention package and the PyTorch version you are using (the missing symbol is from PyTorch). Could you double check by just calling this failing import line in python without calling Transformer Engine at all? What is the PyTorch version that you use and how did you install Flash Attention? If you installed FA before and the upgraded PyTorch version you would need to reinstall FA most probably - unfortunately PyTorch does not keep ABI compatibility between versions (they are working on it but it is not really ready yet).

The other workaround would be to just uninstall FA and use the cuDNN backend for attention in TE.

[ZMY-0309]

Thank you for your reply. I checked and confirmed that the issue was indeed caused by a version mismatch between PyTorch and flash-attn. After adjusting the PyTorch version, the error no longer occurs.

[dlnn123]

Thank you for your reply. I checked and confirmed that the issue was indeed caused by a version mismatch between PyTorch and flash-attn. After adjusting the PyTorch version, the error no longer occurs.

hello! I also encountered the same problem. Could you tell me your versions of torch and flash-attn?