diff --git a/docs/flagrelease_en/model_list.txt b/docs/flagrelease_en/model_list.txt index 24e5a82ad..5581e92c1 100644 --- a/docs/flagrelease_en/model_list.txt +++ b/docs/flagrelease_en/model_list.txt @@ -235,6 +235,15 @@ FlagRelease/Qwen3.8-27B-BF16-tsingmicro-FlagOS FlagRelease/Qwen3.8-27B-BF16-zhenwu-FlagOS-Express FlagRelease/Qwen3.8-27B-FP8-mthreads-FlagOS FlagRelease/Qwen3.8-27B-W4A8-arm-FlagOS-Express +FlagRelease/Qwen3.8-Flash-Next-BF16-ascend-FlagOS +FlagRelease/Qwen3.8-Flash-Next-BF16-hygon-FlagOS +FlagRelease/Qwen3.8-Flash-Next-BF16-iluvatar-FlagOS +FlagRelease/Qwen3.8-Flash-Next-BF16-kunlunxin-FlagOS +FlagRelease/Qwen3.8-Flash-Next-BF16-metax-FlagOS +FlagRelease/Qwen3.8-Flash-Next-BF16-mthreads-FlagOS +FlagRelease/Qwen3.8-Flash-Next-BF16-nvidia-FlagOS +FlagRelease/Qwen3.8-Flash-Next-BF16-tsingmicro-FlagOS +FlagRelease/Qwen3.8-Flash-Next-BF16-zhenwu-FlagOS FlagRelease/RoboBrain-X0 FlagRelease/RoboBrain-X0-Preview-FlagOS FlagRelease/RoboBrain-X0-Preview-ascend-FlagOS @@ -255,8 +264,6 @@ FlagRelease/Seed-OSS-36B-Instruct-metax-FlagOS FlagRelease/Seed-OSS-36B-Instruct-mthreads-FlagOS FlagRelease/Seed-OSS-36B-Instruct-nvidia-FlagOS FlagRelease/TeleChat3-36B-Thinking-mthreads-FlagOS -FlagRelease/ZCK-Qwen3-8B-metax-FlagOS -FlagRelease/ZCK-Qwen3-8B-nvidia-FlagOS FlagRelease/deepseek-r1-1.5b-nvidia-FlagOS FlagRelease/farm_molecular_representation-hygon-FlagOS FlagRelease/farm_molecular_representation-nvidia-FlagOS diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-ascend-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-ascend-FlagOS.md new file mode 100644 index 000000000..f5ef53b42 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-ascend-FlagOS.md @@ -0,0 +1,142 @@ +--- +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +On August 26, Alibaba open-sourced its multimodal MoE model Qwen3.8-Flash-Next. The FlagOS community completed Day-0 synchronized adaptation across multiple chips, having accomplished multi-chip adaptation, precision alignment, and deployment validation based on the FlagOS unified open-source technology stack on 8 AI chips — including T-Head (Pingtouge), NVIDIA, Moore Threads, Ascend, MetaX, Kunlunxin, Hygon, and Iluvatar CoreX. The first batch uniformly provides a BF16-precision version. The multi-chip versions have been open-sourced on the ModelScope and Hugging Face platforms, where developers can directly obtain out-of-the-box solutions for their respective chips. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Ascend** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + +# Evaluation Results +## Benchmark Result +| Metrics | Qwen3.8-Flash-Next-Nvidia-Origin | Qwen3.8-Flash-Next-Ascend-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 92.9 | Evaluating | +| MuSR| 78.57 | Evaluating | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 20.10.8 | +| Operating System | openEuler 22.03 (LTS-SP4) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-next-ascend001-gems5.3.0-tree0.6.0-cx0.13.0-pluginnone-vllmnone-sglang0.5.11-sglangfl0.1.0-cp311-ptnpu28-cann85-a64-25.5.0:202608260936 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Qwen3.8-Flash-Next-BF16-ascend-FlagOS --local_dir /data/Qwen3.8-Flash-Next +``` + +### Start the Container +```bash +docker run -dit \ + --name flagos \ + --privileged \ + --init \ + --network=host --ipc=host --shm-size=512g \ + --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ + --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ + --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \ + --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \ + --device=/dev/davinci_manager \ + --device=/dev/hisi_hdc \ + --volume /usr/local/sbin:/usr/local/sbin \ + --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ + --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ + --volume /etc/ascend_install.info:/etc/ascend_install.info \ + --volume /var/queue_schedule:/var/queue_schedule \ + -v /etc/localtime:/etc/localtime:ro \ + -v /etc/timezone:/etc/timezone:ro \ + -v /data:/data \ + -v /public-flash/models:/models \ + --entrypoint=bash \ + harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-next-ascend001-gems5.3.0-tree0.6.0-cx0.13.0-pluginnone-vllmnone-sglang0.5.11-sglangfl0.1.0-cp311-ptnpu28-cann85-a64-25.5.0:202608260936 \ + -c "tail -f /dev/null" +docker exec -it flagos /bin/bash +``` +### Start the Server +```bash +bash /opt/qwen38-adaptation/current/launch-graph-bs1.sh +``` + +## Service Invocation +### Invocation Script +```bash +curl -X POST http://127.0.0.1:30100/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen3.8-Flash-Next", + "messages": [ + {"role": "user", "content": "中国的首都是哪里？"} + ], + "temperature": 0.7, + "max_tokens": 500 + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., "Explain the basics of quantum computing") +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a "develop once, run anywhere" workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of . This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Qwen/Qwen3.8-Flash-Next and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-hygon-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-hygon-FlagOS.md new file mode 100644 index 000000000..6d2764298 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-hygon-FlagOS.md @@ -0,0 +1,180 @@ +--- +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +On August 26, Alibaba open-sourced its multimodal MoE model Qwen3.8-Flash-Next. The FlagOS community completed Day-0 synchronized adaptation across multiple chips, having accomplished multi-chip adaptation, precision alignment, and deployment validation based on the FlagOS unified open-source technology stack on 8 AI chips — including T-Head (Pingtouge), NVIDIA, Moore Threads, Ascend, MetaX, Kunlunxin, Hygon, and Iluvatar CoreX. The first batch uniformly provides a BF16-precision version. The multi-chip versions have been open-sourced on the ModelScope and Hugging Face platforms, where developers can directly obtain out-of-the-box solutions for their respective chips. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Hygon** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Qwen3.8-Flash-Next-Nvidia-Origin | Qwen3.8-Flash-Next-Hygon-FlagOS | +|--------------|----------------------------------|---------------------------------| +| GPQA_Diamond | 92.9 | Evaluating | +| MuSR | 78.57 | Evaluating | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 20.10.5, build 55c4c88 | +| Operating System | Ubuntu 22.04.4 LTS (Jammy Jellyfish) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-next-hygon001-gems5.4.0-tree0.6.1-cxnone-plugin0.2.0-vllm0.26.1-cp310-pt210-dtk2604-x64-6.3.30-v1.4.1a:202608262000 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Qwen3.8-Flash-Next-BF16-hygon-FlagOS --local_dir /data/Qwen3.8-Flash-Next +``` + +### Start the Container +```bash +docker run -d --name flagos \ + --network host --ipc host \ + --device /dev/kfd --device /dev/mkfd --device /dev/dri \ + --mount type=bind,src=/dev/infiniband,dst=/dev/infiniband \ + --mount type=bind,src=/sys/class/infiniband,dst=/sys/class/infiniband,readonly \ + --mount type=bind,src=/sys/class/infiniband_verbs,dst=/sys/class/infiniband_verbs,readonly \ + --mount type=bind,src=/sys/class/net,dst=/sys/class/net,readonly \ + --mount type=bind,src=/usr/etc/libibverbs.d,dst=/usr/etc/libibverbs.d,readonly \ + --mount type=bind,src=/lib/x86_64-linux-gnu/libibverbs.so.1.14.44.0,dst=/lib/x86_64-linux-gnu/libibverbs.so.1.14.47.0,readonly \ + --mount type=bind,src=/lib/x86_64-linux-gnu/libshca-rdmav34.so,dst=/lib/x86_64-linux-gnu/libshca-rdmav34.so,readonly \ + --mount type=bind,src=/lib/x86_64-linux-gnu/libnl-3.so.200,dst=/lib/x86_64-linux-gnu/libnl-3.so.200,readonly \ + --mount type=bind,src=/lib/x86_64-linux-gnu/libnl-route-3.so.200,dst=/lib/x86_64-linux-gnu/libnl-route-3.so.200,readonly \ + -v /opt/hyhal:/opt/hyhal \ + -v /data:/data:rslave \ + -v /public-flash:/public-flash \ + --group-add video \ + --cap-add SYS_PTRACE \ + --security-opt seccomp=unconfined \ + --security-opt label=disable \ + -e HSA_FORCE_FINE_GRAIN_PCIE=1 \ + -e NCCL_IB_DISABLE=0 \ + harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-next-hygon001-gems5.4.0-tree0.6.1-cxnone-plugin0.2.0-vllm0.26.1-cp310-pt210-dtk2604-x64-6.3.30-v1.4.1a:202608262000 \ + bash -lc 'sleep infinity' +docker exec -it flagos /bin/bash +``` +### Start the Server +```bash +set +o nounset +source /opt/dtk-26.04-DCC2602-0317/env.sh +set -o nounset +export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 +export HSA_FORCE_FINE_GRAIN_PCIE=1 +export TRITON_HIP_CLANG_PATH=/opt/dtk-26.04-DCC2602-0317/aillvm/bin/clang-18 +export LD_LIBRARY_PATH="/opt/ucx-shca/lib:/opt/rccl-shca-net/lib:${LD_LIBRARY_PATH:-}" +export UCX_MODULE_DIR=/opt/ucx-shca/lib/ucx +export MASTER_ADDR=10.232.2.19 +export GLOO_SOCKET_IFNAME=ib0 +export NCCL_SOCKET_IFNAME=ib0 +export NCCL_NET_PLUGIN=shca +export NCCL_IB_DISABLE=0 +export NCCL_IB_HCA=shca_0,shca_1,shca_2,shca_3 +export HF_HUB_OFFLINE=1 +export TRANSFORMERS_OFFLINE=1 +export HF_DATASETS_OFFLINE=1 +# vLLM 0.24+ RPC timeout for execute_model calls (seconds). +# W8A8 quantized models need extra time on first inference for kernel compilation. +export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 +vllm serve "$MODEL_PATH" \ + --served-model-name "$SERVED_MODEL_NAME" \ + --trust-remote-code \ + --distributed-executor-backend mp \ + --nnodes 1 \ + --node-rank "$NODE_RANK" \ + --master-addr "$MASTER_ADDR" \ + --master-port "$MASTER_PORT" \ + --distributed-timeout-seconds 1800 \ + --tensor-parallel-size "$TP" \ + --pipeline-parallel-size "$PP" \ + --data-parallel-size "$DP" \ + --load-format "$LOAD_FORMAT" \ + --mm-encoder-tp-mode data \ + --disable-custom-all-reduce \ + --moe-backend triton \ + --max-model-len "${MAX_MODEL_LEN:-32768}" \ + --reasoning-parser qwen3 \ + --max-num-seqs 16 \ + --gpu-memory-utilization 0.95 \ + --language-model-only \ + --no-enable-log-requests +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:$MASTER_PORT/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "$SERVED_MODEL_NAME", + "messages": [{"role": "user", "content": "你好"}] + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Qwen/Qwen3.8-Flash-Next and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-iluvatar-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-iluvatar-FlagOS.md new file mode 100644 index 000000000..6cbf44d28 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-iluvatar-FlagOS.md @@ -0,0 +1,157 @@ +--- +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +On August 26, Alibaba open-sourced its multimodal MoE model Qwen3.8-Flash-Next. The FlagOS community completed Day-0 synchronized adaptation across multiple chips, having accomplished multi-chip adaptation, precision alignment, and deployment validation based on the FlagOS unified open-source technology stack on 8 AI chips — including T-Head (Pingtouge), NVIDIA, Moore Threads, Ascend, MetaX, Kunlunxin, Hygon, and Iluvatar CoreX. The first batch uniformly provides a BF16-precision version. The multi-chip versions have been open-sourced on the ModelScope and Hugging Face platforms, where developers can directly obtain out-of-the-box solutions for their respective chips. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Iluvatar** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + +# Evaluation Results +## Benchmark Result +| Metrics | Qwen3.8-Flash-Next-Nvidia-Origin | Qwen3.8-Flash-Next-Iluvatar-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 92.9 | Evaluating | +| MuSR| 78.57 | Evaluating | + + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 20.10.12 | +| Operating System | 22.04 LTS | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/flagrelease-qwen3.8-flash-next-iluvatar-tree_none-gems_5.0.0-vllm_0.20.0-plugin_main-cx_none-python_3.12.11-torch_2.10.0_corex.4.5.0-pcp_ixml4.4.0-gpu_iluvatar001-arc_amd64-driver_4.5.0:202608261637 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Qwen3.8-Flash-Next-BF16-iluvatar-FlagOS --local_dir /data/Qwen3.8-Flash-Next +``` + +### Start the Container +```bash +IMAGE="harbor.baai.ac.cn/flagrelease-public/flagrelease-qwen3.8-flash-next-iluvatar-tree_none-gems_5.0.0-vllm_0.20.0-plugin_main-cx_none-python_3.12.11-torch_2.10.0_corex.4.5.0-pcp_ixml4.4.0-gpu_iluvatar001-arc_amd64-driver_4.5.0:202608261637" +# 对应plugin-FL 分支 ilvita-int8-main +docker run -d --network host \ + --privileged --ipc=host \ + -v /dev:/dev -v /lib/modules:/lib/modules \ + -v /sys:/sys -v /data:/data -v /mnt:/mnt -v /mnt/share/models/:/models \ + -w /workspace \ + --name flagos -it ${IMAGE} +docker exec -it flagos bash +cd /models/running_common +``` +### Start the Server +```bash +export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 +export VLLM_PLUGINS=fl +export VLLM_WORKER_MULTIPROC_METHOD=spawn + +# Timeout — Iluvatar Triton compilation takes a long time +export VLLM_ENGINE_ITERATION_TIMEOUT_S=72000 +export VLLM_RPC_TIMEOUT=72000000 +export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200 + +# === Modify the following parameters according to actual conditions. === +MODEL_PATH="/data/Qwen3.8-Flash-Next" +MODEL_NAME="qwen38_flash" +PORT=8030 +TP=8 +MAX_MODEL_LEN=32768 +export CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES && \ +export VLLM_PLUGINS=$VLLM_PLUGINS && \ +export VLLM_WORKER_MULTIPROC_METHOD=$VLLM_WORKER_MULTIPROC_METHOD && \ +export VLLM_ENGINE_ITERATION_TIMEOUT_S=$VLLM_ENGINE_ITERATION_TIMEOUT_S && \ +export VLLM_RPC_TIMEOUT=$VLLM_RPC_TIMEOUT && \ +export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=$VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS && \ +vllm serve $MODEL_PATH \ + --served-model-name $MODEL_NAME \ + --dtype bfloat16 \ + --tensor-parallel-size $TP \ + --pipeline-parallel-size 2 \ + --max-model-len $MAX_MODEL_LEN \ + --gpu-memory-utilization 0.95 \ + --port $PORT \ + --enforce-eager \ + --trust-remote-code 2>&1 | tee -a $LOG_FILE +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8030/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen38_flash", + "messages": [{"role": "user", "content": "中国的首都是哪里？"}], + "temperature": 0.7, + "max_tokens": 1024 + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., "Explain the basics of quantum computing") +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a "develop once, run anywhere" workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of . This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Qwen/Qwen3.8-Flash-Next and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-kunlunxin-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-kunlunxin-FlagOS.md new file mode 100644 index 000000000..6da41fd94 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-kunlunxin-FlagOS.md @@ -0,0 +1,166 @@ +--- +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +On August 26, Alibaba open-sourced its multimodal MoE model Qwen3.8-Flash-Next. The FlagOS community completed Day-0 synchronized adaptation across multiple chips, having accomplished multi-chip adaptation, precision alignment, and deployment validation based on the FlagOS unified open-source technology stack on 8 AI chips — including T-Head (Pingtouge), NVIDIA, Moore Threads, Ascend, MetaX, Kunlunxin, Hygon, and Iluvatar CoreX. The first batch uniformly provides a BF16-precision version. The multi-chip versions have been open-sourced on the ModelScope and Hugging Face platforms, where developers can directly obtain out-of-the-box solutions for their respective chips. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Kunlunxin** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Qwen3.8-Flash-Next-Nvidia-Origin | Qwen3.8-Flash-Next-Kunlunxin-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 92.9 | Evaluating | +| MuSR | 78.57 | Evaluating | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 28.2.2, build e6534b4 | +| Operating System | 22.04.4 LTS (Jammy Jellyfish) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-next-kunlunxin001-gems5.0.0-treenone-cx0.13.0-plugin0.2.0-vllm0.20.2-cp310-pt29-xrt50-x64-5.0.21.47:202608262111 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Qwen3.8-Flash-Next-BF16-kunlunxin-FlagOS --local_dir /data/Qwen3.8-Flash-Next +``` + +### Start the Container +```bash +set -euo pipefail + +docker run -d \ + --name qwen38-flash-next-flagos-p800 \ + --privileged \ + --network host \ + --shm-size 128g \ + --cap-add SYS_PTRACE \ + --ulimit memlock=-1:-1 \ + --ulimit nofile=120000:120000 \ + --ulimit stack=67108864:67108864 \ + -v /data/Qwen3.8-Flash-Next:/models/Qwen3.8-Flash-Next:ro \ + --entrypoint /bin/bash \ + harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-next-kunlunxin001-gems5.0.0-treenone-cx0.13.0-plugin0.2.0-vllm0.20.2-cp310-pt29-xrt50-x64-5.0.21.47:202608262111 \ + -lc 'sleep infinity' +``` +### Start the Server +```bash +set -euo pipefail + +docker exec -d qwen38-flash-next-flagos-p800 /bin/bash -lc ' + export USE_FLAGGEMS=1 + export VLLM_DISABLE_PYNCCL=1 + export VLLM_FL_FLAGOS_WHITELIST=rms_norm + export VLLM_FL_FLAGGEMS_ATEN_PLAN_CACHE=1 + export VLLM_FL_FLAGGEMS_ATEN_PLAN_CACHE_REPORT=1 + export FLAGGEMS_ATEN_PLAN_CACHE=1 + export FLAGGEMS_ATEN_PLAN_CACHE_SIZE=128 + export QWEN4_QSA_FUSED_COMPRESS=1 + export VLLM_FL_QWEN4_STABLE_TP_ALL_REDUCE=1 + export VLLM_FL_COMMON_SLOT_MAPPING_GRAPH=0 + export VLLM_PLUGINS=fl + export VLLM_WORKER_MULTIPROC_METHOD=spawn + export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 + export PYTHONUNBUFFERED=1 + exec /root/miniconda/envs/python310_torch29_cuda/bin/vllm serve /models/Qwen3.8-Flash-Next \ + --served-model-name Qwen3.8-Flash-Next \ + --host 0.0.0.0 \ + --port 8000 \ + --tensor-parallel-size 8 \ + --distributed-executor-backend mp \ + --load-format safetensors \ + --dtype bfloat16 \ + --disable-custom-all-reduce \ + --max-model-len 102400 \ + --max-num-seqs 64 \ + --max-num-batched-tokens 2048 \ + --limit-mm-per-prompt '\''{"image":16}'\'' \ + --block-size 128 \ + --gpu-memory-utilization 0.85 \ + --enable-chunked-prefill \ + --no-enable-prefix-caching \ + --mm-encoder-tp-mode data \ + --mm-encoder-attn-backend TORCH_SDPA \ + --enforce-eager >>/var/log/qwen38-flash-next-vllm.log 2>&1 +' +``` + +## Service Invocation +### Invocation Script +```bash +set -euo pipefail + +curl --noproxy '*' -fsS http://127.0.0.1:8000/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"Qwen3.8-Flash-Next","messages":[{"role":"user","content":"Compute 17 * 19. Return only the integer."}],"temperature":0,"max_tokens":32}' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Qwen/Qwen3.8-Flash-Next and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-metax-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-metax-FlagOS.md new file mode 100644 index 000000000..9b4fd654d --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-metax-FlagOS.md @@ -0,0 +1,146 @@ +--- +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +On August 26, Alibaba open-sourced its multimodal MoE model Qwen3.8-Flash-Next. The FlagOS community completed Day-0 synchronized adaptation across multiple chips, having accomplished multi-chip adaptation, precision alignment, and deployment validation based on the FlagOS unified open-source technology stack on 8 AI chips — including T-Head (Pingtouge), NVIDIA, Moore Threads, Ascend, MetaX, Kunlunxin, Hygon, and Iluvatar CoreX. The first batch uniformly provides a BF16-precision version. The multi-chip versions have been open-sourced on the ModelScope and Hugging Face platforms, where developers can directly obtain out-of-the-box solutions for their respective chips. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Metax** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + +# Evaluation Results +## Benchmark Result +| Metrics | Qwen3.8-Flash-Next-Nvidia-Origin | Qwen3.8-Flash-Next-Metax-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 92.9 | 91.16 | +| MuSR| 78.57 | Evaluating | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 29.3.1 | +| Operating System | 22.04 LTS | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-next-metax001-gems5.4.0-treenone-cxnone-plugin3.0.0-vllm0.24.0-cp312-pt28-maca37-x64-3.8.1:202608261009 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Qwen3.8-Flash-Next-BF16-metax-FlagOS --local_dir /data/Qwen3.8-Flash-Next +``` + +### Start the Container +```bash +IMAGE="harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-next-metax001-gems5.4.0-treenone-cxnone-plugin3.0.0-vllm0.24.0-cp312-pt28-maca37-x64-3.8.1:202608261009" +docker run -d \ + --name flagos \ + --network host \ + --shm-size 64g \ + --device /dev/dri:/dev/dri:rwm \ + --device /dev/mxcd:/dev/mxcd:rwm \ + -v /data:/data \ + ${IMAGE} \ + sleep infinity + +docker exec -it flagos /bin/bash +``` +### Start the Server +```bash +export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 +export VLLM_WORKER_MULTIPROC_METHOD=spawn +export GEMS_VENDOR=metax +export VLLM_PLUGINS=fl +export VLLM_FL_FLAGOS_BLACKLIST="mm,mm_out,sort,stable_sort,masked_fill,masked_fill_,log_softmax,log_softmax_out,log_softmax_backward,log_softmax_backward_out,pad,constant_pad_nd,copy_,topk" +export VLLM_ENGINE_ITERATION_TIMEOUT_S=7200 +export VLLM_ENGINE_READY_TIMEOUT_S=1800 +export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200 +export VLLM_RINGBUFFER_WARNING_INTERVAL=300 + +vllm serve /data/Qwen3.8-Flash-Next/ \ + --dtype bfloat16 \ + --tensor-parallel-size 8 \ + --max-model-len 32768 \ + --gpu-memory-utilization 0.9 \ + --port 8000 \ + --enforce-eager \ + --trust-remote-code \ + --served-model-name qwen38_flash +``` + +## Service Invocation +### Invocation Script +```bash +curl http://localhost:8030/v1/chat/completions \ +-H "Content-Type: application/json" \ +-d '{ + "model": "qwen38_flash", + "messages": [{"role": "user", "content": "中国的首都是哪里？"}], + "temperature": 0.7, + "max_tokens": 1024 +}' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., "Explain the basics of quantum computing") +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a "develop once, run anywhere" workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of . This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Qwen/Qwen3.8-Flash-Next and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-mthreads-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-mthreads-FlagOS.md new file mode 100644 index 000000000..3ac40c896 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-mthreads-FlagOS.md @@ -0,0 +1,142 @@ +--- +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +On August 26, Alibaba open-sourced its multimodal MoE model Qwen3.8-Flash-Next. The FlagOS community completed Day-0 synchronized adaptation across multiple chips, having accomplished multi-chip adaptation, precision alignment, and deployment validation based on the FlagOS unified open-source technology stack on 8 AI chips — including T-Head (Pingtouge), NVIDIA, Moore Threads, Ascend, MetaX, Kunlunxin, Hygon, and Iluvatar CoreX. The first batch uniformly provides a BF16-precision version. The multi-chip versions have been open-sourced on the ModelScope and Hugging Face platforms, where developers can directly obtain out-of-the-box solutions for their respective chips. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Mthreads** container image supporting deployment within minutes + +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + +# Evaluation Results +## Benchmark Result + +| Metrics | Qwen3.8-Flash-Next-Nvidia-Origin | Qwen3.8-Flash-Next-Mthreads-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 92.9 | 91.66 | +| MuSR | 78.57 | Evaluating | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|-------------------------| +| Docker Version | Docker version 20.10.12 | +| Operating System | 22.04 LTS | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/sglang-qwen3.8-flash-next-mthreads001-gems5.0.2-treenone-cx0.13.0-pluginnone-vllmnone-cp310-pt29-musa43-x64-3.3.5-server:202608262100 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Qwen3.8-Flash-Next-BF16-mthreads-FlagOS --local_dir /data/Qwen3.8-Flash-Next +``` + +### Start the Container +```bash +docker run -dit \ + --name flagos \ + --privileged \ + --ipc host \ + --network host \ + --shm-size 512g \ + -w /workspace \ + -v /data/:/data/ \ + -v /etc/localtime:/etc/localtime:ro \ + -v /etc/timezone:/etc/timezone:ro \ + --env MTHREADS_VISIBLE_DEVICES=all \ + harbor.baai.ac.cn/flagrelease-public/sglang-qwen3.8-flash-next-mthreads001-gems5.0.2-treenone-cx0.13.0-pluginnone-vllmnone-cp310-pt29-musa43-x64-3.3.5-server:202608262100 \ + sleep infinity +docker exec -it flagos /bin/bash +``` + +### Start the Server +```bash +bash /workspace/official-main-align/start_worker32081_bs16_graph_experimental.sh +``` + +## Service Invocation +### Invocation Script +```bash +curl -X POST http://127.0.0.1:30015/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen3.8-Flash-Next", + "messages": [ + {"role": "user", "content": "中国的首都是哪里？"} + ], + "temperature": 0.7, + "max_tokens": 500 + }' +``` + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., "Explain the basics of quantum computing") +- Click the send button to get a response + +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a "develop once, run anywhere" workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of . This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. + +## FlagGems +FlagGems is a high-performance, generic operator library implemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutral kernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. + +## FlagTree +FlagTree is an open source, unified compiler for multiple AI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. For upstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. + +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to support the entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. + +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. + +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** +FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: +- **Multi-dimensional Evaluation**: Supports 800+ model evaluations across NLP, CV, Audio, and Multimodal fields, covering 20+ downstream tasks including language understanding and image-text generation. +- **Industry-Grade Use Cases**: Has completed horizontal evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support + +# License +The model weights are derived from Qwen/Qwen3.8-Flash-Next and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-nvidia-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-nvidia-FlagOS.md new file mode 100644 index 000000000..d5305d6d2 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-nvidia-FlagOS.md @@ -0,0 +1,134 @@ +--- +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +On August 26, Alibaba open-sourced its multimodal MoE model Qwen3.8-Flash-Next. The FlagOS community completed Day-0 synchronized adaptation across multiple chips, having accomplished multi-chip adaptation, precision alignment, and deployment validation based on the FlagOS unified open-source technology stack on 8 AI chips — including T-Head (Pingtouge), NVIDIA, Moore Threads, Ascend, MetaX, Kunlunxin, Hygon, and Iluvatar CoreX. The first batch uniformly provides a BF16-precision version. The multi-chip versions have been open-sourced on the ModelScope and Hugging Face platforms, where developers can directly obtain out-of-the-box solutions for their respective chips. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Nvidia** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Qwen3.8-Flash-Next-Nvidia-Origin | Qwen3.8-Flash-Next-Nvidia-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 92.9 | 91.3 | +| MuSR | 78.57 | Evaluating | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 24.0.0, build 98fdcd7 | +| Operating System | 22.04.4 LTS (Jammy Jellyfish) | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-next-nvidia003-gems5.3.3-tree0.5.0-cxnone-plugin0.3.0-vllm0.24.0-cp312-pt211-cu129-x64-580.126.20:202608262106 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Qwen3.8-Flash-Next-BF16-nvidia-FlagOS --local_dir /data/Qwen3.8-Flash-Next +``` + +### Start the Container +```bash +set -euo pipefail +IMAGE_URI='harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-next-nvidia003-gems5.3.3-tree0.5.0-cxnone-plugin0.3.0-vllm0.24.0-cp312-pt211-cu129-x64-580.126.20:202608262106' +MODEL_DIR='/data/Qwen3.8-Flash-Next' +CONTAINER='qwen38_flash_next_flagos_release' +test -d "$MODEL_DIR" +docker run -d --name "$CONTAINER" --network host --shm-size 64g \ + --device /dev/nvidia0 --device /dev/nvidia1 --device /dev/nvidia2 --device /dev/nvidia3 \ + --device /dev/nvidia4 --device /dev/nvidia5 --device /dev/nvidia6 --device /dev/nvidia7 \ + --device /dev/nvidiactl --device /dev/nvidia-uvm --device /dev/nvidia-uvm-tools \ + --device /dev/nvidia-nvlink --device /dev/nvidia-nvswitch0 --device /dev/nvidia-nvswitch1 \ + --device /dev/nvidia-nvswitch2 --device /dev/nvidia-nvswitch3 --device /dev/nvidia-nvswitchctl \ + --device /dev/nvidia-caps/nvidia-cap0 --device /dev/nvidia-caps/nvidia-cap1 --device /dev/nvidia-caps/nvidia-cap2 \ + -v /usr/lib/x86_64-linux-gnu/libcuda.so.580.126.20:/driver/libcuda.so:ro \ + -v /usr/lib/x86_64-linux-gnu/libcuda.so.580.126.20:/driver/libcuda.so.1:ro \ + -v /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.580.126.20:/driver/libnvidia-ml.so:ro \ + -v /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.580.126.20:/driver/libnvidia-ml.so.1:ro \ + -v /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.580.126.20:/driver/libnvidia-ptxjitcompiler.so:ro \ + -v /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.580.126.20:/driver/libnvidia-ptxjitcompiler.so.1:ro \ + -v "$MODEL_DIR:/models/Qwen3.8-Flash-Next:ro" \ + -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \ + "$IMAGE_URI" -lc 'exec sleep infinity' +``` +### Start the Server +```bash +docker exec -d qwen38_flash_next_flagos_release bash -lc 'export FLAGGEMS_ATEN_PLAN_CACHE=0 VLLM_PLUGINS=fl USE_FLAGGEMS=1 VLLM_FL_PREFER=flagos VLLM_FL_PREFER_ENABLED=true QWEN4_QSA_FUSED_COMPRESS=1 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_ENABLE_V1_MULTIPROCESSING=1 VLLM_MOE_USE_DEEP_GEMM=0 VLLM_USE_DEEP_GEMM=0 VLLM_USE_FLASHINFER_MOE_FP8=0 CUDA_DEVICE_ORDER=PCI_BUS_ID OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1; export VLLM_FL_FLAGOS_BLACKLIST=index_put_,index_put,_index_put_impl_,nonzero,copy_,to_copy,index,index_select,conv1d,_conv_depthwise2d,conv2d,pad,constant_pad_nd,mul; export LD_LIBRARY_PATH=/driver:/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}; exec python3 -m vllm.entrypoints.cli.main serve /models/Qwen3.8-Flash-Next --served-model-name Qwen3.8-Flash-Next --host 0.0.0.0 --port 8001 --tensor-parallel-size 8 --distributed-executor-backend mp --max-model-len 102400 --max-num-seqs 64 --max-num-batched-tokens 32768 --limit-mm-per-prompt '\''{"image":16}'\'' --mm-encoder-tp-mode data --gpu-memory-utilization 0.90 --no-enable-prefix-caching --disable-custom-all-reduce --moe-backend triton --compilation-config '\''{"mode":"NONE","cudagraph_mode":"FULL","cudagraph_capture_sizes":[1,2,4,8,16],"max_cudagraph_capture_size":16,"pass_config":{"fuse_allreduce_rms":false}}'\'' >/tmp/qwen38-flagrelease-server.log 2>&1' +``` + +## Service Invocation +### Invocation Script +```bash +curl --fail --silent --show-error http://127.0.0.1:8001/health >/dev/null +curl --fail --silent --show-error http://127.0.0.1:8001/v1/chat/completions \ + --header 'Content-Type: application/json' \ + --data-raw '{"model":"Qwen3.8-Flash-Next","messages":[{"role":"user","content":"Reply with exactly: OK"}],"temperature":0,"max_tokens":8}' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Qwen/Qwen3.8-Flash-Next and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-tsingmicro-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-tsingmicro-FlagOS.md new file mode 100644 index 000000000..e08bcbb11 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-tsingmicro-FlagOS.md @@ -0,0 +1,136 @@ +--- +base_model: +- "" +frameworks: +- "" +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +On August 26, Alibaba open-sourced its multimodal MoE model Qwen3.8-Flash-Next. The FlagOS community completed Day-0 synchronized adaptation across multiple chips, having accomplished multi-chip adaptation, precision alignment, and deployment validation based on the FlagOS unified open-source technology stack on 8 AI chips — including T-Head (Pingtouge), NVIDIA, Moore Threads, Ascend, MetaX, Kunlunxin, Hygon, and Iluvatar CoreX. The first batch uniformly provides a BF16-precision version. The multi-chip versions have been open-sourced on the ModelScope and Hugging Face platforms, where developers can directly obtain out-of-the-box solutions for their respective chips. + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Tsingmicro** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + +# Evaluation Results +## Benchmark Result +| Metrics | Qwen3.8-Flash-Next-Nvidia-Origin | Qwen3.8-Flash-Next-Tsingmicro-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 92.9 | Evaluating | +| MuSR | 78.57 | Evaluating | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 20.10.12 | +| Operating System | 22.04 LTS | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/qwen38-flash-next-test-tsingmicro001-gems4.2.1-treenone-cx0.1.0-plugin0.0.0-vllm0.20.2-cp310-pt211-raisa0.2927-x64-v0.29277.8:202608271915 + +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Qwen3.8-Flash-Next-BF16-tsingmicro-FlagOS --local_dir /data/Qwen3.8-Flash-Next +``` + +### Start the Container +```bash +docker run --privileged -dit \ + --name flagos \ + --shm-size=128g \ + --network host \ + --ipc=host \ + -v /sys:/sys \ + -v /dev:/dev \ + -v /lib/modules:/lib/modules \ + -v /mnt/nvme_data:/mnt/nvme_data \ + -v /data:/data \ + harbor.baai.ac.cn/flagrelease-public/qwen38-flash-next-test-tsingmicro001-gems4.2.1-treenone-cx0.1.0-plugin0.0.0-vllm0.20.2-cp310-pt211-raisa0.2927-x64-v0.29277.8:202608271915 \ + /bin/bash +docker exec -it flagos /bin/bash +``` +### Start the Server +```bash +cd /home/secure/260629145101/qwen38-flash-next/ +source qwen_env.sh +bash qwen3.8_flash_next_server.sh +``` + +## Service Invocation +### Invocation Script +```bash +curl -X POST http://localhost:9005/v1/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "/mnt/nvme_data/models/Qwen3.8-Flash-Next/", + "prompt": "中国的首都是哪里？", + "max_tokens": 10, + "temperature": 0.0 +}' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., "Explain the basics of quantum computing") +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a "develop once, run anywhere" workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of . This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Qwen/Qwen3.8-Flash-Next and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + + diff --git a/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-zhenwu-FlagOS.md b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-zhenwu-FlagOS.md new file mode 100644 index 000000000..b29bb3974 --- /dev/null +++ b/docs/flagrelease_en/model_readmes/FlagRelease_Qwen3.8-Flash-Next-BF16-zhenwu-FlagOS.md @@ -0,0 +1,138 @@ +--- +language: +- zh +- en +license: apache-2.0 +--- + +# Introduction +On August 26, Alibaba open-sourced its multimodal MoE model Qwen3.8-Flash-Next. The FlagOS community completed Day-0 synchronized adaptation across multiple chips, having accomplished multi-chip adaptation, precision alignment, and deployment validation based on the FlagOS unified open-source technology stack on 8 AI chips — including T-Head (Pingtouge), NVIDIA, Moore Threads, Ascend, MetaX, Kunlunxin, Hygon, and Iluvatar CoreX. The first batch uniformly provides a BF16-precision version. The multi-chip versions have been open-sourced on the ModelScope and Hugging Face platforms, where developers can directly obtain out-of-the-box solutions for their respective chips. + + +### Integrated Deployment +- Out-of-the-box inference scripts with pre-configured hardware and software parameters +- Released **FlagOS-Zhenwu** container image supporting deployment within minutes +### Consistency Validation +- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public. + + +# Evaluation Results +## Benchmark Result +| Metrics | Qwen3.8-Flash-Next-Nvidia-Origin | Qwen3.8-Flash-Next-Zhenwu-FlagOS | +|--------------|--------------------------------|--------------------------------------| +| GPQA_Diamond | 92.9 | Evaluating | +| MuSR | 78.57 | Evaluating | + +# User Guide +Environment Setup + +| Item | Version | +|------------------|----------------------| +| Docker Version | Docker version 28.1.0, build 4d8c241 | +| Operating System | Ubuntu 24.04.2 LTS | + +## Operation Steps + +### Download FlagOS Image +```bash +docker pull harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-ppu001-gems0.0-tree0.6.1-cxnone-plugin0.2.0-vllm0.24.0-cp312-pt210-x64-none:202608271317 +``` + +### Download Open-source Model Weights +```bash +pip install modelscope +modelscope download --model FlagRelease/Qwen3.8-Flash-Next-BF16-zhenwu-FlagOS --local_dir /data/Qwen3.8-Flash-Next +``` +### Start the Container +```bash +sudo docker run --privileged -dit \ + --network=host \ + --device=/dev/infiniband \ + --ipc=host \ + --device=/dev/alixpu_ctl \ + --device=/dev/alixpu \ + --ulimit memlock=-1 \ + --ulimit stack=67108864 \ + --init \ + -v /data:/data \ + -w /mnt/ \ + --name flagos \ + harbor.baai.ac.cn/flagrelease-public/qwen3.8-flash-ppu001-gems0.0-tree0.6.1-cxnone-plugin0.2.0-vllm0.24.0-cp312-pt210-x64-none:202608271317 +docker exec -it flagos /bin/bash +``` +### Start the Server +```bash +export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=6000 +vllm serve /data/Qwen3.8-Flash-Next/ \ + --port 8000 \ + --trust-remote-code \ + --served-model-name Qwen3.8-flash \ + --tensor-parallel-size 8 \ + --max-model-len 100000 \ + --gpu-memory-utilization 0.85 \ + --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "cudagraph_capture_sizes": [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128]}' + +``` + +## Service Invocation +### Invocation Script +```bash +curl http://127.0.0.1:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [{"role": "user", "content": "中国首都是？"}], + "max_tokens": 128 + }' +``` + + +### AnythingLLM Integration Guide + +#### 1. Download & Install + +- Visit the official site: https://anythingllm.com/ +- Choose the appropriate version for your OS (Windows/macOS/Linux) +- Follow the installation wizard to complete the setup + +#### 2. Configuration + +- Launch AnythingLLM +- Open settings (bottom left, fourth tab) +- Configure core LLM parameters +- Click "Save Settings" to apply changes + +#### 3. Model Interaction + +- After model loading is complete: +- Click **"New Conversation"** +- Enter your question (e.g., “Explain the basics of quantum computing”) +- Click the send button to get a response +# Technical Overview +**FlagOS** is a fully open-source system software stack designed to unify the "model–system–chip" layers and foster an open, collaborative ecosystem. It enables a “develop once, run anywhere” workflow across diverse AI accelerators, unlocking hardware performance, eliminating fragmentation among vendor-specific software stacks, and substantially lowering the cost of porting and maintaining AI workloads. With core technologies such as the **FlagScale**, together with vllm-plugin-fl, distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the **FlagOS** stack to automatically produce and release various combinations of \. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application. +## FlagGems +FlagGems is a high-performance, generic operator libraryimplemented in [Triton](https://github.com/openai/triton) language. It is built on a collection of backend-neutralkernels that aims to accelerate LLM (Large-Language Models) training and inference across diverse hardware platforms. +## FlagTree +FlagTree is an open source, unified compiler for multipleAI chips project dedicated to developing a diverse ecosystem of AI chip compilers and related tooling platforms, thereby fostering and strengthening the upstream and downstream Triton ecosystem. Currently in its initial phase, the project aims to maintain compatibility with existing adaptation solutions while unifying the codebase to rapidly implement single-repository multi-backend support. Forupstream model users, it provides unified compilation capabilities across multiple backends; for downstream chip manufacturers, it offers examples of Triton ecosystem integration. +## FlagScale and vllm-plugin-fl +Flagscale is a comprehensive toolkit designed to supportthe entire lifecycle of large models. It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vLLM](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models. +vllm-plugin-fl is a vLLM plugin built on the FlagOS unified multi-chip backend, to help flagscale support multi-chip on vllm framework. +## **FlagCX** +FlagCX is a scalable and adaptive cross-chip communication library. It serves as a platform where developers, researchers, and AI engineers can collaborate on various projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community. + +## **FlagEval Evaluation Framework** + FlagEval is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features: + - **Multi-dimensional Evaluation**: Supports 800+ modelevaluations across NLP, CV, Audio, and Multimodal fields,covering 20+ downstream tasks including language understanding and image-text generation. + - **Industry-Grade Use Cases**: Has completed horizonta1 evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation. + +# Contributing + +We warmly welcome global developers to join us: + +1. Submit Issues to report problems +2. Create Pull Requests to contribute code +3. Improve technical documentation +4. Expand hardware adaptation support +# License +The model weights are derived from Qwen/Qwen3.8-Flash-Next and are open‑sourced under the Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0.txt + +